There are a lot of rankings out there – universities, think tanks, countries, you name it. There’s also a lot of criticism of rankings, like this recent Twitter thread by OTT Director Enrique Mendizabal pointing out the flaws in think tank ranking. Enrique’s critique is what we would call ‘methodological criticism’, and it’s by far the most common type.
People also criticise rankings for their adverse effect on behaviour and decisions, for biases, or for the commercial interests of rankers (a common criticism in the university sector). But even those flaws are very often explained by invoking methodological criticism. Somehow, for many people it’s difficult to criticise rankings beyond that.
In the research I do with my colleagues at the Faculty of Sociology at Bielefeld University, we explicitly do not discuss ranking methods from a normative point of view. Instead, we look at rankings sociologically, which means that we are interested in what rankings are as a social phenomenon and how that affects reality and the way people see it. We are especially interested in examining the premises that rankings rest on, which is something critics rarely pick up.
The reputational race
In our research we conceptualise rankings as a social operation, consisting of zero-sum comparisons of performances that are repeatedly and regularly published.
The exact hierarchical table, which is part and parcel of every ranking, suggests that reputation for performance is scarce. In a ranking, there can only be one best, one second-best, and so on. This is what distinguishes a ranking from other devices, such as ratings or benchmarks.
However, in and of itself, reputation for performance is not necessarily a scarce resource. The amount of ‘goodness of performance’ that can be attributed to an actor, be it a university, a think tank or a country, does not have to be limited.
And yet, rankings present this reputation as limited. So that in a ranking, actors must compete for it. The more rankings are normalised as a way of looking at the social world, the more this notion of ‘reputational scarcity’ is accepted.
Another critical element in accounting for the effect of rankings on public discourse is that they are – especially the very influential ones – produced and published not only repeatedly, but also regularly.
This imposes a calendar. It paces reputational changes in ways which may disrupt well-established practices. In a context in which rankings are normalised, a ranked actor can never take a break from competing.
This is probably why phrases such as ‘reputation race’ resonate. Because it is kind of a race. Rankers may argue that this race exists independently of rankings, whereby rankings only help navigate it. But a closer look reveals that rankings do not accelerate a race, they create the race. This is, in a nutshell, how rankings produce competition.
Rankings reflect how society constructs worth
In this sense, we do not discuss rankings as flawed in and of themselves, or some rankings as more flawed than others. All rankings are staged performances, in a manner of speaking. And we are the audience. They seek to convince us that their performance is the reality. It’s up to us if we ‘buy’ it or not. And if anything is ‘flawed’, it’s our ways of judging the value or worth of things by looking at rankings for guidance.
So why do think tanks choose to trust a ranking then, especially if there is, as critics say, a broad consensus that rankings do more harm than good? While this is an important question to ask, there isn’t a simple and straightforward answer to it. We must understand that rankings do not operate in isolation from the rest of social reality and we should not think of them as an external imposition on organisations or countries.
Rather, we need to improve our understanding of how they resonate with the broader cultural and institutional environment from which they emerge and against which they ‘make sense.’ For example, we live in times when rationality is celebrated, and rankings are perfectly in tune with what the sociologists Meyer and Rowan called ‘rationalised myths’, such as development, transparency, accountability, competition, excellence, strategy, and so on.
As seemingly rational instruments, rankings are enveloped in an aura of objectivity. But in order for them to be trusted, they need to do some work of convincing that they are truly impartial and based on ‘facts’ and scientific reasoning.
Therefore, in discourses on performance, reputation or value even, rankings have a tendency to replace other ways of constructing ‘worth’, but also performance and improvement, of individual actors.
In a way, rankings are also an expression of the meritocratic ideal of self-making through continuous striving to improve and ‘climb’. Because rankings discursively ‘fit’ so many other things in society, people tend to accept them as natural, and even necessary and desired.
Are there alternatives?
The question of alternatives is the logical one and I would think that, first, this depends on the context. What may work for universities, may not necessarily work for think tanks or some of the country rankings.
The same logic applies when asking about the negative and positive effects of rankings. Here it is especially important to make the distinction between a short-term and long-term perspective. Something may work in the short run, but could be a disaster in the long run. It also may work for some actors, but not for others.
In this sense, if we want to develop worldviews that allow for a viable alternative to rankings, we need to start with normalising the idea that alternatives are possible and then create spaces for discussing them.
And when I say alternative, I do not mean necessarily an alternative to rankings in the narrow sense. Rather, in our thinking about how we define and recognise value in our area of work and interest. We need to look beyond the entrenched idea of the modern world as a stratified order, whose members are imagined as continuously striving to overtake those they are compared with. It is this imaginary that rankings capture and which is critical to their widespread allure.
Challenging that may require a leap of imagination as well as a good deal of readiness to adopt a long-term view on things, but I would think there’s no way around it.