New Global Ranking of Think Tanks: is the Chinese version better?

30 May 2016

Ranking things is a universal sport. Think tanks had never quite been the subject of attempts to assess, measure or rank their work. The problem is that when one attempts to rank think tanks (even only measure their influence -which is an obsession of those in the sector) one will inevitably affect them.

This is not a simple bean-counting exercise. Think tanks are political agents -always in the lookout for threats and opportunities to their influence. By ranking them we (however neutral we may think we are) affect their political standing -we award them credibility or undermine it.

Among the many critiques levelled at the University of Pennsylvania’s ranking. its definition of think tanks is possibly the most significant. Over the years, the authors have accepted that the single-narrow definition used in its earlier issues failed to capture the diversity of think tank models that exist around the world.


Read more about the definition of think tanks in this new series:

Defining the Think Tank Label: An Ongoing Topic of Debate, Discussion and Analysis


The definition has widened now but in doing so it has shed light onto another challenge: how to compare think tanks in entirely different political context when, not only are they different, but the nature of the societies in which they exist are different, too? The boundaries of the definition are not the same everywhere.

Alternatives exist. Prospect Magazine in the UK and PODER Magazine in Peru have developed their own national think tank awards that explicitly link the assessment of think tanks’ performance to what is going on in the relevant policy sectors.

A review of existing models

The drive to develop China’s think tank community has been accompanied by an increase in research about think tanks that has focused, among other things, on making sense to the call for “think tanks with Chinese characteristics”.

In attempting to address this, researchers in China have developed an approach to evaluate and think tanks globally. You can download the 2015 report here.


Do you want to know more about think tanks’ rankings?

A national ranking of influence: Chinese and American models

Think tank rankings and awards: rigged, futile, or useful

National Think Tank Awards: Premio PODER 2014 in Peru celebrates innovation

This year, instead of ranking think tanks lets think about them more carefully


At a regional conference on think tanks in Asia held in Beijing in April 2016, Jing Linbo, from CASS Social Sciences Evaluation Center, presented the evaluation method used for ranking think tanks globally.

Definitions

The approach addresses a recurring challenge: what is a think tank?+

The authors accept that there is no consensus on the definition but they emphasise:

First, a think tank is a form of organization, rather than a natural person.

Second, think tanks must produce independent intellectual outputs…[and] should be equipped with staff possessing professional knowledge and skills to create new products of thoughts.

Finally, think tanks are supposed to have an impact on the formulation of public policy, which is the core function of a think tank. We believe that its influence upon public policy does not need to be understood as the special tendency of political ideology.

Methods to evaluate think tanks

The authors review a series of approaches to evaluate think tanks. They begin with the University of Pensilvania ranking and identify 6 major problems with it:

  1. The evaluation method lacks objectivity;
  2. It lacks a sufficiently strong research support;
  3. Expert selection is not transparent and it is unclear if they are sufficiently aware of the various regions and countries included;
  4. There are several factual errors, inclusions and commissions that undermine its credibility;
  5. Quality control and project management concerns leave it open to mistakes; and
  6. Many large and respectable think tanks do not approve or recognise the ranking.

Next, the authors review the Chinese Think Tank Reports by Shanghai Academy of Social Sciences. They report that the Reports attempted to combine objective and subjective evidence of think tanks’ value.

But their main contribution is probably a classification of think tanks that reflects think tank in China:

They classified them into the four think tank categories of Party, Politics and Army, Academy of Social Sciences, Colleges and Universities, and Private think tanks.

The authors suggest that this method presents three main problems:

  1. The definition of think tanks needs to be further clarified -certainly if it is to be used globally;
  2. The evaluation method needs to be improved. The current method: “Nomination + Evaluation + Ranking” could be improved to provide greater balance between objective and subjective approaches; and
  3. Greater transparency of the approach is necessary.

Finally, they review the Think Tank Evaluation by Horizon Research Consultancy Group and China Network, which considered four categories of influence indicators : professional influence, influence on government, social influence, and international influence.

This method builds on from the Shanghai Academy of Social Sciences evaluation by using the evidence they gather on the think tanks. This approach attempts to reduce the reliance on subjective indicators and increase the importance of objective ones.

Global versus domestic efforts

Their choice of rankings to review demands that we consider a key difference in scope. Where the University of Pensilvania ranking is global the other two focus on Chinese think tanks. We have often argued that this is problematic because think tanks are relevant to a political/policy space. A great think tank can only be great if it is meaningful for its society.

The CASS Social Sciences Evaluation Center‘s ranking is global in nature and so a direct competition to the University of Pennsylvania model. Still, an alternative is for countries to develop their own approaches to assess and evaluate think tanks, much like Prospect Magazine’s Award in the UK or the Premio PODER in Peru.

The AMI Index System for Comprehensive Evaluation of Global Think Tanks

Key characteristics

AMI refers to:+

  • Attractive Power: reputation attraction, staff attraction, product/outcomes attraction, and capital attraction;
  • Management Power: structure, systems, staff, style, shared values, and skills; and
  • Impact Power: policy influence, academic influence, social influence, and international influence.

Unlike the Shanghai and the Horizon Research Consultancy Group rankings,  the CASS Social Sciences Evaluation Center‘s ranking attempts to evaluate think tanks globally -thus competing, in essence, with the University of Pennsylvania ranking.

Crucially it:

  1. Attempts to construct the comprehensive evaluation index system which combines qualification with quantification;
  2. The system design is based on a view of think tanks work: they attract sources of power, manage them, and use them to influence or have an impact on power;
  3. Each category of indicators allows for depth and breadth;
  4. Makes use of expert groups and third party reviewers to build on one another; and
  5. Involves a significant research process including the generation of both primary and secondary evidence.

Challenges

No ranking is perfect, though. And this approach is limited by the following:

  1. It demands a great deal of effort to gather all the necessary data required to evaluate all think tanks directly;
  2. It still attempts to compare think tanks across countries and regions -one of the main critiques levelled at the UPENN ranking; and
  3. It does not include think tanks within CASS to ensure fairness and objectivity -but leave out a number of important think tanks within China.

And the winner is…

The AMI model ranks the top 5 think tanks:

  1. Carnegie Endowment for International Peace (US)
  2. Bruegel (Belgium)
  3. Heritage Foundation (US)
  4. Chatham House (UK)
  5. Stockholm International Peace Research Institute (Sweden)

Is it better?

This approach has a few weaknesses. I would argue that the greatest one is that to comprehensively rank all think tanks in the world one would need a team and resources that cannot be justifiably appropriated for this purpose.

Concerns about cross-country and cross-regional comparisons may have been addressed by the rather diverse nature of the top 5 (and top 10) spots. Six countries are included there.

But this is hard to tell as not all think tanks in the world have been evaluated -who is to say that there isn’t a great Argentinean think tank that wasn’t included in the analysis? Even if more were included this would raise concerns about how well the researchers involved could assess think tanks in every region and country.

Can we be objective?

This is a question I wish to leave us with. To a certain degree we could look at a think tank and determine how good it is: the quality of its work, how influential are its researchers, how credible it is, etc. These qualities can be measured (and described).

But think tanks are intended to influence policy -to attempt to affect the outcomes of public debate and policymaking.

While they can control the quality of their work and their efforts to achieve this, they have no control over influence itself. Their greatest contribution may be in keeping an idea alive, in developing new generations of policymakers, in acting as the intellectual opposition to those in power, etc.