{"id":895,"date":"2014-07-28T18:24:50","date_gmt":"2014-07-28T18:24:50","guid":{"rendered":"https:\/\/onthinktanks.org\/?p=895"},"modified":"2019-09-04T13:50:29","modified_gmt":"2019-09-04T18:50:29","slug":"think-tank-rankings-and-awards-rigged-futile-or-useful","status":"publish","type":"post","link":"https:\/\/onthinktanks.org\/articles\/think-tank-rankings-and-awards-rigged-futile-or-useful\/","title":{"rendered":"Think tank rankings and awards: rigged, futile, or useful"},"content":{"rendered":"

Prospect Magazine announced the winners of the\u00a02014 Prospect Think Tank of the Year Award\u00a0<\/a>two weeks ago. I have written about\u00a0my preference for this kind of award in the past<\/a>, so\u00a0it should not come as a surprise that I have been promoting a Peruvian version with a local magazine there:\u00a0Premio PODER<\/a>.<\/p>\n

The announcement of the winners sparked a bit of a transatlantic discussion about the validity of the award. Think Tank Watched asked:\u00a0Are Prospect Magazine\u2019s Think Tank of the Year Awards Rigged?<\/a>\u00a0(they do not think it was rigged, by the way, and no accusation has been made) and later:\u00a0Think Tank Awards & Rankings: A Futile Exercise?<\/a><\/p>\n

Both are valid questions. How come Brookings, top of the UPENN ranking (every year) was not even shortlisted. Heritage? Cato? AEI? None of the big American think tanks, with the exception of Carnegie, made it to the short list:<\/p>\n

The\u00a0Brookings Institution,\u00a0<\/b>although not on the shortlist,was cited this year for its especially strong work on the Syria crisis and was described by one judge as \u201chuge, but nimble.\u201d\u00a0The American Enterprise Institute<\/b>\u00a0was also noted for its significance, especially in its attempts to fashion a more moderate policy offering for the Republican Party, while the\u00a0Centre\u00a0<\/b>[sic]\u00a0for American Progress\u00a0<\/b>drew plaudits for its work on the left of the US political spectrum.<\/p><\/blockquote>\n

In discussions over email with Think Tank Watch I argued that in my opinion the award is not necessarily biased against large think tank, as the article asked. Rather, it was not biased against smaller think tanks. And this is a quality that makes it interesting for the majority of think tanks.<\/p>\n

The\u00a0second questions is more interesting and important: What is the point of these rankings and awards? Are they worth the effort?<\/p>\n

Since I have already expressed by preference in the world of rankings and awards I may as well try to put forward an argument for national think tank awards modelled after the Prospect Magazine Think Tank of the Year Award. I will acknowledge some of its weaknesses and try to outline its strengths.<\/p>\n

Discussions like these are very useful. As we attempt to develop the model for Peru we are keen\u00a0to learn from our experience<\/a>\u00a0and that of others. And questions like these are the right way forward. What I hope will be clear by the end of this post is that the subjective, value-ladden, and unscientific model promoted by Prospect Magazine is, by its very nature, perfectly able to address these questions. It can test new ideas, adopt the ones that work, and adapt as it grows and develops.<\/p>\n

I should acknowledge, finally, that I consider the rankings and awards to be a means to something else. So, in a way, they are, like Think Tank Watch says, futile only if they do not achieve anything else. And the question I ask is: are they helping think tanks to be better?<\/p>\n

Weaknesses<\/h2>\n

In the spirit of fairness, and given that\u00a0I have (and probably will) criticised the UPENN ranking<\/a>, I should begin with some criticism of the Prospect Awards model (the same model used for the Premio PODER).<\/p>\n

The best of all or the best of those who applied?<\/strong>\u00a0The first valid critique is that the award only considers the think tanks that applied for the award. So, to say that it is the best in the UK or Peru is to assume that all think tanks applied. While it is likely that the vast majority of think tanks in the UK have applied (it is, after all, the 14th edition of the award), not all think tanks in Peru applied last year (the first time the award was held). But, this is something that could be said of all awards. After all, are all films produced considered for the Oscar?<\/p>\n

I am confident that Prospect\u2019s judges\u2019 choices for the UK political scene come close to a \u2018best of all UK think tanks\u2019 decision but not so sure this is the case for the US and the European categories.<\/p>\n

For Europe, the award could focus on the EU policy community. Think tanks are political and to judge them one must tie into account their political space. This is what Prospect does rather well in the UK by linking the award to political relevance. If the European category focuses on Europe-wide or EU politics then it is likely to reach the same level of \u2018representativeness\u2019 as it has in the UK.<\/p>\n

For the US category, however, the challenge will remain. How can the Prospect award judge\u00a0all the think tanks in the US in a single category?\u00a0It could follow my European category suggestion and focus the award on the best US think tanks dealing with the UK or Europe, for instance; but this may be no more than reducing\u00a0the US applications\u00a0to Foreign Affairs only.<\/p>\n

It could add\u00a0categories to mimic the UK ones; but this could easily be met with a US based publication coming up with its own Award.\u00a0And just as Prospect can claim \u2018ownership\u2019 of the UK category, this new publication would claim the US for itself.<\/p>\n

\u201cIt is too subjective\u201d<\/strong>. This is a claim I have heard before. Sure, as Jeff Knezovich showed in his excellent\u00a0data visualisation exercise<\/a>\u00a0the winners of the Prospect Award were not necessarily the ones with most publications, Facebook or Twitter followers, or Google ranking.\u00a0CGD\u2019s work on measuring the public profile of think tanks<\/a>\u00a0showed the same thing in the case of the US. Numbers matter, sure, but it\u2019s what is behind them that matters more.<\/p>\n

In Jeff\u2019s visualisation, ODI and IDS (IDS was not shortlisted) come on top for:<\/p>\n