{"id":895,"date":"2014-07-28T18:24:50","date_gmt":"2014-07-28T18:24:50","guid":{"rendered":"https:\/\/onthinktanks.org\/?p=895"},"modified":"2019-09-04T13:50:29","modified_gmt":"2019-09-04T18:50:29","slug":"think-tank-rankings-and-awards-rigged-futile-or-useful","status":"publish","type":"post","link":"https:\/\/onthinktanks.org\/articles\/think-tank-rankings-and-awards-rigged-futile-or-useful\/","title":{"rendered":"Think tank rankings and awards: rigged, futile, or useful"},"content":{"rendered":"
Prospect Magazine announced the winners of the\u00a02014 Prospect Think Tank of the Year Award\u00a0<\/a>two weeks ago. I have written about\u00a0my preference for this kind of award in the past<\/a>, so\u00a0it should not come as a surprise that I have been promoting a Peruvian version with a local magazine there:\u00a0Premio PODER<\/a>.<\/p>\n The announcement of the winners sparked a bit of a transatlantic discussion about the validity of the award. Think Tank Watched asked:\u00a0Are Prospect Magazine\u2019s Think Tank of the Year Awards Rigged?<\/a>\u00a0(they do not think it was rigged, by the way, and no accusation has been made) and later:\u00a0Think Tank Awards & Rankings: A Futile Exercise?<\/a><\/p>\n Both are valid questions. How come Brookings, top of the UPENN ranking (every year) was not even shortlisted. Heritage? Cato? AEI? None of the big American think tanks, with the exception of Carnegie, made it to the short list:<\/p>\n The\u00a0Brookings Institution,\u00a0<\/b>although not on the shortlist,was cited this year for its especially strong work on the Syria crisis and was described by one judge as \u201chuge, but nimble.\u201d\u00a0The American Enterprise Institute<\/b>\u00a0was also noted for its significance, especially in its attempts to fashion a more moderate policy offering for the Republican Party, while the\u00a0Centre\u00a0<\/b>[sic]\u00a0for American Progress\u00a0<\/b>drew plaudits for its work on the left of the US political spectrum.<\/p><\/blockquote>\n In discussions over email with Think Tank Watch I argued that in my opinion the award is not necessarily biased against large think tank, as the article asked. Rather, it was not biased against smaller think tanks. And this is a quality that makes it interesting for the majority of think tanks.<\/p>\n The\u00a0second questions is more interesting and important: What is the point of these rankings and awards? Are they worth the effort?<\/p>\n Since I have already expressed by preference in the world of rankings and awards I may as well try to put forward an argument for national think tank awards modelled after the Prospect Magazine Think Tank of the Year Award. I will acknowledge some of its weaknesses and try to outline its strengths.<\/p>\n Discussions like these are very useful. As we attempt to develop the model for Peru we are keen\u00a0to learn from our experience<\/a>\u00a0and that of others. And questions like these are the right way forward. What I hope will be clear by the end of this post is that the subjective, value-ladden, and unscientific model promoted by Prospect Magazine is, by its very nature, perfectly able to address these questions. It can test new ideas, adopt the ones that work, and adapt as it grows and develops.<\/p>\n I should acknowledge, finally, that I consider the rankings and awards to be a means to something else. So, in a way, they are, like Think Tank Watch says, futile only if they do not achieve anything else. And the question I ask is: are they helping think tanks to be better?<\/p>\n In the spirit of fairness, and given that\u00a0I have (and probably will) criticised the UPENN ranking<\/a>, I should begin with some criticism of the Prospect Awards model (the same model used for the Premio PODER).<\/p>\n The best of all or the best of those who applied?<\/strong>\u00a0The first valid critique is that the award only considers the think tanks that applied for the award. So, to say that it is the best in the UK or Peru is to assume that all think tanks applied. While it is likely that the vast majority of think tanks in the UK have applied (it is, after all, the 14th edition of the award), not all think tanks in Peru applied last year (the first time the award was held). But, this is something that could be said of all awards. After all, are all films produced considered for the Oscar?<\/p>\n I am confident that Prospect\u2019s judges\u2019 choices for the UK political scene come close to a \u2018best of all UK think tanks\u2019 decision but not so sure this is the case for the US and the European categories.<\/p>\n For Europe, the award could focus on the EU policy community. Think tanks are political and to judge them one must tie into account their political space. This is what Prospect does rather well in the UK by linking the award to political relevance. If the European category focuses on Europe-wide or EU politics then it is likely to reach the same level of \u2018representativeness\u2019 as it has in the UK.<\/p>\n For the US category, however, the challenge will remain. How can the Prospect award judge\u00a0all the think tanks in the US in a single category?\u00a0It could follow my European category suggestion and focus the award on the best US think tanks dealing with the UK or Europe, for instance; but this may be no more than reducing\u00a0the US applications\u00a0to Foreign Affairs only.<\/p>\n It could add\u00a0categories to mimic the UK ones; but this could easily be met with a US based publication coming up with its own Award.\u00a0And just as Prospect can claim \u2018ownership\u2019 of the UK category, this new publication would claim the US for itself.<\/p>\n \u201cIt is too subjective\u201d<\/strong>. This is a claim I have heard before. Sure, as Jeff Knezovich showed in his excellent\u00a0data visualisation exercise<\/a>\u00a0the winners of the Prospect Award were not necessarily the ones with most publications, Facebook or Twitter followers, or Google ranking.\u00a0CGD\u2019s work on measuring the public profile of think tanks<\/a>\u00a0showed the same thing in the case of the US. Numbers matter, sure, but it\u2019s what is behind them that matters more.<\/p>\n In Jeff\u2019s visualisation, ODI and IDS (IDS was not shortlisted) come on top for:<\/p>\n But they are nowhere to be found for media mentions. And for an award like Prospect\u2019s this matter more. Media mentions are closely associated with the think tanks\u2019 political relevance. Did their ideas matter and inform the public debate? ODI and IDS, both focusing on international development policy, having a global audience, and being rather large organisations relative to other think tanks in the UK \u00a0are likely to do much better on the three mentioned above. But will always struggle, certainly in the UK political space, to get the same attention that the IFS, IPPR and other winners receive from the media and politicians.<\/p>\n And the problem with a subjective award is that good important work, if unpopular, may go unnoticed and unrewarded.<\/p>\n The strengths of this model are best illustrated in comparison with alternatives. Let\u2019s take three other models (I am not including\u00a0David Roodman and Julia Clark\u2019s\u00a0<\/a>effort nor\u00a0WonkComms\u2019 Top Trumps<\/a>\u00a0because they are not intended to reward think tanks, just to make a point -both very serious ones, in their own way).<\/p>\n The first one is, of course the UPENN ranking of think tanks. This ranking has been growing over the years but is yet to offer any meaningful contrition to our understanding of think tanks and what makes them valuable. The raking has, for sure, raised the profile of think tanks in many countries where the label was unknown. But the response from think tanks has not been the most helpful for their own interests.<\/p>\n There is ample criticism of the methodology; and not just by think tanks that didn\u2019t do well enough.\u00a0David Roodman at CGD has addressed the ranking\u2019s shortcoming in the past<\/a>, and so have\u00a0Jan Trevisan<\/a>, at the\u00a0International Centre for Climate Governance<\/a>, and Christian Seiler and Klaus Wohlrabe published their own, very well researched,\u00a0critique of the 2009 think tank index<\/a>.<\/p>\n In essence, the ranking fails on a number of factors, including:<\/p>\n The second alternative model is the RePEc \u2018awards\u2019.\u00a0RePEc<\/a>\u00a0is\u00a0an\u00a0effort of \u201chundreds of volunteers in\u00a082 countries<\/a>\u00a0to enhance the dissemination of research in Economics and related sciences\u201d. It hosts a number of services, including\u00a0IDEAS<\/a>, an open bibliographic database on economics and\u00a0EDIRC<\/a>, a data base of research institutions. The size of the data base is impressive:<\/p>\n Currently\u00a012,966<\/b>\u00a0institutions in\u00a0231<\/b>\u00a0countries and territories are listed<\/p><\/blockquote>\n IDEAS uses its database to develop a ranking and this year\u2019s has been recently published. The Top 10 offer an interesting view:<\/p>\n First, of all, Brookings is not first (as in the UPENN ranking). Second, there are quite a few European organisations in the list. So what is going on? The rankings\u2019\u00a0methodology<\/a>\u00a0may offer some clues, but let\u2019s use the same criteria:<\/p>\n The RePEc\u00a0and\u00a0the UPENN rankings\u00a0are\u00a0delinked to politics and that is their most significant flaw.<\/p>\n The ICCG publishes an annual\u00a0Climate Change Think Tank award<\/a>. The award focuses on climate change policy so it already defines a very specific (yet global) political space. The criteria used are quite interesting. It combines quantitative measure with what it calls\u00a0Fuzzy Measures;\u00a0<\/em>basically \u201cNon-Additive Measures and Aggregation Operators\u201d, which, I must\u00a0confess, I do not fully understand.<\/em><\/p>\n <\/p>\n These calculations provide two rankings: \u201cthe\u00a0Standardized Ranking<\/i>\u2013 measures the most efficient think tanks in per capita\/researcher terms, while the\u00a0Absolute Ranking\u00a0<\/i>measures the best think tanks in absolute terms, regardless of their efficiency and hence size\u201d. This is interesting but efficient in this case is not related to impact or relevance. It is mainly an issue of size (smaller organisations with lots of publications and participation in lots events are better ranked).<\/p>\n So on the same criteria, to be fair:<\/p>\n All three models described above share another flaw (the main one being the absence of politics): they are poor guides of overall performance. Think tank directors that showcase their high UPENN ranking and board members or funders who demand them are really doing a disservice to their organisations. They brush aside many more important aspects of a think tank (see, for example the case of\u00a0SIPRI: ranked among the top but an undesirable place to work according to its unions<\/a>). RePEc is less known but could not be used to give an overall sense of the think tanks\u2019 performance. It would be a good judge of its research production, but only if its outputs were picked up by the RePEc database. And ICCG could be too focused on participation in international\u00a0events; an excellent indicator but not one that can paint a whole picture of the think tank\u2019s performance.<\/p>\n But they offer something that the Prospect Award does not: the opportunity to cover quite a few (and potentially all) countries and regions in a single ranking or exercise. And developing country think tank funders like scalable projects like these.<\/p>\n Nick Scott<\/a>, from WonkComms, said it best:<\/p>\n it\u2019s not about being the best, it\u2019s about being better.<\/p><\/blockquote>\n First the criteria:<\/p>\n The value of a think tank to its community and broader society, its influence, the outcome of that influence, these are all subjective questions. Quantifiable measures can help. RePEc and the ICCG manage to do this well. Popularity is also important; the UPENN ranking is, in essence, a popularity contest. But in the end, it is what those involved in the same political space as the think tanks think about them and about the impact they had on it, what matters.<\/p>\nWeaknesses<\/h2>\n
\n
Strengths -or rather the weaknesses of others<\/h2>\n
\n
\n
\n
\n
Is the Prospect Award the Best? Maybe just better<\/h2>\n
\n