{"id":1642,"date":"2013-11-26T13:02:41","date_gmt":"2013-11-26T18:02:41","guid":{"rendered":"https:\/\/onthinktanks.org\/articles\/\/"},"modified":"2016-01-23T13:04:18","modified_gmt":"2016-01-23T18:04:18","slug":"this-year-instead-of-ranking-think-tanks-lets-think-about-them-more-carefully","status":"publish","type":"post","link":"https:\/\/onthinktanks.org\/articles\/this-year-instead-of-ranking-think-tanks-lets-think-about-them-more-carefully\/","title":{"rendered":"This year, instead of ranking think tanks lets think about them more carefully"},"content":{"rendered":"
David Roodman<\/a>\u00a0and\u00a0Julia Clark<\/a>\u00a0from the Center for Global Development have posted a very interesting reflection on the now unfortunately famous McGann think tank ranking. In\u00a0Envelope, Please: Seeking a New Way to Rank Think Tanks<\/a>, they offer an alternative to a global ranking exercise.<\/p>\n By now most people working in or for think tanks must have received a few (more than a few, actually) email reminders to submit nominations for the ranking. I have even heard of \u00a0emails ‘threatening’ a fall in the rankings for those think tanks not willing to participate. But this is still no more that rumour.<\/p>\n Criticism to the ranking on this blog has been rather consistent (See:\u00a0on\u00a0rankings<\/a>,\u00a0Another year, another ranking of think tanks (and surprise surprise, Brookings is still the\u00a0best)<\/a>,\u00a0Goran\u2019s recommendations on think tank\u00a0rankings<\/a>,\u00a0The mighty influence of think tanks (in the\u00a0US)<\/a>,\u00a0And the winner is: Brookings \u2026 but, once again, the loser: critical\u00a0analysis<\/a>, and\u00a0The Go to Think Tank Index: two\u00a0critiques<\/a>). Even before, when I was working at ODI, I felt that the effort put into this ranking exercise could make a more important contribution elsewhere (I should stress that McGann’s expertise on think tanks is not under question here: in fact I wish he used his time and the time of his many assistants more productively). The process, and the ranking itself, is in my view (and that of others)\u00a0inherently flawed<\/a>:\u00a0It confuses visibility with influence on the substance of policy and politics.<\/p>\n Roodman and Clark offer an alternative: not a ranking but an exercise in attempting to measure the aspects of think tanks and their actions that can be measured. This may considered by some as being too cautious; but their caution is based on experience:<\/p>\n Our experience with\u00a0building policy indexes<\/a>\u00a0such as the\u00a0Commitment to Development Index<\/a>makes us keenly aware of the limitations of any such exercise. Think tank profile is not think tank impact. Fundamentally, success is hard to quantify because think tanks aim to shift the thinking of communities. But the operative question is not whether we can achieve perfection. It is whether the status quo can be improved upon. Seemingly, it can be: it only took us a few weeks to choose metrics and gather data, and thus produce additional, meaningful information.<\/p><\/blockquote>\n And this meaningful information has provided ample opportunities for a meaningful discussion. So what the McGann ranking has failed to do year after year, Roodman’s and Clark’s exercise has managed in a single post. The authors identify four key methodological issue that could open up several lines of very interesting reflection (I quote in full to encourage others to engage with their own reflections and maybe suggest alternative solutions to the challenges they faced):<\/p>\n Should we focus only on what can be measured? I do not think so. I think that subjectivity is important when assessing the contribution of think tanks to any society or community because the value of a think tank to that community is subjective. After all, when assessing value we have no other way but to ask, directly or indirectly, whether those who use, or could use, their research and advice, value them or not. But subjectivity can be managed better when the policy space or the community is more clearly defined. Comparing think tanks across an entire continent offers no valuable insights unless a common playing field of characteristics is used. Argentinean and Brazilian think tanks are more likely to feature on a Latin American ranking than Bolivians but they are not likely to have much influence over Bolivian policy. Surely Bolivian think tank can learn from their peers in Brazil but it does not help to rank them against each other.<\/p>\n Location<\/strong>\u00a0then is a key unit of analysis; and a methodological issue that is absent from CGD’s list above. When comparing think tanks we should think hard about the space that these organisations share. Comparing think tanks in Indonesia would be better than comparing think tanks in South East Asia. But comparing regionally focused or foreign policy think tanks in the region may be better than just looking at these in a single country. Similarly, comparing sub-national think tanks to national think tanks may not be a straight forward affair. While their strategies may be the same their policy audiences are likely to be different and the scale of their influence incomparable: sub-national think tanks are more likely to focus on influencing policy at the provincial or state level while their national peers would be expected to operate in national or federal spaces. As a consequence the visibility and overall influence of the national think tank may be much greater than the sub-national one: but it would not be appropriate to rank one before the other.<\/p>\n Lets hope that the ranking (which is coming) encourages more think tanks to do what CGD has done. Instead of buying into a ranking that they know is flawed (and they should; after all they are supposed to be all for quality research) they should respond by challenging its flaws and searching for more appropriate alternatives and a better use of the information that is now more readily available than ever.<\/p>\n","protected":false},"excerpt":{"rendered":" David Roodman\u00a0and\u00a0Julia Clark\u00a0from the Center for Global Development have posted a very interesting reflection on the now unfortunately famous McGann think tank ranking. In\u00a0Envelope, Please: Seeking a New Way to Rank Think Tanks, they offer an alternative to a global ranking exercise. By now most people working in or for think tanks must have received […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","footnotes":""},"tags":[209,489,307,488],"acf":[],"_links":{"self":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/1642"}],"collection":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/comments?post=1642"}],"version-history":[{"count":0,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/1642\/revisions"}],"wp:attachment":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/media?parent=1642"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/tags?post=1642"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
\n