Skip to content

Evaluating a think tank from the AEA365 (but should we?)

Johanna Morariu, Senior Associate with Innovation Network. Innovation Network (http://www.innonet.org/) describes an approach to evaluating think tanks in the American Evaluation Association’s AEA365 blog. She draws from some studies and studies focused on think tank evaluation to address 10 categories or assessment areas:

  • Organisation infrastructure, capacity and management
  • Strategy and direction -and both organisation and project levels
  • Organisation visibility and reputation
  • Effectiveness of communication and outreach strategy
  • Quality of research products
  • Participation in congressional testimony
  • Research relevance, quality, usefulness, and rigour
  • Uptake of research in media and policy
  • Research influences the work of other leading reseachers
  • Research influences decision makers and/or policy

Again, as in most evaluations of the work of think tanks, this leaves out other contributions these organisations make:

  • What about their capacity to train and prepare new generations of policymakers? -track movement of staffers into policy or the private sector, for instance
  • And what about the power of think tanks to convene people and organisations from different sides of the argument -and help them find common ground?
  • Or the contribution that think tanks make to the education of elites -not to change their minds but to enlighten their own arguments?
  • Think tanks also trade on power and providing their supporters with access to key spaces. Do we rather not measure this? Let’s not pretend that think tanks have no political or economic allegiances (very few can).

By the way the studies she used are here, courtesy of Johanna:

  • Donald E. Abelson (2010). Is Anybody Listening? Assessing the Influence of Think Tanks. Chapter 1 in the edited volume, Think Tanks and Public Policies in Latin America.
  • Richard Bumgarner, Douglas Hattaway, Geoffery Lamb, James G. McGann, and Holly Wise (2006). Center for Global Development: Evaluation of Impact. Arabella Philanthropic Investment Advisors, LLC for the Bill & Melinda Gates Foundation, the William and Flora Hewlett Foundation, the John D. and Catherine T. MacArthur Foundation, and the Rockefeller Foundation.
  • Ingie Hovland (2007). Making a Difference: M&E of Policy Research. Working paper 281 for the Overseas Development Institute, London, UK.
  • James G. McGann (2006). Best Practices for Funding and Evaluating Think Tanks & Policy Research. McGann Associates for the William and Flora Hewlett Foundation.

But critical to this method was the development of a Theory of Change for the think tank. And from it they were able to develop the right indicators and tools to assess the organisation’s performance. The Theory of Change in this case is not the typical one often described by many projects or programmes. And in fact it looks more like a strategy diagram. This is what we want to achieve and this is how we’ll do it.

I have been asked before about evaluating think tanks. And approaches like Johanna Morariu’s are certainly useful. But I am not too sure if an evaluation is what we want for an organisation. I can see how we may evaluate a project (we can tell when the activities of the project are done -even if it is difficult to assess its effects in the short term). The same goes for a programme or policy. There is something that is done, then that something is done no more, and then we assess how (well) that something was done and if that something had the intended effect.

An organisation, however, does not operate in bursts of activity than then wind down to nothing. It does not start and then stop to be evaluated. Its parts may do, but not the organisation. Yes, there is ongoing evaluation but this all points towards the final (ex-post) evaluation.

For an organisation, a strategic review that considers all of the think tank’s functions is far more relevant. A reflection process around annual staff or research retreats is far more useful.

About these ads
One Comment Post a comment
  1. Hi Enrique, thanks for your insights into my blog post about evaluating an environmental think tank/advocacy organization. One point that your post did not clarify was that the organization I worked with was not a traditional think tank, in the way that we think of Heritage, Brookings, RAND, etc. Thinking about these more traditional U.S. think tanks (or even the whole gamut of European think tanks that seem to have a closer working relationship with government), your point the outcome of capacity to train new generations of policymakers is especially relevant. But for the organization that this case was about, an organization whose strategy is not focused on policymakers to the extent that traditional think tanks are, an outcome like that doesn’t make as much sense.

    I also read with interest your discussion of evaluation versus reflection. My definition of evaluation supports the idea of reflection–we encourage all organizations to engage in ongoing reflection and learning as the work unfolds to ensure it has the greatest impact possible. Evaluation is the systematic collection of data that can be used to engage in reflection, it provides the information that can be reflected upon.

    Again, thanks so much for your thoughtful treatment of my blog post. Best, -Johanna

    June 22, 2011

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 5,464 other followers

%d bloggers like this: