Evaluating a think tank from the AEA365 (but should we?)

20 May 2011

Johanna Morariu, Senior Associate with Innovation Network. Innovation Network (http://www.innonet.org/) describes an approach to evaluating think tanks in the American Evaluation Association’s AEA365 blog. She draws from some studies and studies focused on think tank evaluation to address 10 categories or assessment areas:

  • Organisation infrastructure, capacity and management
  • Strategy and direction -and both organisation and project levels
  • Organisation visibility and reputation
  • Effectiveness of communication and outreach strategy
  • Quality of research products
  • Participation in congressional testimony
  • Research relevance, quality, usefulness, and rigour
  • Uptake of research in media and policy
  • Research influences the work of other leading reseachers
  • Research influences decision makers and/or policy

Again, as in most evaluations of the work of think tanks, this leaves out other contributions these organisations make:

  • What about their capacity to train and prepare new generations of policymakers? -track movement of staffers into policy or the private sector, for instance
  • And what about the power of think tanks to convene people and organisations from different sides of the argument -and help them find common ground?
  • Or the contribution that think tanks make to the education of elites -not to change their minds but to enlighten their own arguments?
  • Think tanks also trade on power and providing their supporters with access to key spaces. Do we rather not measure this? Let’s not pretend that think tanks have no political or economic allegiances (very few can).

By the way the studies she used are here, courtesy of Johanna:

  • Donald E. Abelson (2010). Is Anybody Listening? Assessing the Influence of Think Tanks. Chapter 1 in the edited volume, Think Tanks and Public Policies in Latin America.
  • Richard Bumgarner, Douglas Hattaway, Geoffery Lamb, James G. McGann, and Holly Wise (2006). Center for Global Development: Evaluation of Impact. Arabella Philanthropic Investment Advisors, LLC for the Bill & Melinda Gates Foundation, the William and Flora Hewlett Foundation, the John D. and Catherine T. MacArthur Foundation, and the Rockefeller Foundation.
  • Ingie Hovland (2007). Making a Difference: M&E of Policy Research. Working paper 281 for the Overseas Development Institute, London, UK.
  • James G. McGann (2006). Best Practices for Funding and Evaluating Think Tanks & Policy Research. McGann Associates for the William and Flora Hewlett Foundation.

But critical to this method was the development of a Theory of Change for the think tank. And from it they were able to develop the right indicators and tools to assess the organisation’s performance. The Theory of Change in this case is not the typical one often described by many projects or programmes. And in fact it looks more like a strategy diagram. This is what we want to achieve and this is how we’ll do it.

I have been asked before about evaluating think tanks. And approaches like Johanna Morariu’s are certainly useful. But I am not too sure if an evaluation is what we want for an organisation. I can see how we may evaluate a project (we can tell when the activities of the project are done -even if it is difficult to assess its effects in the short term). The same goes for a programme or policy. There is something that is done, then that something is done no more, and then we assess how (well) that something was done and if that something had the intended effect.

An organisation, however, does not operate in bursts of activity than then wind down to nothing. It does not start and then stop to be evaluated. Its parts may do, but not the organisation. Yes, there is ongoing evaluation but this all points towards the final (ex-post) evaluation.

For an organisation, a strategic review that considers all of the think tank’s functions is far more relevant. A reflection process around annual staff or research retreats is far more useful.