Caroline Cassidy writes about a novel approach to measuring the impact of public events. The post provides practical advice on how to design and use M&E tools for events. She argues that since not all events have the same objectives, not all events should be evaluated using the same indicators.
Posts tagged ‘evaluation’
The call for proposals for the second phase of the Think Tank Initiative presents an opportunity to reflect on what such an evaluation should entail. This 'wish-list' brings together some of the ideas presented in this blog over the years in an effort to encourage more and better applications.
Attempts to measure influence miss out on two fundamental questions related to current efforts and ideas focused on monitoring and evaluating think tanks: who is responsible for a think tank's influence? and what are they actually responsible for? Attempting to answer them led to two further issues: a question: when should think tanks claim influence? and, a conclusion: any claims of influence are political acts; they are claims of power over others.
Is research uptake measurable? Can it be planned? Or is it just luck? This blog post reviews a number of issues that ought to be considered when trying to measure it. The post argues that instead of measuring it, we should attempt to understand it.
This article on how Western think-tanks got it wrong on the Arab Spring got me thinking about recent discussions about think tanks’ impact. A great deal of emphasis is placed on whether think tanks should measure their influence -and even on the individual tools that think tanks sometimes use to communicate their work. In the last few weeks I have been asked to review a couple of papers on monitoring and evaluating think tanks policy influence and a couple more M&E framework proposals. This focus on policy influence often:
- Forgets about all the other positive (and, at least, neutral) contributions think tanks can make to society (educate, provide oversight, improve political debate, break the consensus, strengthen parties, help fund research, etc.);
- Overestimates the influence think tanks have; and
- Tends to assume that think tanks are always right about what they say.
The fact is that think tanks play very small roles even in the most think tank savvy societies and quite often they do not know what they are talking about. This is a quote referring to Anthony Seldon that Emma Broadbent wrote for a study on think tanks in the UK:
Rohrer (2008) quotes Prof Anthony Seldon, editor of Ideas and Think Tanks in Contemporary Britain and biographer of Tony Blair, who believes their influence is overstated. Of the three major prime ministerial periods of post-war Britain, the Attlee, Thatcher and Blair eras, he believes only Atlee was significantly influenced by think tanks. For Blair, he says, “What is striking, as Blair’s biographer, is how little impact they [think tanks] made. You see hardly any influence on policy at all. It is very hard to see how ideas get into the system.” Seldon argues that “As the numbers of think tanks have accelerated their influence has declined. Influence comes from people who break off them and come into government.”
Last year, Prospect Magazine’s Annual Think Tank Awards had no real winner for the foreign policy category. According to the judges the winner would have had to predict the European financial crisis and the Arab Spring. None did. So not only are think tanks not as influential as sometimes we’d like to think they are, but they can also get it wrong; even where the resources and opportunities are as readily accessible as they are in the UK.
This is important for two reasons:
- According to the Prospect judges and to The National’s article think tanks play an important role not only in influencing policy directly (by telling governments what to do) but also by informing decision makers of things they may not be aware of. Think tanks, according to both political publications, fulfil a key function often overlooked by those too focused on tangible indicators of impact: enlightenment, information, inspiration… When attempting to assess think tanks contributions therefore we must pay attention to this more indirect yet crucial aspect of their work.
- Often, even the best think tanks, with all their resources and top academics -even with local offices and programmes, get it wrong or miss key processes and developments entirely. This means that we should not simply assume that everything a think tank says should influence policy. This would be quite dangerous. What we should be looking for is evidence that their recommendations have informed the public debate and the decisions made by those with the legitimate power to make them.
We should genuinely worry when donors put pressure on their grantees/sub-contractors to influence policy (and show them evidence of their influence). What they should be looking for is more informed policymaking and not just cases of policy influence.