Skip to content

Posts tagged ‘evaluation’

How on earth do you measure the impact of your events?

Caroline Cassidy writes about a novel approach to measuring the impact of public events. The post provides practical advice on how to design and use M&E tools for events. She argues that since not all events have the same objectives, not all events should be evaluated using the same indicators.

Read more

Think Tank Initiative Evaluation: some ideas for the evaluators

The call for proposals for the second phase of the Think Tank Initiative presents an opportunity to reflect on what such an evaluation should entail. This 'wish-list' brings together some of the ideas presented in this blog over the years in an effort to encourage more and better applications.

Read more

Call for Proposals: Evaluation of Phase 2 of the Think Tank Initiative

IDRC has launched a call for proposals for the provision of an independent evaluation team to undertake an evaluation of the second phase of the Think Tank Initiative over 4.5 years. The evaluation will provide periodic, timely and actionable feedback to allow for the adaptive management of the Initiative, as well as providing rigorously documented and validated learning about the program.

Read more

Building sustainable think tanks is a long-run endeavour: the future of ACBF

This post outlines the Africa Capacity Building Foundation's new plans. It appears that a stronger focus on results and deeper engagement with 'senior demand side' players is on the cards. Also interesting, especially for a funder used to helping set up new think tanks, a focus on mature ones. Peter da Costa offers this as a closing post on a series dealing with a recent think tank summit in Africa.

Read more

Find Policy: “Ideas Worth Searching”

Find Policy is a new website offering targeted search. See what the leading think tanks are saying about specific issues. Hans Gutbrod describes why such faster and more focused search can be useful, and why it will can help less prominent institutions.

Read more

The external evaluation of the Think Tank Initiative: “what we’re learning, and how we’re responding to those lessons”

In this post, Peter Taylor, Program Manager of Think Tank Initiative, outlines the main findings of the TTI's first phase evaluation. The evaluators also identified a number of lessons and recommendations for the second phase: sharing, learning and collaboration are among them.

Read more

Who is responsible for a think tank’s influence?

Attempts to measure influence miss out on two fundamental questions related to current efforts and ideas focused on monitoring and evaluating think tanks: who is responsible for a think tank's influence? and what are they actually responsible for? Attempting to answer them led to two further issues: a question: when should think tanks claim influence? and, a conclusion: any claims of influence are political acts; they are claims of power over others.

Read more

Research uptake: what is it and can it be measured?

Is research uptake measurable? Can it be planned? Or is it just luck? This blog post reviews a number of issues that ought to be considered when trying to measure it. The post argues that instead of measuring it, we should attempt to understand it.

Read more

Think Tank Initiative 2012 Exchange: Sustaining quality in social policy research – lessons learned from institutional approaches

This is the third set of videos from the TTI Exchange sessions. This panel was about reviewing the lessons learned from institutional approaches on sustaining quality in social policy research.

First up was Rajeev Bhargava of the Centre for the Study of Developing Societies (CSDS).

Rajeev points out several factors that help nurture quality research. Some of these are creating a milieu that acknowledges that academics “get it right”: they grasp what is going on and strive for internal goods such as truth and plausibility. However, research also produces external goods, such as power, and think tanks can be lured by these goods, which is why they should not become the aim of research practice. He also emphasizes evidence based research and the importance of pluralism in any good institution.

Watch Rajeev’s talk:

Next was Mahmood Mamdani of the Makerere Institute of Social Research

Mahmood points out that while think tanks’ goal is to generate public debate on issues of public policy, researchers cannot assume that there is a causal relationship between policy makers and researchers. Academics must also not forget that the relationship between think tanks and policy makers stems from the understanding that think tanks are autonomous, and that the public agenda is not defined by the existing scope of public policy.

Watch Mahmood’s talk: 

Sukhadeo Thorat of the Indian Council of Social Science Research (ICSSR)

Sukhadeo stresses that sustainable policy demands that all policy suggestions are based on a realistic understanding of the issue at hand. Research is understanding, and policy is action based on understanding. This is why methodology is also an extremely important factor in research. He also states that ideal solutions may not be politically acceptable, so research should always try to offer more than one solution.

Watch Sukhadeo’s talk: 

And finally, Roxana Barrantes of the Instituto de Estudios Peruanos (IEP) 

Roxana first gives an account of the history of research institutions in Latin America, particularly in Peru, to point out that for strong academic contribution to knowledge, you need strong academic leadership. She mentions a couple of well known Latin American intellectuals and their impact on research and policy, and also emphasizes the importance of attracting young talent from universities.

Watch Roxana’s talk: 

Even the mighty get it wrong: putting think tanks in their place

This article on how Western think-tanks got it wrong on the Arab Spring got me thinking about recent discussions about think tanks’ impact. A great deal of emphasis is placed on whether think tanks should measure their influence -and even on the individual tools that think tanks sometimes use to communicate their work. In the last few weeks I have been asked to review a couple of papers on monitoring and evaluating think tanks policy influence and a couple more M&E framework proposals. This focus on policy influence often:

  • Forgets about all the other positive (and, at least, neutral) contributions think tanks can make to society (educate, provide oversight, improve political debate, break the consensus, strengthen parties, help fund research, etc.);
  • Overestimates the influence think tanks have; and
  • Tends to assume that think tanks are always right about what they say.

The fact is that think tanks play very small roles even in the most think tank savvy societies and quite often they do not know what they are talking about. This is a quote referring to Anthony Seldon that Emma Broadbent wrote for a study on think tanks in the UK:

Rohrer (2008) quotes Prof Anthony Seldon, editor of Ideas and Think Tanks in Contemporary Britain and biographer of Tony Blair, who believes their influence is overstated. Of the three major prime ministerial periods of post-war Britain, the Attlee, Thatcher and Blair eras, he believes only Atlee was significantly influenced by think tanks. For Blair, he says, “What is striking, as Blair’s biographer, is how little impact they [think tanks] made. You see hardly any influence on policy at all. It is very hard to see how ideas get into the system.” Seldon argues that “As the numbers of think tanks have accelerated their influence has declined. Influence comes from people who break off them and come into government.”

Last year, Prospect Magazine’s Annual Think Tank Awards had no real winner for the foreign policy category. According to the judges the winner would have had to predict the European financial crisis and the Arab Spring. None did. So not only are think tanks not as influential as sometimes we’d like to think they are, but they can also get it wrong; even where the resources and opportunities are as readily accessible as they are in the UK.

This is important for two reasons:

  1. According to the Prospect judges and to The National’s article think tanks play an important role not only in influencing policy directly (by telling governments what to do) but also by informing decision makers of things they may not be aware of. Think tanks, according to both political publications, fulfil a key function often overlooked by those too focused on tangible indicators of impact: enlightenment, information, inspiration… When attempting to assess think tanks contributions therefore we must pay attention to this more indirect yet crucial aspect of their work.
  2. Often, even the best think tanks, with all their resources and top academics -even with local offices and programmes, get it wrong or miss key processes and developments entirely. This means that we should not simply assume that everything a think tank says should influence policy. This would be quite dangerous. What we should be looking for is evidence that their recommendations have informed the public debate and the decisions made by those with the legitimate power to make them.

We should genuinely worry when donors put pressure on their grantees/sub-contractors to influence policy (and show them evidence of their influence). What they should be looking for is more informed policymaking and not just cases of policy influence.

Follow

Get every new post delivered to your Inbox.

Join 5,957 other followers