Skip to content

Posts tagged ‘evaluation’

Think Tank Initiative Evaluation: some ideas for the evaluators

The call for proposals for the second phase of the Think Tank Initiative presents an opportunity to reflect on what such an evaluation should entail. This 'wish-list' brings together some of the ideas presented in this blog over the years in an effort to encourage more and better applications.

Read more

Call for Proposals: Evaluation of Phase 2 of the Think Tank Initiative

IDRC has launched a call for proposals for the provision of an independent evaluation team to undertake an evaluation of the second phase of the Think Tank Initiative over 4.5 years. The evaluation will provide periodic, timely and actionable feedback to allow for the adaptive management of the Initiative, as well as providing rigorously documented and validated learning about the program.

Read more

Building sustainable think tanks is a long-run endeavour: the future of ACBF

This post outlines the Africa Capacity Building Foundation's new plans. It appears that a stronger focus on results and deeper engagement with 'senior demand side' players is on the cards. Also interesting, especially for a funder used to helping set up new think tanks, a focus on mature ones. Peter da Costa offers this as a closing post on a series dealing with a recent think tank summit in Africa.

Read more

Find Policy: “Ideas Worth Searching”

Find Policy is a new website offering targeted search. See what the leading think tanks are saying about specific issues. Hans Gutbrod describes why such faster and more focused search can be useful, and why it will can help less prominent institutions.

Read more

The external evaluation of the Think Tank Initiative: “what we’re learning, and how we’re responding to those lessons”

In this post, Peter Taylor, Program Manager of Think Tank Initiative, outlines the main findings of the TTI's first phase evaluation. The evaluators also identified a number of lessons and recommendations for the second phase: sharing, learning and collaboration are among them.

Read more

Who is responsible for a think tank’s influence?

Attempts to measure influence miss out on two fundamental questions related to current efforts and ideas focused on monitoring and evaluating think tanks: who is responsible for a think tank's influence? and what are they actually responsible for? Attempting to answer them led to two further issues: a question: when should think tanks claim influence? and, a conclusion: any claims of influence are political acts; they are claims of power over others.

Read more

Research uptake: what is it and can it be measured?

Is research uptake measurable? Can it be planned? Or is it just luck? This blog post reviews a number of issues that ought to be considered when trying to measure it. The post argues that instead of measuring it, we should attempt to understand it.

Read more

Think Tank Initiative 2012 Exchange: Sustaining quality in social policy research – lessons learned from institutional approaches

This is the third set of videos from the TTI Exchange sessions. This panel was about reviewing the lessons learned from institutional approaches on sustaining quality in social policy research.

First up was Rajeev Bhargava of the Centre for the Study of Developing Societies (CSDS).

Rajeev points out several factors that help nurture quality research. Some of these are creating a milieu that acknowledges that academics “get it right”: they grasp what is going on and strive for internal goods such as truth and plausibility. However, research also produces external goods, such as power, and think tanks can be lured by these goods, which is why they should not become the aim of research practice. He also emphasizes evidence based research and the importance of pluralism in any good institution.

Watch Rajeev’s talk:

Next was Mahmood Mamdani of the Makerere Institute of Social Research

Mahmood points out that while think tanks’ goal is to generate public debate on issues of public policy, researchers cannot assume that there is a causal relationship between policy makers and researchers. Academics must also not forget that the relationship between think tanks and policy makers stems from the understanding that think tanks are autonomous, and that the public agenda is not defined by the existing scope of public policy.

Watch Mahmood’s talk: 

Sukhadeo Thorat of the Indian Council of Social Science Research (ICSSR)

Sukhadeo stresses that sustainable policy demands that all policy suggestions are based on a realistic understanding of the issue at hand. Research is understanding, and policy is action based on understanding. This is why methodology is also an extremely important factor in research. He also states that ideal solutions may not be politically acceptable, so research should always try to offer more than one solution.

Watch Sukhadeo’s talk: 

And finally, Roxana Barrantes of the Instituto de Estudios Peruanos (IEP) 

Roxana first gives an account of the history of research institutions in Latin America, particularly in Peru, to point out that for strong academic contribution to knowledge, you need strong academic leadership. She mentions a couple of well known Latin American intellectuals and their impact on research and policy, and also emphasizes the importance of attracting young talent from universities.

Watch Roxana’s talk: 

Even the mighty get it wrong: putting think tanks in their place

This article on how Western think-tanks got it wrong on the Arab Spring got me thinking about recent discussions about think tanks’ impact. A great deal of emphasis is placed on whether think tanks should measure their influence -and even on the individual tools that think tanks sometimes use to communicate their work. In the last few weeks I have been asked to review a couple of papers on monitoring and evaluating think tanks policy influence and a couple more M&E framework proposals. This focus on policy influence often:

  • Forgets about all the other positive (and, at least, neutral) contributions think tanks can make to society (educate, provide oversight, improve political debate, break the consensus, strengthen parties, help fund research, etc.);
  • Overestimates the influence think tanks have; and
  • Tends to assume that think tanks are always right about what they say.

The fact is that think tanks play very small roles even in the most think tank savvy societies and quite often they do not know what they are talking about. This is a quote referring to Anthony Seldon that Emma Broadbent wrote for a study on think tanks in the UK:

Rohrer (2008) quotes Prof Anthony Seldon, editor of Ideas and Think Tanks in Contemporary Britain and biographer of Tony Blair, who believes their influence is overstated. Of the three major prime ministerial periods of post-war Britain, the Attlee, Thatcher and Blair eras, he believes only Atlee was significantly influenced by think tanks. For Blair, he says, “What is striking, as Blair’s biographer, is how little impact they [think tanks] made. You see hardly any influence on policy at all. It is very hard to see how ideas get into the system.” Seldon argues that “As the numbers of think tanks have accelerated their influence has declined. Influence comes from people who break off them and come into government.”

Last year, Prospect Magazine’s Annual Think Tank Awards had no real winner for the foreign policy category. According to the judges the winner would have had to predict the European financial crisis and the Arab Spring. None did. So not only are think tanks not as influential as sometimes we’d like to think they are, but they can also get it wrong; even where the resources and opportunities are as readily accessible as they are in the UK.

This is important for two reasons:

  1. According to the Prospect judges and to The National’s article think tanks play an important role not only in influencing policy directly (by telling governments what to do) but also by informing decision makers of things they may not be aware of. Think tanks, according to both political publications, fulfil a key function often overlooked by those too focused on tangible indicators of impact: enlightenment, information, inspiration… When attempting to assess think tanks contributions therefore we must pay attention to this more indirect yet crucial aspect of their work.
  2. Often, even the best think tanks, with all their resources and top academics -even with local offices and programmes, get it wrong or miss key processes and developments entirely. This means that we should not simply assume that everything a think tank says should influence policy. This would be quite dangerous. What we should be looking for is evidence that their recommendations have informed the public debate and the decisions made by those with the legitimate power to make them.

We should genuinely worry when donors put pressure on their grantees/sub-contractors to influence policy (and show them evidence of their influence). What they should be looking for is more informed policymaking and not just cases of policy influence.

Tracking research impact through Twitter

Cameron Neylon of the LSE blog Impact of Social Sciences has recently written a piece on the possibility of tracking research impact via Twitter. Monitoring the way how research influences policy and how professionals use the studies they’ve read on their day-to-day practice has proven to be difficult for a number of reasons: professionals don’t usually write new research papers citing the work they’ve used as sources; identifying said sources can be tricky because they may be several steps behind from the new study; and sometimes researchers aren’t even aware of their work being used because they are so far removed from its practical application.

Neylon mentions an example of a research article on HIV status, domestic violence and rape, reaching a practitioner community, which he found via Altmetric, a web app that helps track conversations around scientific articles online. The article was tweeted by several accounts, particularly by two South African support and advocacy groups. This example shows that it is possible to identify where research is being discussed and by whom.

It is possible, however, to go further than this:

More recently I’ve shown some other examples of heavily tweeted papers that relate to work funded by cancer charities. In one of those talks I made the throw away comment “You’ve always struggled to see whether practitioners actually use your research…and there are a lot of nurses on Twitter”. I hadn’t really followed that up until yesterday when I asked on twitter about research into the use of social media by nurses and was rapidly put in touch with a range of experts on the subject (remind me, how did we ask speculative research questions before Twitter?) . So the question I’m interested in probing is whether the application of research by nurses is something that can be tracked using links shared on Twitter as a proxy?

The hypothesis is that the links shared by nurses and their online community via Twitter are a viable proxy of a portion of the impact of certain research on clinical practice. This, of course, could be used for other professions as well, by monitoring what research is tweeted, how much it is retweeted and how often.

The Impact of Social Sciences blog also has a guide to using Twitter in university research, teaching, and impact activities.


Get every new post delivered to your Inbox.

Join 5,713 other followers