Think Tank Initiative Evaluation: some ideas for the evaluators
The Think Tank Initiative has launched a call to recruit an evaluation team for its second phase. You can find details on the Request for Proposal and how to apply on their website.
Not long ago, Peter Taylor, the head of the initiative, wrote about the lessons learned from the first phase evaluation, thus providing some good ideas about what to look for in developing the new proposals for this project.
With the objective of encouraging new players in this sector (particularly from developing countries) to apply or to play greater roles in the evaluation teams, let me offer some ideas of my own. These are in no way to be taken as binding or to have any significance to the TTI selection process. They will evaluate the proposals on their own merits according to their own criteria.
But here is my wish-list of things I’d like to see in the proposals:
- It would be great is more bids, and the winning bid, were from or involved evaluators from developing countries in leading roles. It would be particularly exiting if the bids were from or included organisations in the countries where the TTI is working. This is not necessarily about quality (although it could be about cost); there is no reason why European or US based organisations wouldn’t have the technical skills for this task. But in my view the previous evaluation lacked a deep knowledge of the policy research communities that only belonging to those communities gives you. I do not think a few days (even a few days very year) in each country for a few interviews will do. the budget would have to consider large stints in each country, but even then it may not be enough. I have argued this many times (including in a blog on ‘tourist funders‘). Maybe an option would be to find leading evaluators in each region and bring them together in a network type of approach. TTI will have to consider its budget for the evaluation if it wants to delve deep into each region and country where it has been working; and bidders will need to look harder for leading evaluators in each of the regions.
- New faces: It would be fantastic if the evaluators were not ‘part of the evidence based/informed policy in development‘ industry. I find that there are too many vested interests in the sector already. Can one be openly critical about a funder when one depends on it for future work (a lot of future work)? Can one be critical of certain think tanks when one is also engaging with them as partners in other projects? Can one be neutral about what works and what doesn’t when one has one’s sights on opportunities to build capacity on what one knows more about? There are no reasons why professional evaluators working in other sectors could not do this. Competent people can learn about new contexts and, lets face it, this is not rocket science.
- A good alternative would be to look at academics, consultancies or think tanks in developed or non-TTI countries (working on domestic/foreign policy issues -so not international development only) who would be able to bring their own experience of their think tank or evidence based policy communities into the evaluation. I would hope to see applications from Mexico, Argentina, Colombia, South Africa, india (it is big enough), Indonesia, Singapore, etc. as well as from Canada, the US, the UK, Europe, etc.
- But if this is not possible (reaching out to other sectors in the Aid industry is very hard), then it would be amazing if the evaluators were at least open about their other responsibilities and commitments and conflicts of interest. Constant publication of progress (including the data and their analyses) would go a long way.
- It would also be great if the evaluation itself awarded a much greater emphasis to the effect of the TTI on the broader think tank communities of each of the countries in which it has intervened. This is important because the TTI did not set out to support just the think tanks it funds (this is what I understand). If it did, then it would have had to take into account that choosing which think tanks to support was an inherently political (and ideological) action. Funding has given more power to a set of political actors (a think tank and its networks) over others (other think tanks and their networks) in a the country. Therefore, understanding the nature of the think tank communities (the political and academic players, as well as other intellectuals, ideologues, parties, interest groups, etc) before the TTI arrived, how it has changed (for good or worse,) and what direction it seems to be taking is crucial. This is why local evaluators should be in leading roles.
- As a follow-up, then, it would be great is the questions focused on whether the TTI has helped or contributed in developing a more sustainable policy research community in the countries where it has worked. In my view the TTI and the evaluators should assume that this is the second and last phase. IDRC has been funding many of these think tanks (before and during the TTI) for decades. How much longer can it keep doing so? Surely it is time for domestic funders to step up to the plate. Success then does not necessarily mean that all the TTI think tanks will make it through post TTI. What matters is that the community, who ever it includes, is strong and sustainable.
- It would be a shame if the evaluation focused too much on instances or cases of policy influence. There is a lot more to think tanks and the contributions they make to their countries and communities than plain old direct influence. Their roles include educating the public, preparing new generations of policymakers, informing and strengthening academic practice, creating and maintaining spaces for debate, challenging the status quo (even if unsuccessfully), etc. Being influential depends little on the think tanks and greatly on other factors and actors.
- It would also be a shame if the evaluation focused too much on metrics of ‘having’ new skills or resources. What counts is not that now a think tank has a communications officer or more funding from other sources. What counts is that now the think tank has the capacity to decide when it needs them, how to get them, and is able to if it wants to. And this is much about the think tanks capabilities as about their context.
- The evaluators should not see themselves as the only ones in this role. The TTI and the evaluators should adopt an Open Data Policy (that its funders already support) and share all the data is has on the think tanks it supports around the world (the think tanks should be doing it anyway). This would make it possible for others to contribute with fresh analysis and insights.
- Finally, it would be just perfect if the evaluation approach was in part (or greatly) informed by the think tanks themselves. This is not an easy thing to do. TTI has worked hard to find and test different models over the years. But we have to be honest in saying that we do not really know what is the best way to evaluate a programme that doesn’t have many other close comparators or that is navigating through unchartered waters. The consultants that claim to have the right experience in evaluating these efforts may be stretching the truth a bit. They, maybe, have a hunch or a preference for one or another approach based on their experience. This is fine, by the way. Finding the best possible approach (not even there right one) should be a researchable effort with lots of failures along the way. And who better to find the best approaches than the very same think tanks that the TTI is funding? These centres have economists, political scientists, sociologists, anthropologists, historians, mathematicians, medical doctors, scientists of all kinds. Together they can surely come up with something that would work for all. Maybe the bid could include an inception phase that involves the think tanks in the development of the evaluation framework.
I hope these ideas will encourage more applications. The more the merrier, I always say.
I have already received emails from some researchers and evaluators interested in getting involved. If I hear of anyone who is interested in putting together a bid I’ll pass on your contact info to them; or, if alternatively, you are putting a bid together and would like their contact info, drop me a line.