{"id":862,"date":"2014-08-11T21:12:25","date_gmt":"2014-08-11T21:12:25","guid":{"rendered":"https:\/\/onthinktanks.org\/?p=862"},"modified":"2016-01-06T18:18:17","modified_gmt":"2016-01-06T18:18:17","slug":"think-tank-initiative-evaluation-some-ideas-for-the-evaluators","status":"publish","type":"post","link":"https:\/\/onthinktanks.org\/articles\/think-tank-initiative-evaluation-some-ideas-for-the-evaluators\/","title":{"rendered":"Think Tank Initiative Evaluation: some ideas for the evaluators"},"content":{"rendered":"

The Think Tank Initiative has launched a call to recruit an evaluation team for its second phase. You can find\u00a0details on the Request for Proposal and how to apply on their website<\/a>.<\/p>\n

Not long ago,\u00a0Peter Taylor, the head of the initiative, wrote about the\u00a0lessons learned from the first phase evaluation<\/a>, thus providing some good ideas about what to look for in developing the new proposals for this project.<\/p>\n

With the objective of encouraging new players in this sector (particularly from developing countries) to apply or to play greater roles in the evaluation teams, let me offer some ideas of my own. These are in no way to be taken as binding or to have any significance to the TTI selection process. They will evaluate the proposals on their own merits according to their own criteria.<\/p>\n

But here is my wish-list of things I\u2019d like to\u00a0see in the proposals:<\/p>\n

    \n
  1. It would be great is more bids, and the winning bid, were from or involved evaluators from\u00a0developing countrie<\/strong>s in leading roles. It would be particularly exiting if the\u00a0bids were from or included organisations in the countries where the TTI is working<\/strong>. This is not necessarily about quality (although it could be about cost); there is no reason why European\u00a0or US based organisations wouldn\u2019t have the technical skills for this task. But in my view the previous evaluation lacked a deep knowledge of the policy research communities that only belonging to those communities gives you. I do not think a few days (even a few days very year) in each country for a few interviews will do. the budget would have to consider large stints in each country, but even then it may not be enough. I have argued this many times (including in a blog on \u2018tourist funders<\/a>\u2018). Maybe an option would be to find leading evaluators in each region\u00a0and bring them together in a network type of approach. TTI will have to consider its budget for the evaluation if it wants to delve deep into each region and country where it has been working; and bidders will need to look harder for leading evaluators in each of the regions.<\/li>\n
  2. New faces<\/strong>: It would be fantastic if the evaluators were not \u2018part of the evidence based\/informed policy\u00a0in development<\/em>\u2018 industry. I find that there are too many vested interests in the sector already. Can one be openly critical about a funder when one depends on it for future work (a lot of future work)? Can one be critical of certain think tanks when one is also engaging with them as partners in other projects? Can one be neutral about what works and what doesn\u2019t when one has one\u2019s\u00a0sights on opportunities to build capacity on what one knows more about? There are no reasons why professional evaluators working in other sectors could not do this. Competent people can learn about new contexts and, lets face it, this is not rocket science.<\/li>\n
  3. A good alternative would be to look at\u00a0academics,\u00a0consultancies or think tanks in developed or non-TTI countries<\/strong>\u00a0(working on domestic\/foreign policy issues -so not international development only) who would be able to\u00a0bring their own experience of their think tank or evidence based policy communities into the evaluation<\/a>. I would hope to see applications from Mexico, Argentina, Colombia, South Africa, india (it is big enough), Indonesia, Singapore, etc. as well as from Canada, the US, the UK, Europe, etc.<\/li>\n
  4. But if this is not possible (reaching out to other sectors in the Aid industry is very hard), then it would be amazing if the evaluators were at least\u00a0open about their other responsibilities and commitments and conflicts of interest<\/strong>. Constant publication of progress (including the data and their analyses) would go a long way.<\/li>\n
  5. It would also be great if the evaluation itself awarded a much\u00a0greater emphasis to the effect of the TTI on the broader think tank communities of each of the countries in which it has intervened<\/strong>. This is important because the TTI did not set out\u00a0to support just the think tanks it funds (this is what I understand). If it did, then it would have had to take into account that choosing which think tanks to support was an inherently political (and ideological) action.\u00a0Funding has given more power to a set of political actors (a think tank and its networks) over others (other think tanks and their networks) in a the country. Therefore, understanding the nature of the think tank communities (the political and academic players, as well as other intellectuals, ideologues, parties, interest groups, etc) before the TTI arrived, how it has changed (for good or worse,) and what direction it seems to be taking is crucial. This is why local evaluators should\u00a0be in leading roles.<\/li>\n
  6. As a follow-up, then, it would be great is the\u00a0questions focused on whether the TTI has helped or contributed in developing a more sustainable policy research community<\/strong>\u00a0in the countries where it has worked. In my view the TTI and the evaluators should assume that this is the second and last phase. IDRC has been funding many of these think tanks (before and during the TTI) for decades. How much longer can it keep doing so? Surely it is time for domestic funders to step up to the plate. Success then does not necessarily mean that all the TTI think tanks will make it through post TTI. What matters is that the community, who ever it includes, is strong and sustainable.<\/li>\n
  7. It would be\u00a0a shame if the evaluation focused too much on instances or cases of policy influence<\/strong>. There is a lot more to think tanks and the contributions\u00a0they make to their countries and communities than plain old direct influence. Their roles include educating the public, preparing new generations of policymakers, informing and strengthening academic practice, creating and maintaining spaces for debate, challenging the status quo (even if unsuccessfully), etc.\u00a0Being influential depends little on the think tanks and greatly on other factors and actors.<\/a><\/li>\n
  8. It would also be a\u00a0shame if the evaluation focused too much on metrics of \u2018having\u2019 new skills or resources<\/strong>. What counts is not that now a think tank has a communications officer or more funding from other sources. What counts is that now the think tank has the capacity to decide when it needs them, how to get them, and is able to if it wants to. \u00a0And this is much about the think tanks capabilities as\u00a0about their context<\/a>.<\/li>\n
  9. The evaluators should not see themselves as\u00a0the only ones in\u00a0this role. The\u00a0TTI and the evaluators should adopt an\u00a0Open Data Policy<\/a><\/strong>\u00a0(that its funders already support) and share all the data is has on the think tanks it supports around the world (the\u00a0think tanks should be doing it anyway<\/a>). This would make it possible for others to contribute with fresh analysis and insights.<\/li>\n
  10. Finally, it would be just\u00a0perfect if the evaluation approach was in part (or greatly) informed by the think tanks themselves<\/strong>. This is not an easy thing to do. TTI has worked hard to find and test different models over the years. But we have to be honest in saying that we do not really know what is the best way to evaluate a programme that doesn\u2019t have many other close comparators or that is navigating through unchartered waters. The consultants that claim to have the right experience in evaluating\u00a0these efforts may be stretching the truth a bit. They, maybe, have a hunch or a preference for one or another approach based on their experience. This is fine, by the way. Finding the best possible approach (not even there right one) should be a researchable effort with lots of failures along the way.\u00a0And who better to find the best approaches than the very same think tanks that the TTI is funding?<\/strong>\u00a0These centres have economists, political scientists, sociologists, anthropologists, historians, mathematicians, medical doctors, scientists of all kinds. Together they can surely come up with something that would work for all. Maybe the bid could include an inception phase that involves the think tanks in the development of the evaluation framework.<\/li>\n<\/ol>\n

    I hope these ideas will encourage more applications. The more the merrier, I always say.<\/p>\n

    I have already received emails from some researchers and evaluators interested in getting involved. If I hear of anyone who is interested in putting together a bid I\u2019ll pass on your contact info to them; or, if alternatively, you are putting a bid together and would like their contact info, drop me a line.<\/p>\n","protected":false},"excerpt":{"rendered":"

    The call for proposals for the second phase of the Think Tank Initiative presents an opportunity to reflect on what such an evaluation should entail. This ‘wish-list’ brings together some of the ideas presented in this blog over the years in an effort to encourage more and better applications.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","footnotes":""},"tags":[182,189],"acf":[],"_links":{"self":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/862"}],"collection":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/comments?post=862"}],"version-history":[{"count":0,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/862\/revisions"}],"wp:attachment":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/media?parent=862"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/tags?post=862"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}