Challenges and possibilities of monitoring, evaluation and learning in think tanks
Measuring the results of the work of think tanks poses many challenges; the results are often intangible –for instance, building strong relationships with policy makers, or playing a key role in a policy network or debate. Another key challenge is that success is often framed as policy change, but this tends to take a long time, it is hard to attribute this change to a specific organization, and policy makers are unlikely to admit to policy influence.
So we need to think differently about measuring success when it comes to policy influence. The most important thing is to define success – what do the policy objectives look like for each organization. For this it is important to understand where you locate yourselves in the policy process with any particular issue.
Learn more about Monitoring and Evaluation of Policy Research:
On Think Tanks School Monitoring and Evaluation courses
On Think Tanks School Webinar on Monitoring and Evaluating Policy Research
One useful mechanism to ensure clarity and viability of objectives is to break the policy processes in parts (see Figure 1 below). Organizations should not only focus their goals on driving direct policy change but also try to affect what happens before, throughout and after a new policy is designed: for example, to illuminate the way some policy problems are perceived.
Guided by these policy process “parts” it is easier to identify the various strategies that can be deployed to have an effect on each and, consequently, what do look for when evaluating impact. See, for example, the figure below:
We also have to strike a balance accountability requirements and being honest about our achievements. How can we go about this?
- The first step is to be clear about your policy influence objectives and make sure that these are realistic given the work that you do. This will be drawn from your organizational mission, project proposals, strategic discussions and should be informed by research into the problem that you are trying to address. Conducting a situation analysis, mapping exercise, or diagnostic analysis could inform these objectives.
- Secondly, identify your key policy influence strategies and identify the best ways to measure these. For example, most think tanks do one or all of the following research, advocacy, communications, capacity building. This series includes an article on MEL for research and MEL for communications which explores how this can be done
- Lastly, ensure that your organization is well equipped to motivate and manage the staff and resources necessary to achieve good results. This series includes an article on the MEL of governance and management, which if done well should improve performance. The article by Nana Davies of Southern Hemisphere, will provide some insights and guidance in this regard.
This series on MEL for think tanks will provide some insights into these steps. The key themes that are explored within the series regard the MEL of research, communications, governance and management. The series will cover the following topics:
MEL of Research: This article explores how the Research Quality Plus (RQ+) framework, recently developed by IDRC, can be valuable for think tanks to improve their research quality. The article describes the rationale for the new approach, the instrument that has been designed, and includes a discussion of IDRC’s first effort to apply the framework in its External Reviews. In it Zenda Offir highlights how the RQ+ approach can be used for the management of research quality in programs, program portfolios and grant funding portfolios.
Case study of MEL design in a South African think tank: This is a case study of a think tank that describes the process and lessons from designing a MEL system for a project in a well-respected South African think tank. The key message here is that organizations should start to develop MEL systems to suit their existing capacity and expand as they gain confidence and skills.
MEL of Communications: Doing great research is only a part of a think tank’s mission. In order to be effective, think tanks must also communicate their findings to a range of stakeholders. This article provides some insight into how the effectiveness of these communication efforts can be monitored and evaluated, and how organizations can learn about improving their communications in order to achieve greater impact.
MEL of governance and management: Research and communication tend to receive more attention than governance and management when it comes to the study and support of think tanks. Yet, without sound governance and management think tanks cannot deliver their missions. This post provides an overview of issues related to governance and management that think tanks should be paying attention to in their MEL systems, and some methods for doing this.
Join us !
Write a blog post: We invite participation in this series. If you have a contribution – a case study, an experience, a tool or anything you would like to share, write to onthinktanks: [email protected]