{"id":1466,"date":"2013-06-24T13:48:08","date_gmt":"2013-06-24T18:48:08","guid":{"rendered":"https:\/\/onthinktanks.org\/articles\/\/"},"modified":"2016-01-21T13:51:32","modified_gmt":"2016-01-21T18:51:32","slug":"monitoring-and-evaluation-lessons-from-latin-american-think-tanks","status":"publish","type":"post","link":"https:\/\/onthinktanks.org\/articles\/monitoring-and-evaluation-lessons-from-latin-american-think-tanks\/","title":{"rendered":"Monitoring and evaluation: Lessons from Latin American think tanks"},"content":{"rendered":"<p>The\u00a0<a href=\"http:\/\/thinktankinitiative.org\" target=\"_blank\">Think Tank Initiative<\/a>\u00a0organised, back in April, a workshop on Monitoring and Evaluation for Performance and Impact in Lima. It was an interesting meeting, out of which came a few (I hope) useful blog posts:<\/p>\n<ul>\n<li><a href=\"https:\/\/onthinktanks.org\/2013\/04\/15\/a-cheaper-and-more-useful-alternative-to-an-roi-for-a-think-tank-force-field-analysis\/\" target=\"_blank\">A cheaper and more useful alternative to ROI for a think tank: Force Field\u00a0Analysis<\/a><\/li>\n<li><a href=\"https:\/\/onthinktanks.org\/2013\/04\/04\/who-is-responsible-for-a-think-tanks-influence\/\">Who is responsible for a think tank\u2019s influence?<\/a><\/li>\n<li><a href=\"https:\/\/onthinktanks.org\/2013\/04\/19\/strategic-plans-a-simple-version\/\">Strategic plans: a simple version<\/a><\/li>\n<\/ul>\n<p>An internal report was prepared by the facilitator,\u00a0<a href=\"http:\/\/english.iifac.org\/about-iifac\/beatrice-briggs\/\" target=\"_blank\">Beatrice Briggs<\/a>, and with her permission and that of the TTI&#8217;s, I have picked out a few interesting bits of information that I think are important to share. I&#8217;ve made an effort to keep things entirely anonymous;\u00a0<a href=\"http:\/\/www.chathamhouse.org\/about-us\/chathamhouserule\" target=\"_blank\">Chatham House Rule<\/a>.<\/p>\n<p>In preparation for the event, the think tanks were asked to answer a series of questions. The first one related to their M&amp;E strengths -or what they were doing particularly well; the second to the challenges they sill faced -or what they were not doing particularly well; and the third refers to the kind of information that they were collecting. There were, of course, other questions but these are the ones I&#8217;ll be sharing in this post.<\/p>\n<h2>What were the think tanks already doing?<\/h2>\n<p>I have arrange some of the statements provided by the think tanks according to four categories: planning, systems and tools, learning and feedback, and human resources. A surprising finding from the process (and the workshop itself) was that most of the think tanks had a person (or more) dedicated to M&amp;E. This is not something I often come across before. Should there be a dedicated person or is this something that everyone in the organisation should do. I feel that this responds more to pressure from funders for accountability purposes than a genuine interest for learning by the think tanks. But, let&#8217;s see.<\/p>\n<p>Many readers will find the following statements rather typical or at least familiar to them and their organisations.<\/p>\n<p>Planning:<\/p>\n<ul>\n<li>We have an internal monitoring strategy and achieve quite effectively the implementation of the monitoring instruments designed.Human resources<\/li>\n<li>Theory of change and defined strategic plan<\/li>\n<li>Have a Project Approval Committee that allows us to take the first step in the monitoring of projects<\/li>\n<li>The organization&#8217;s work is based on a five-year strategic plan, to the elaboration of which the entire team of the organization participated, giving everyone an important stake in its success. Every year, the planning is evaluated against the new context, national and international, and each area carries out operational planning. This allows the each program to be completely aware of the objectives of its program for the year, and for the longer-term, and therefore gives great unity to the monitoring and evaluation processes.<\/li>\n<\/ul>\n<p>Systems and tools:<\/p>\n<ul>\n<li>Incorporation and use of tools that permit us to monitor both the activities in which oe an internal monitoring strategy and achieve quite effectively the implementation of the monitoring instruments designed.<\/li>\n<li>We conduct a semi-annual monitoring report that identifies strengths and weaknesses, and includes general and specific recommendations for each research team.<\/li>\n<li>We perform quarterly and annual reports on the institutional political incidence based on each research team<\/li>\n<li>We have certain tools for registering and collecting information: activities bulletin, media scan, surveys, monitoring of the performance of the communication channels (web page, social media), etc.<\/li>\n<li>The lessons learned in the course of more than 30 years of institutional life, a period in which we have passed through different stages and implemented different strategies, have allowed us to define a clear institutional mission<\/li>\n<\/ul>\n<div title=\"Page 5\">\n<div title=\"Page 5\">\n<div title=\"Page 5\">\n<div title=\"Page 5\">\n<div title=\"Page 5\">\n<div title=\"Page 5\">\n<p>Learning and reporting back:<\/p>\n<div title=\"Page 6\">\n<ul>\n<li>We perform quarterly and annual reports on the institutional political incidence based on each research team<\/li>\n<li>The organisation&#8217;s work is based on a five-year strategic plan, to the elaboration of which the entire team of the organisation participated, giving everyone an important stake in its success. Every year, the planning is evaluated against the new context, national and international, and each area carries out operational planning. This allows the each program to be completely aware of the objectives of its program for the year, and for the longer-term, and therefore gives great unity to the monitoring and evaluation processes.<\/li>\n<li>The lessons learned in the course of more than 30 years of institutional life, a period in which we have passed through different stages and implemented different strategies, have allowed us to define a clear institutional mission.<\/li>\n<\/ul>\n<div title=\"Page 6\">\n<p>Human resources:<\/p>\n<ul>\n<li>Our Institutional Communications Unit conducts monitoring of the media.<\/li>\n<li>Each department and program monitors the achievement of its activities, projects, budgets, etc.<\/li>\n<li>M&amp;E work at the institutional level is being implemented, little by little by different areas\/departments and has personnel in charge of this. We have consultancies\/studies that will permit us to orient the future M&amp;E system in communications.<\/li>\n<\/ul>\n<div title=\"Page 7\">\n<div title=\"Page 7\">\n<div title=\"Page 8\">\n<div title=\"Page 8\">\n<h2>What were the think tanks main challenges or things that were not working so well?<\/h2>\n<p>Overall, my sense was that most of the think tanks had gone a long way in thinking about M&amp;E and establishing the basic building blocks for it. Where they seem to struggle is on how to institutionalise these small yet important steps.<\/p>\n<p>I have organised these statements according to four broad categories: general challenges, what is next?, barriers, and resources (as before, some are repeated):<\/p>\n<h3>1) General challenges and things that are missing:<\/h3>\n<p style=\"padding-left: 30px;\">This is interesting as some of the challenges refer to the nature of influence itself but most (and I have emphasised this here) focus on methodological and organisational challenges:<\/p>\n<p style=\"padding-left: 30px;\">Nature of influence:<\/p>\n<ul>\n<li>The relatively [long and uncertain] lapse of influence often lessens\/affects the daily lobbying work with certain decision makers<\/li>\n<\/ul>\n<p style=\"padding-left: 30px;\">Methodological challenges:<\/p>\n<ul>\n<li>We still have not succeeded in developing indicators of the impact of our research in public policies that monitor the consideration and application of the recommendations of our studies among the policy makers, how much impact they have, the extent of the impact, etc.<\/li>\n<li>We lack the development of tools to measure and understand better the impact of research in 1) public policy influence 2) at regional reach 3) international reach.<\/li>\n<\/ul>\n<p style=\"padding-left: 30px;\">Organisational challenges:<\/p>\n<ul>\n<li>Do not align the objectives of particular projects with the strategic planning<\/li>\n<li>We do not have a formal M&amp;E process (and tools) at the overall management level of the organization, and even less one focused on our public policy influence<\/li>\n<li>Lack of articulation between monitoring of media, of policy influence and or activities and products.<\/li>\n<li>The result of the lack of role definition means that tacks can be duplicated. By the same token, the lack of a better institutional M&amp;E plan and a strategic timeline for evaluation means that the agreed upon deadlines are not met.<\/li>\n<li>Absence of systematic means of communicating to our Communication Dept. the advances and achievements in the different areas<\/li>\n<li>We lack tools for systematic monitoring of public policy influence and public opinion<\/li>\n<li>No formal mechanism for sharing the monitoring results of what the departments and programs are currently doing<\/li>\n<\/ul>\n<h3>2) What is next?<\/h3>\n<ul>\n<li>These mechanisms are not formally established<\/li>\n<li>Continuous updating of maps of actors, not just a list of actors<\/li>\n<li>Need to monitor most recent statements, actions of key actors and institutional policies to identify more precisely the contribution (political incidence) of \u00a0[the think tank] in such aspects.<\/li>\n<li>Improvements could be made in the way some of our programs are evaluated in order to make the impact of our action clearer both internally and externally.<\/li>\n<li>Need to reinforce this with mechanism that allow us to monitor and evaluate our influence as well as the fulfillment of the strategic objectives that we set<\/li>\n<li>We still have not succeeded in developing indicators of the impact of our research in public policies that monitor the consideration and application of the recommendations of our studies among the policy makers, how much impact they have, the extent of the impact, etc.<\/li>\n<\/ul>\n<h3>3) Barriers:<\/h3>\n<p style=\"padding-left: 30px;\">Some of the barriers refer to the role of individuals while others to absence of key organisational &#8216;institutions&#8217;.<\/p>\n<p style=\"padding-left: 30px;\">People:<\/p>\n<ul>\n<li>Weak culture of obligatory accountability in our environment that also is evident in our daily activities.<\/li>\n<li>There is a lack of interest among certain researchers in cooperating with the work of M&amp;E at the institutional level and, especially in providing information in a timely manner<\/li>\n<\/ul>\n<p style=\"padding-left: 30px;\">&#8216;Institutions&#8217;:<\/p>\n<ul>\n<li>Do not have automated processes that allow us to document and follow up on our projects<\/li>\n<li>Do not have adequate follow up of the implementation of suggestions made in the consultation mentioned<\/li>\n<li>We do not have a structured system related to the strategic objectives<\/li>\n<\/ul>\n<h3>4) Resources (or lack of):<\/h3>\n<div title=\"Page 5\">\n<ul>\n<li>We do not have a person dedicated full-time to M&amp;E activities<\/li>\n<li>No one in charge of coordinating M&amp;E<\/li>\n<li>We have no experience with the design and implementation of a formal M&amp;E system<\/li>\n<li>We lack a professional or officer who leads the implementation of the M&amp;E system.<\/li>\n<li>The tasks of collecting impact indicators are divided between the members of the Information and Communication Unit; nevertheless, we believe that we still lack a clear definition of who should be the person in charge of implementing and M&amp;E strategy or what the specific role is of the members of the unit.<\/li>\n<li>In the communications area we lack tools and staff to carry out a complete M&amp;E job (reaching all the media, monitoring mass mailings, academic activities, among others). By the same token, we lack the implementation of M&amp;E systems in a more professional manner in certain areas: library, bookstore, administration, etc.<\/li>\n<\/ul>\n<p>My own judgement is that a great deal of emphasis has been placed on human resources. I am yet unsure about this issue. Should think tanks, especially small ones, have a dedicated team for M&amp;E? Given that most do not have an adequately staffed communications team, is this the best use of their resources? (The jury is out: I have written something related to this here:\u00a0<a href=\"https:\/\/onthinktanks.org\/2013\/05\/22\/tourist-funders-are-unhelpful-when-supporting-and-evaluating-think-tanks\/\">\u2018Tourist\u2019 funders are unhelpful when supporting and evaluating think\u00a0tanks<\/a>)<\/p>\n<h2>So what information do they collect?<\/h2>\n<p>Of course all these systems and tools that the think tanks have can mean very little unless we think about the kind of information that they are collecting -and for what purpose. The following table was put together by the workshop&#8217;s facilitator and it shows some interesting things:<\/p>\n<ul>\n<li>Project planning, work-plans, etc. are covered;<\/li>\n<li>Budgeting and other financial information, too; and<\/li>\n<li>There is a big inconsistency in relation to impact&#8217; information, however. I think this last point is important. It reflects the difficulty that this issue presents: methodologically and operationally.<\/li>\n<\/ul>\n<p>I say there is an inconsistency because my own conversations with the participants suggested that. In general, think tanks are better equipped in gathering data related to activities they control but, not surprisingly, less so when it comes to assessing their influence.<\/p>\n<p>Another point that is worth highlighting related to information related to budgets and financing. While these Latin American think tanks are strong at recording and reporting on their finances and accounting obligations, they are not, as most think tanks, as strong in their capacity to\u00a0<a title=\"Managing human and financial resources: a Ray Struyk\u00a0masterclass\" href=\"https:\/\/onthinktanks.org\/2013\/03\/04\/managing-human-and-financial-resources-a-ray-struyk-masterclass\/\" target=\"_blank\">monitor the use of the resources by project or activity<\/a>. When I asked them (not all the think tanks present, granted) none could say that they had the right systems and tools to say exactly whether a project was over, under or bang on budget.<\/p>\n<div title=\"Page 9\">\n<table>\n<tbody>\n<tr>\n<td>Kind of information collected<\/td>\n<td>Number of institutions that reported collecting\u00a0this information<\/td>\n<\/tr>\n<tr>\n<td>Work plan and timeline and advances for activities\/projects<\/td>\n<td>\u221a\u221a\u221a\u221a\u221a<img decoding=\"async\" src=\"\/\/\/page9image8688\" alt=\"page9image8688\" width=\"0.480000\" height=\"0.480000\" \/>\u00a0<img decoding=\"async\" src=\"\/\/\/page9image9008\" alt=\"page9image9008\" width=\"0.480000\" height=\"0.480000\" \/><\/td>\n<\/tr>\n<tr>\n<td>Academic activities, internal and external<\/td>\n<td>\u221a\u221a<\/td>\n<\/tr>\n<tr>\n<td>Publications in various formats (including magazines, DDT, policy briefs, etc.)<\/td>\n<td>\n<div>\n<p>\u221a\u221a<\/p>\n<\/div>\n<p><img decoding=\"async\" src=\"\/\/\/page9image14240\" alt=\"page9image14240\" width=\"0.480000\" height=\"0.480000\" \/><img decoding=\"async\" src=\"\/\/\/page9image14560\" alt=\"page9image14560\" width=\"0.480000\" height=\"0.480000\" \/><\/td>\n<\/tr>\n<tr>\n<td>Researchers participation in different meetings, conferences and networks, both national and international<\/td>\n<td>\u221a\u221a<\/td>\n<\/tr>\n<tr>\n<td>Impact indicators for different activities and publications<\/td>\n<td>\u221a\u221a\u221a\u221a\u221a<\/td>\n<\/tr>\n<tr>\n<td>Policy impact indicators regarding results of our studies and research<\/td>\n<td>\u221a\u221a\u221a\u221a\u221a\u221a\u221a<\/td>\n<\/tr>\n<tr>\n<td>Financing, budget<\/td>\n<td>\u221a\u221a\u221a\u221a<\/td>\n<\/tr>\n<tr>\n<td>Research quantity and quality<\/td>\n<td>\u221a\u221a\u221a<\/td>\n<\/tr>\n<tr>\n<td>Policy influence<\/td>\n<td>\u221a<\/td>\n<\/tr>\n<tr>\n<td>Projects to be implemented in the future and their alignment with strategic plan<\/td>\n<td>\u221a<\/td>\n<\/tr>\n<tr>\n<td>Opportunities for improving project implementation<\/td>\n<td>\u221a<\/td>\n<\/tr>\n<tr>\n<td>Hard facts to tell the institutional story and demonstrate our experience in different fields<\/td>\n<td>\u221a<\/td>\n<\/tr>\n<tr>\n<td>Qualitative information about our social impact<\/td>\n<td>\u221a<\/td>\n<\/tr>\n<tr>\n<td>Info about how our advocacy has impacted target populations<\/td>\n<td>\u221a<\/td>\n<\/tr>\n<tr>\n<td>Satisfaction of users, receivers or clients of our services and products<\/td>\n<td>\u221a<\/td>\n<\/tr>\n<tr>\n<td>Information to enrich the strategic plan<\/td>\n<td>\u221a<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Final reflections<\/h2>\n<p>The workshop presented the participants with a range of approaches and tools and the opportunity to share their concerns and ideas. In the end, they had a chance to reflect on what they had learned -or at least what they would take with them. The following are statements as recorded by the organisers and the facilitator:<\/p>\n<p>On why is M&amp;E useful for a think tank:<\/p>\n<div title=\"Page 16\">\n<ul>\n<li>M&amp;E helps to order data.<\/li>\n<li>M&amp;E provides an opportunity to integrate tasks and processes.<\/li>\n<li>Developing a strong commitment to M&amp;E implies generating awareness about the usefulness of these activities.<\/li>\n<li>M&amp;E improves the performance of the researcher, the consultant in all the areas of\u00a0the organization where M&amp;E is implemented. In short, it improves organizational\u00a0management.<\/li>\n<li>M&amp;E can improve organizational performance only if the team is committed to\u00a0doing the work. Knowledge management is very important.<\/li>\n<li>We cannot achieve new results doing what we have always done.<\/li>\n<li>M&amp;E provides information not only about what you are doing but also about what\u00a0could be done, starting with what you are already doing. It is a useful tool for\u00a0looking ahead.<\/li>\n<li>M&amp;E asks \u201cWhat more should we do?<\/li>\n<\/ul>\n<p>On evaluation culture and key roles:<\/p>\n<ul>\n<li>The executive director is doing M&amp;E every day.<\/li>\n<li>Researchers make little attempt to include the culture of evaluation. The institutional challenge is to generate an M&amp;E culture. To achieve this, the leader or leaders of each organization need to be committed with constant M&amp;E in all areas.<\/li>\n<li>Rather that a \u201cstrong\u201d commitment to M&amp;E we should strive for a \u201creasonable\u201d\u00a0commitment.<\/li>\n<li>Above all, it is necessary to create, maintain and develop a culture of evaluation, and find the times to make use of M&amp;E. An evaluative culture is created by competition. This helps people to evaluate or assign value to the projects they are doing.<\/li>\n<\/ul>\n<p>Remaining challenges<\/p>\n<ul>\n<li>We should integrate resource mobilization with M&amp;E and develop them internally.<\/li>\n<li>We need to educate donors about social impact.<\/li>\n<li>The challenge of all\u00a0organizations is to discern the way to apply all this learning.<\/li>\n<li>Organizations must commit to educate the rest about the lessons learned and be capable of transmitting this knowledge.<\/li>\n<li>We must create or rely on systematization that is useful and effective for our organizational purposes. To do this it is important to have incentives. To generate greater commitment and involvement that is not the result of crisis situations or of complex processes, but rather comes from the need to make use of new tools, including a new way of looking more fully at what evaluation can help us accomplish.<\/li>\n<li>A sense of belonging provokes interest.<\/li>\n<li>Sometimes there can be too much M&amp;E.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>The\u00a0Think Tank Initiative\u00a0organised, back in April, a workshop on Monitoring and Evaluation for Performance and Impact in Lima. It was an interesting meeting, out of which came a few (I hope) useful blog posts: A cheaper and more useful alternative to ROI for a think tank: Force Field\u00a0Analysis Who is responsible for a think tank\u2019s [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","footnotes":""},"tags":[194,182,204],"acf":[],"_links":{"self":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/1466"}],"collection":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/comments?post=1466"}],"version-history":[{"count":0,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/1466\/revisions"}],"wp:attachment":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/media?parent=1466"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/tags?post=1466"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}