[This post was originally published on Politics & Ideas in March 2015.]
- There is no doubt that M&E is important and beneficial to organizations and their development;
- Organizations and their staff should pass through M&E trainings (the more, the better);
- To influence practices, use theories which are applicable or can be adapted without damaging their core concepts;
- Any theory applied into practice should be adapted to the context of a particular organization and to the context of the particular environment the organization operates in.
To illustrate these, here are some examples from the Center for Innovations in Education (CIE).
MEL system
The CIE has had many M&E studies conducted both internally and externally. External MEL has only been conducted when funding has been available to hire external contractors. Internally, MEL was used to meet monitoring indicators provided by donors or to provide data for the CIE annual reports. Unfortunately, MEL has never been a formal part of the organization’s ongoing operations. It was considered time consuming, needing a lot of staff and time commitment. In general, MEL often does not seem to be affordable for small organizations.
Until the right formula for M&E is found, organizations like ours will be balancing between donor demands and the daily routines/realities of the organization where everyone is performing several roles and everyone is under high workload. The course offered a practical and feasible way of addressing MEL challenge for small organizations.
Can we afford a formal, well-structured M&E system, built on the best applicable theories? My thoughts are that no, we cannot. We simply do not have sufficient resources to afford that. Can we try to build that system into our organizational practice and budget? Yes. Will it be sustainable? I am not sure. This will require more fundraising, more resource allocation, and it will make our services more expensive. It will also increase the pressure to maintain it.
Several of the “lessons” we learned during the MEL course, like setting up our own “notions of success and failure” and our own standards to measure ourselves, were important to us. We have come to understand that by being a part of the organizational conversation, discussion and reflection, MEL becomes implicitly and informally embedded into the organizational life. This is probably a good solution for organizations like ours.
Policy Unit’s Influence: The Story on Success and Failure
For the Policy Unit (PU), assessing what success is or not in terms of policy influence is more important today than establishing an objective measurement of what we have or have not achieved. Here’s an example:
The PU policy paper on private-public partnership in Early Childhood Education (ECE) has justified a need for promoting community-based alternative preschool service provision and shared results of successful pilots in three community-based models. One of the alternative models presented the provision of paid services by local municipalities. The low cost of the service made them affordable for the majority of community members. This PU policy paper was disseminated using various channels, including round table presentations and discussions with policy makers; mass media; and a presentation at the annual Education Research Forum. It also was used as a case study for learning activities during training on education policy planning conducted by the PU staff.
Young researchers at the PU were upset with the lack of feedback provided by policy makers on the PU’s policy analysis and recommendations. Another concern was related to the overall policy influence of the CIE/PU’s research and policy recommendations. The PU’s research studies imply significant funding, hard team work and advocacy efforts: a big investment is made in policy papers. However, we first perceived that their influence at policy level was either not significant or even opposite to what was expected.
After the PPP on the ECE policy paper was released, the Ministry of Education started piloting paid ECE services provided by public general schools. But the high cost of these services has made them affordable for only a small minority. Is that influence? Yes. Is that kind of influence researchers were looking for? Definitely, not. Is that success or failure?
At first we thought it was a failure, but the MEL course changed our perception. It has helped us revisit our notions of success and failure by conducting a critical analysis of what “good” policy influence is for us. We exercised an internal discussion on possible reasons for the “specific” influence at policy level (instead of “low” or “false” as were thinking initially) and reviewed some assumptions. One finding was that CIE/PU did not deliver the right message to the right audience- we need to better understand our stakeholders.
Another finding was that due to the complexity of the context we operate in, it would be a purely theoretical assumption that if we deliver the right message to the right audience the expected policy influence would take place. The problem is that we can hardly deliver our message to the right audience. In fact, those who attend the PU’s events and those who participate in our discussions are not those who make the decisions we want to see. This means that we intend to influence policy but that it is actually is beyond our control or maybe beyond our current reach. What to do? Should we give up? Why not start another research on the long term social and political consequences of access to ECE being limited to only the better off minority population? Why not continue educating policy makers? Why not to revisit our advocacy and awareness campaign?
Even if the ideas resulting from this internal discussion cannot be implemented immediately (taking into account available funding and an approved research agenda) the exercise provided an opportunity to transform our initially perceived failure at policy influence, into an opportunity to learn and act further.
Advantage of MEL
Based on our organizational experience dealing with MEL and our experience implementing the MEL tools we gained, I see M&E not only as a system to measure our success or failure in terms of influence policy. M&E is not a system to meet donor demands (we can simple use project generated data). The biggest advantage and benefit of MEL as an exercise and as a system is that it proposes a way to engage an organization and people into the joint reflection; thinking and rethinking processes. M&E becomes a part of the team conversation, and that is success!