Previous
Providing external advisory support to governments is different to most other approaches to influencing change – such as direct implementation, working with civil society, research, or advocacy and campaigning.
As demonstrated in OTT’s recent review of donor models for government engagements, little is documented around how to effectively provide successful advisory support to governments, so establishing a solid evidence base of best practices is key.
In this article, we’ll explore six ways that philanthropic organisations can tailor their monitoring, evaluation and learning (MEL) for government engagement, drawing on our learning partnership and the team’s experience in working on the MEL of evidence-informed policy influencing.
Six ways to tailor MEL for government engagement
1. Gauge the level of complexity
Working with governments to bring about change is usually a complex endeavour. And the approaches to planning and understanding results must be different to those used when you’re implementing a programme yourself.
A complex problem is distinct from a simple or complicated one.
In a simple problem, the relationship between cause and effect is known and predictable: you do X and you will always get Y result – think of administering a vaccine.
Complicated problems have knowable solutions but will usually require experts to figure them out – for example, the logistics around vaccine cold storage and distribution.
Complex problems have high levels of uncertainty, where we either don’t know the solutions or disagree on what they could be – when the relationship between cause and effect is only knowable in retrospect. For example, when rolling out a vaccine promotion programme.
Therefore, complex problems need different approaches to MEL. For example, we need to consider incremental change at multiple levels, focusing particularly on the behaviour of/ relationships among multiple actors.
It may not be possible to define success in advance and MEL will involve a negotiation between different, perhaps competing, ideas of success. It’s likely that there won’t be standard indicators or measurement methods to assess progress – in fact, using such methods may even impede progress.
2. Clarify an actor-centred theory of change
A theory of change for government advisory work tells us how our actions will contribute to the change we would like to see.
The more complex the problem, the more useful it will be to take a systemic approach to the theory of change. This means clarifying the roles of different actors in the system and the relationships among them, considering how these could change for the better.
These actors can include different branches of government, politicians, civil servants, outside experts and activists. It’s helpful to consider who you can work with, support or influence directly and who’s outside of your influence.
The level of detail in the theory of change will depend on the level of uncertainty.
In highly unpredictable contexts, it should be kept simple, with broad pathways of change and only the ultimate outcomes defined. This is because the situation will most likely unfold differently because of factors outside of our control.
We can then observe and document the actual outcomes during implementation and adapt interventions as they go.
3. Define outcomes as ‘changes in behaviour’
When it’s possible to determine the intended outcomes in advance, we define them in terms of behaviour change – such as the actions, activities, relationships or policies of the actors in the system.
This is because behaviour can be directly observed during implementation; therefore, we can collect monitoring data without having to pause implementation to run a survey or use technical measurement tools to monitor quantitative indicators.
It also enables us to monitor the evolution of the partnership itself, which can be broken down into small behaviour changes – such as participation in activities, allocation of resources to support the partnership, and inviting other stakeholders to participate.
Finally, this allows us to define and monitor undesirable behaviours, such as conflict between actors, which can help normalise the reporting of negative outcomes.
4. Document and share results through qualitative approaches
Documenting and sharing the results of government advisory work is a crucial element of learning, both within the partnership and with other colleagues and collaborators.
Qualitative approaches are best suited to capture the complexity and nuance of this kind of work.
Stories of change (or outcome cases for some) document a change process, providing an explanation of the causes/effects of the change and of how the interventions contributed to it.
Approaches such as ‘most significant change’, ‘outcome harvesting’, ‘contribution analysis’ and ‘qualitative comparative analysis’ all centre around narrative stories of change.
These stories are a natural means of engaging diverse audiences and bringing MEL ‘to life’ by connecting the data with what matters to people.
The process of developing a story of change is valuable as a monitoring exercise as it encourages those involved to collect and analyse data, and to reflect on what’s working/not working and how.
5. Monitor and adapt to changes in context
Political, economic, social, technological or organisational factors beyond our control can affect the outcomes of an intervention – positively or negatively. By monitoring changes in the context, we can adapt our interventions to take advantage of potential benefits or mitigate potential risks.
Context monitoring can also help us to understand the contribution of an intervention to a particular outcome, given that multiple factors are likely to have played a role.
For example, was the lack of outcomes for a new economic policy due to deficiencies in the technical advice given to the Ministry of Finance or due to unexpected exchange rate fluctuations?
Assessing context is often done intuitively by programme managers working in a sector. However, whenever possible, it’s useful to do it systematically, at regular intervals, and using comparisons – for instance, at different project sites, over time or by comparing to other sectors.
Setting learning questions about the context can help structure this, for example: Who else is working in the same policy space and what are their agendas and motivations? How are power structures changing and what might this mean for the partnership?
6. Integrate MEL into partnerships
In government engagement and advisory work, MEL will require the participation of government partners. The level of participation and the specific roles and responsibilities of each party will depend on the context and will need to be negotiated.
As a general principle, as the providers of advisory services, we’ll be responsible for monitoring and evaluating our outcomes – such as, how the behaviour of the actors within our influence is changing. (These actors are usually the government partners.)
Monitoring changes beyond our influence – for example, around the implementation of government programmes and policies – requires government partners to cooperate with us, as we may not have access to data or programme participants.
Government partners may or may not have their own monitoring and evaluation systems and may not have the capacity in place. And even if they do, they may not be aligned with our approach.
Given this, it’s advisable to ensure a good understanding of each other’s MEL approaches and to agree on mutual roles and responsibilities, which cover core tasks such as data collection, management, analysis, sense-making and use.
Clear communication is important in this process to build ownership and incentives to participate in MEL. There may also be a need for capacity strengthening, including resourcing for MEL staff.