To identify entry points for your work with policymakers, you’ll first need to understand the political context for the acquisition and use of evidence. Doing this requires an understanding of five things: 1) the issue or problem being discussed 2) the policy context 3) evidence processes 4) types of evidence that are used and 5) motivations to draw on and use evidence.
Work on an institution or an issue?
Before you proceed though, you’ll need to agree on your unit of analysis. Are you exploring the use of evidence in an institution or on an issue? If the former, you could take a look at this ODI working paper, which proposes a framework for understanding the organisational context for evidence informed policymaking. It brings together four evidence related processes (see below) and three elements for understanding the institutional context within which policymaking takes place. The framework was used to help illuminate systems and practices for using evidence in relation to diagnosing, developing, implementing and reporting on policy within South Africa’s Department for Environmental Affairs.
What are people talking about?
Let’s assume you’re exploring the use of evidence on an issue. Evidence is generally used as a resource by actors as they engage in policy debate. But what is a policy debate? It’s an exchange of argument and counter argument, involving any number of actors, in relation to a contested issue and involves any aspect of policy including the policy problem, policy options, means of implementation and monitoring and evaluation. Perhaps the issue is one where poor outcomes persist, despite the existence of a significant body of knowledge in country about how to alleviate constraints. For example, this study focussing on Indonesia explored the use of knowledge in the debate around the poor quality of research and teaching in the majority of the country’s universities as well as the debate around poor practices within the civil service.
How does policy debate unfold? Who has a stake?
Understanding ‘evidence use practices’ first requires understanding how policy processes unfold. There’s no isolating one from another. And policy is what policy does. You’ll want to focus on the delivery of policy at sub-national levels and not just elite-led actions and processes in capital cities. You’ll need to make decisions about which and how many sub-national spaces to focus on. And you may want to identify cases to study in detail, such as the development and delivery of specific programmes.
Policy is of course political. So you’ll want to draw on approaches which suggest that policy emerges from the interaction of different actors and their interests, relationships of power and their evolution over time. Moreover, policymakers operate in a complex environment, where they often have limited control. They share power with other actors. For instance, senior policymakers in the UK do not yield the power that they are expected to have. So you’ll need to consider the role of big business, donor and international agencies, civil society, the media, political parties and other stakeholders in your analysis. They all use evidence in some guise to inform their work.
Within government, there may well be different types of agencies. In Indonesia, this study identified spending ministries (including ministries of Finance, Trade, Health and Education) which were well resourced and often had the authority to deliver policy directly and influencing ministries (such as ministries of planning, women’s empowerment and child protection and law and human rights), which usually had to coordinate and influence other ministries to ‘get things done’. These dynamics shaped their evidence practices differently. Moreover, single agencies or actors are far from homogenous: they are often made up of multiple sub-groups up and down the hierarchy each with their own dynamics. Informed by these interests, policy actors come to discussions with a particular ‘ask’ and possess varying capacities, understandings, motivations and commitment. Importantly, only some of these may be made explicit.
But policy actors are unlikely to be rational utility maximisers who respond mechanically to political and economic interests. In some cases, actors may not even know what their own interests are, especially in highly complex and uncertain contexts. And actors’ interests will not only feature the economic and political, but also social, cultural and ethical/moral. They will be complex, multifaced and constantly evolving. Understanding actors’ use of evidence will require appreciating their psychology. Drawing on bounded rationality, policymakers will combine cognitive shortcuts to prioritise and interpret a small number of problems while ignoring the rest and they’ll find evidence for what they feel are the ‘right’ kind of solutions. You’ll need to explore how policymakers, in effect, ignore most information so they can make timely decisions.
You’ll also need to recognise and explore key elements of the policy environment which constrain and enable actors’ ability to deliberate and make choices. These include:
- The institutions or ‘rules of the game’ that, for instance, establish the spaces where policy decisions are made, how actors interact with one another and the type of evidence that is considered. This includes regular planning, budgeting and reporting processes across government
- Policy networks, which determine who is consulted and whose evidence is considered;
- Ideas or paradigms that shape ways of thinking about an issue, based on beliefs, values and ideology. These provide the context for considering new evidence.
- Contextual features such as a political system’s history, geography, demographic profile, economy, mass attitudes and behaviour, and technological changes which policy actors take into account when making policy. For instance, during apartheid in South Africa, science councils were directed to produce evidence which justified exclusionary policies. Although apartheid was abolished, these factors are still working its way through the system: some mistrust remains between researchers and policymakers.
- Events or critical junctures (either routine or unanticipated) which can influence lurches of policymakers’ attention from one issue to another, constrain or enable specific policy solutions and create or close down windows of opportunity.
What does ‘using evidence’ mean in practice?
To use evidence is to consider a broad range of evidence types (which we discuss below) as part of a process that considers other factors such as political realities and current public debates: “dismissal [of evidence] is uptake, too”. So, evidence related practices include more than just the application of evidence during decision making but also:
- Framing the issue and scoping the question;
- Assembling existing evidence;
- Procuring new evidence as necessary; and
- Interpreting the evidence to inform decisions.
You’ll want to understand how each of these processes play out amongst different policy actors or policymaking bodies. Procurement is often a thorny issue. In Indonesia procurement rules prevent policymakers from hiring top-end researchers from outside government to undertake research, whilst in South Africa, universities are often prevented from supplying government with research as they are not able to provide tax clearance certificates or the lowest quotes.
What types of evidence do policy actors draw on?
You’ll need to explore how policymakers think about evidence. Some policymakers may maintain a hierarchy of evidence. For instance, research exploring the factors shaping sustainable development policy in South Africa revealed that relevant institution tended to be staffed with scientific experts and/or economics who prioritised (natural) scientific information (such as levels of air pollution) over the lived realities of people affected by problems. Many policymakers have, however, a more eclectic view of policy and are more interested in its relevance to policy problems. For instance, in Indonesia, I found that policymakers drew on the following forms of evidence (in order of preference): 1) administrative and statistical data 2) Research studies; 3) Expert advice: 4) Citizen experiences and perceptions: and 5) Policy implementation experiences and learning. You’ll also want to explore whether policymakers on an issue prefer natural over social sciences and quantitative over qualitative.
What motivates actors to consider evidence in their work?
Where you’re targeting instrumental use of evidence, you’ll find it useful to understand what factors motivate policy actors to draw on and use evidence (linked to their interests discussed above). For instance, policymakers in Indonesia are often motivated to demand and/or use evidence by factors that are informed primarily by economic or monetary metrics, an assessment of power gained or lost, the need to defend or legitimise a decision, bolster one’s status as well as maintain ties of reciprocity. But they also use evidence for technical purposes: to provide context (about policy problems), inform strategy, monitor and evaluate policy delivery, inform engagement with others and highlight good practice.
Evidence can also be used to reduce ambiguity, reduce uncertainty about an issue, establish a dominant way to frame a problem and clarify the aims of policymakers and shapers. Policymakers may draw on evidence because they want to do their job well and believe that doing so was a good thing. In some cases, policymakers draw on evidence to comply with bureaucratic requirements (to report on activities and not necessarily outcomes). But equally you might find that policymakers have no reason to draw on ‘evidence’. In South Africa over a third of those interviewed for a study could not think of an example of how evidence could be used to improve policymaking.
A final word
The context will inevitably change. Government and agencies within it may be re-organised, some may be abolished while others might be created. Relationships between people and groups may wax and wane. So, you’ll need to ask the same questions again and again at regular intervals and ensure the findings inform revisions to your strategy. Understanding the factors that influence how evidence is used at individual and team levels takes a lot longer than a study at the outset of a project allows: it requires continuous learning.
But how do you put this framework into action. The next post highlights key lessons from conducting evidence studies in a number of contexts.