The politics of the evidence based policy mantra

17 October 2012

Andries Du Toit’s paper on the politics of research is one of the best studies on the links between research and policy that I have ever read. It is also one of the few coming from a developing country and written from that perspective -and in English which that will help in getting some of the points it makes across (if you can read spanish I suggest you have a look at this book that covers similar issues for Latin America).

This paper takes a different tack. It is not concerned with the theory of poverty measurement. Neither is it concerned with the ‘how to’ of ‘getting’ or ‘communicating’ the ‘evidence’. Rather, it seeks to look critically at some of the assumptions informing these concerns. The paper is part of a broader engagement with the politics of knowledge production in policy-oriented social science research. This engagement is based on a desire to better understand the relationship between two different worlds of practice — that of social science research and scholarship on the one hand, and that of social action, intervention and policymaking on the other. Against this background, this paper focuses on the habits of thought and agendas associated with the currently dominant way of thinking about these relationships — the discourse of ‘Evidence-Based Policymaking’ (EBP).

I read a draft version of the paper earlier in the year and have been meaning to review it for some time. And this is the line that got me interested:

This paper argues that while the desire to ensure that policy making is informed by social science may be laudable, the assumptions underlying these assertions about the role of evidence and science turn out to be dubious, and provide a poor guide to the challenges involved.

Emma Broadbent’s papers on the political economy of research uptake in Africa made many similar points. Not surprisingly, though, her ideas have not been rapidly picked up by the ‘sector’. They challenge the very assumptions that many consultants working on EBP and research communications use to keep their clients/donors happy with promises that it is, in fact, possible to influence policy. The paper is worth reading but here are some highlights and comments from me.

First, he is extremely clear in pointing out that:

  • The evidence based policy narrative is a normative and not a descriptive one: it is about what should be rather than about what really is; and
  • It is a deeply political one: the very idea that policy should be about ‘what works’ and not about values is a political statement -a strategy, in fact, to get rid of political debate and clear the way for unilateral reform. This is what the Blair government did in Britain, but also what the Fujimori government did in Peru. (And it is clearly described by researchers in the study of think tanks and political parties I edited a few years ago.)

There are several objections to this discourse then:

  • That it is potentially anti-democratic: in that it keeps the public out of discussions that, under the EBP discourse can only be had by ‘experts’; those who have the evidence (and not any kind of evidence). This is something seen over and over again in Latin America and Woodrow Wilson warned the American public about the threat to democracy posed by technocrats. In fact, many of the first foreign policy think tanks were set up precisely to keep democracy out of foreign policy!
  • It is also difficult to export and attempt to apply the EBP ideal to all contexts where the institutional infrastructure does not exist -nevermind the calls that ‘context matters’ when the implication of it is to still implement a programme designed in London or Washington and often by consultants with little interest in learning about the intricacies of the societies they seek to engineer.
  • But one of the biggest concerns is that EBP works (better) for clear and well-defined objectives: measurable outcomes. Development, however, is not about these alone but about a much broader challenge: society. Many Aid agencies have been praising Rwanda’s reduction of child mortality but they seem to forget the increase in political repression. The means cannot justify the ends, even if the ends are the MDGs and were reduced using randomised controlled trials.
  • And finally, EBP makes it difficult to think about policymaking in a more nuanced way. It limits, right from the start, what and how we study policymaking and the role of research and researchers. The questions EBP advocates ask in their case studies have to do with finding out why evidence is not more influential -as if it had the right to be so. In my 6 years at RAPID nobody (myself included) answered a question I posed from time to time: if we say that not enough evidence is used… then how much is enough? Once we free ourselves from the EBP discourse (or mantra) we can start to ask more interesting and exploratory questions: how does policymaking happen? Who plays what roles? Why? etc.

In fact, EBP is as ideological as the very approaches to policymaking it seeks to discredit. Du Toit provides an excellent illustration of these points using the case of South Africa. He concludes that the claims that we need EBP to move past ideology and guesswork is:

an exaggerated claim and misrepresents the issues. EBP discourse involves a narrow and technicist understanding of what is involved in policy making; it has a naïve empiricist view of the role of evidence in social science; and it misunderstands the importance of politically and ideologically loaded ‘policy narratives’ in policy change, even in situations where these policy debates do involve appeals to ‘evidence’ and research findings.

This is why I am sceptical of claims by many that the recognise that ‘other factors affect policymaking’. Sure, they do, but they wish they didn’t. I know because this is exactly how I used to work and what was behind the development of frameworks and tools like the RAPID Outcome Mapping Approach.

Du Toit does not dismiss the positive role that science (and the evidence it produces) can play in policymaking but instead argues that it must be incorporated into large arguments and policy narratives in which ideology (values) can and must play a role. This is at the core of my own critique to the recent obsession with randomised control trials and impact evaluations (which are useful in some cases), or to the characterisation of the policy space as one in which producers and users of research and linked by a separate group of intermediaries (they often employ terms like demand and supply in an effort to turn what ought to be  public debate into a marketplace).

His conclusion is a call to researchers (in think tanks and in academia) to stand their ground:

it is important to defend the critical independence of academic research, and not to allow a situation in which the need for ‘user uptake’ can cause researchers to abandon their integrity and independence, so that ‘evidence-based policymaking’ starts turning into ‘policy-based evidence making’.

The full paper can be found here and it is well worth a read: Working Paper 21: Making Sense of ‘Evidence’ – Notes on the Discursive Politics of Research and Pro-Poor Policy Making

And, PLAAS is organising a conference on this in November in Cape Town. I hope you’ll be able to make it.