[This post was originally published on Duncan Green’s conversational blog ‘From Poverty to Power’ on June 1, 2018.]
It seems rather hard to come up with anything original to say in the field of evidence-informed policy – unless we consider the change from evidence-based to evidence-informed policy. The case study approach, which has been tried and tested – and continues to be tested – inevitably concludes that researchers need to communicate more and better; that policymakers need to develop different capacities – to access, appraise, analyse, use evidence; and, among other lessons, that context matters – oh, and that we’ve got to consider politics.
Those of us who work in this field appear forever bound to place far too much importance on the ideal role of evidence in policymaking. We are cautious of anyone suggesting that policymaking is value based and that evidence is no more than a useful tool (or fool). But ask any politician, and we have been asking many of them through dozens of events promoted by the Evidence Week in Latin America, and they will agree that emotions rather than facts drive policymaking forward – or backwards.
There is, for sure, interesting research on the subject. Paul Carney’s and Kathryn Oliver’s work on the impossibility of adequately adapting evidence-informed medicine into evidence-informed policy asks an important question: should researchers really pursue impact? Then there is recent work on credibility – which is closely linked to emotion and, therefore, the use of evidence: Andrea Baertl’s study explores the factors that determine the credibility of research organisations, and of the evidence they produce. Spoiler: research quality is not at the top of the list.
(These studies stand out because they pay proper attention to the role of emotion.)
There is also much to learn from research that was never intended to answer questions related to evidence informed policy. Anthony King and Ivor Crewe compiled and analysed several cases of policy blunders by British governments. Their book offers rich insights into how some of the best educated, most capable, best resourced and informed people in Britain could mess it up so catastrophically. Their analysis suggests that the real gaps that need to be bridged are between policymakers’ and the public’s experiences of life (cultural disconnect) and between policy designers and policy implementers (operational disconnect).
They illustrate the pitfalls of groupthink rather well, too, although possibly not as eloquently as Michael Wolff’s Fire and Fury, which offers an unintended account of the Trump White House’ disdain for evidence – at least, for research based evidence. Christopher Rastrick’s paper on think tanks in the Trump administration presents a similar story. The personal and professional characteristics of the leader matter when trying to understand why an administration is close to policy research centres and keen on using evidence. It explains why Trump dislikes “outside” evidence and why Theresa May in the UK has been less ready to engage with think tanks than her predecessors – this last point was made by Nick Pearce at the OTT 2018 Conference.
One reason why these accounts stand out is precisely because they did not set out to find a desired role – or to test an expected role – for evidence. They offer hints at the ways in which information flows (or not) throughout the system. They acknowledge the influence of evidence but don’t overestimate its importance. They describe the way in which certain ideas have informed decisions but do not simplify the complex and long term process through which ideas emerge, consolidate and make their way into the mundane world of speeches, policy memos, business cases for new public programmes, or office squabbles.
To develop a convincing narrative of how research-based evidence helps inform policy decisions, it is far better to forgo our ex-ante theories of change and instead embrace the real world of messy, overlapping and start-stop stories with multiple characters and unpredictable plots. IFAD is attempting just that in its own efforts at monitoring policy engagement. This method does not set out to tell the story of how IFAD’s evidence informed policy; instead it simply wants to find out what happened.
These studies succeed because they present a dialogue between stories as told by the protagonists themselves. Each blunder in King’s and Crewe’s book is a conversation between several players and their ideas. Each battle between the Bannonites and Kushnerites in Wolff’s book is a clash between world views, personal and professional interests, and networks of influence. It is impossible not to find implications for evidence informed policy.
Not surprisingly, many coincide with the lessons emerging from the Latin American Evidence Week (Semana de la Evidencia), which in 2017 convened over 2,000 people who attended 53 events in 10 countries.
The festival offers a regional dialogue between policymakers, thinktankers, researchers, journalists, activists, business leaders and the public from which emerges a different narrative to the one we’ve become accustomed to:
- Most policymakers actually do want evidence to inform their decisions and most researchers are constantly thinking about and trying new ways to draw attention to their work
- Gold-standard evaluations aren’t the most useful source of evidence for decision making – most decisions require rather straightforward research and analysis
- The world is not divided between qualitative and quantitative research and their supporters – everyone is quite happy to use both whenever possible
- Testimonies are perfectly valid sources of evidence – and they are particularly legitimate ones and difficult for policymakers to dismiss
- And it all comes down to dialogue – but not between researchers and policymakers to close the proverbial gap – they tend to be much closer to each other than the literature would have us believe; rather, between policymakers themselves, between designers and implementers, between the national and the local level, between sectors, between disciplines, between the various branches of government, etc.
Whether you are in Latin America or not, find out more at semanadelaevidencia.org and join the conversation.