Methodological choices to inform policy

There is a persistent perception that there is a tension between rigorous research and research being relevant, thus increasing the chance of having an impact. Of course, we understand the origins of this tension. For example, some rigorous methods require more time than those given by policy windows, a reconciliation that is not always easy.

Nonetheless, we believe that there have to be ways by which a researcher can make better methodological choices that can be both rigorous and coherent with the policy context. This is why in Politics & Ideas, we are developing a set of tools, and capacity building strategies that aim at helping researchers to navigate the methodological choices with a compass set on policy priorities. Here, I will share some ideas of navigating methodological choices that researchers can keep in mind for their current and future projects.

The methodological Swiss Army knife

The distinction between qualitative and quantitative research is not very informative

When researchers describe the methodological choices they make, they usually mean that they consider whether they will use quantitative, qualitative or mixed-methods. These categories have become popular, but neither are very helpful to understand exactly what is being done, and how this will answer as a research question. Furthermore, this can scale up into a competition, between those in the “qualitative camp” and those in the “quantitative camp”, a discussion that is not very fruitful, as each method has its strengths for certain types of questions, and its limitations for others. When considering what the best method is, try to think beyond these categories.

Research methods as your own Swiss knife

One way of thinking about methods is to imagine that they are like a Swiss Army knife: a collection of tools, each one with a particular strength. For example, you wouldn’t use a knife to open a bottle of wine when you have a corkscrew among your tools. The same should be true for methods: researchers focused on informing policy should develop a variety of methodological skills to choose from, depending on the specific need of each occasion. Given that you have these tools at hand, the key next step is to choose among them wisely.

Policy problem: the compass to choose methods

But how do we know when to use each method? I believe that our chances of informing policy depend highly on our choices at this stage. To decide what methods to use, it is important to look at the bigger picture:

1. Define a policy problem and be able to describe it in both technical and political terms. This takes some initial effort for us to have a good sense of the context.
2. Identify what is the purpose that research can plays in each specific case. Will research be used to find a solution? To raise a topic into the agenda? To facilitate a political negotiation? Not all research methods will serve all these purposes.
3. Formulate a better research question that is better aligned with the specific problem at hand and of the research that can be done in that particular case with the clarity of the first two steps.
4. Select a method or a combination of methods to use, that make more sense in your context and with your purpose in mind.

A practical example

Let’s imagine a country that has been implementing a Conditional Cash Transfer Program for the last 10 years, and now, there is an interest to reform it because of an economic crisis that has decreased the government’s budget.

In the first step we propose you must  identify what the problem is: although there is political support for a reform to the CCT program, different actors have very different positions on the issue. You notice many times they are referencing research, but the evidence is much dispersed and in some cases even contradictory!

In the second stage, you consider that you can get involved by making sense of the existing evidence. You have identified more than 10 studies on the CCT’s in your country alone – no wonder everyone was referring to such divergent results! Your objective becomes now to give more clarity on the existing evidence; rather than undertaken original research.

In the third stage, you would plan to answer the following questions: what does all the existing evidence means at the end of the day? Is it quality evidence? How can we make sense of contradicting results?

Finally, and only here, do you identify your research method and you may choose to undertake a meta-analysis of the existing studies on the CCT programme and to place this analysis within the broader evidence of CCTs in other countries.

As you can see, making a choice about a method, in the context of informing policy, is much more complex than deciding, up front, to pick between quantitative or qualitative approaches.