Research questions are not the same as policy questions

At a planning and M&E workshop for a very interesting project managed by the South African Institute of International Studies on Global Economic Governance I came across this often forgotten challenge: research questions are not the same as policy questions.

Donors always insist that they want to make sure that the research they fund is relevant or demand driven. Both very difficult asks. And the latter is, in fact, a dangerous one: if we only answer the questions what the policymakers ask then who will answer the questions that they should be asking?

To define relevance we tend to focus on whether the question is of interest to the policymakers. For instance, how are trade patterns with country X’s main trade partners? This is surely relevant to policymakers in the ministry of trade.

But is this the policy question? Let’s go a bit further? What would policymakers want to do with this information?

They could, for example, want to know if there are facing new competition from countries less developed than them in the past; what specific sectors were under stress or which seemed to be picking up or emerging as future opportunities; or  what effects had domestic and international policy changes had on these.

But we can press harder: Policymakers might want to know if their national policy frameworks and trade policy strategies were ready to address future challenges and threats; or what skills needed to be developed to take advantage of the rise of new trading opportunities.

Slowly, by asking these questions we move away from trade research and into trade policy and even education policy. We  could include policy questions relating to infrastructure, labour policy, social security, science and technology, etc.

But recognising this is just the beginning of the challenge for think tanks. Once it is clear what the policy question is the research questions multiply. To answer a policy question think tanks need to answer many research questions.

‘What skills need to be developed?’ demands, for instance:

• What are the trade patterns and how are they changing?
• What are the sectors that present the best opportunities for the country?
• What is the present policy (trade, industrial, legal, labour, etc.) framework?
• What is the current state of the national skills-set?
• What are the demographic trends the country is going through and how would this affect the development of new skills?
• What mechanisms are there to change and develop them?
• What is the state of the national education and innovation sector?
• How much would it cost and what would be the trade-offs?
• What lessons from national and international experiences are there?
• Etc.

And this is where think tanks can find their niche. Think tanks can and should be able to work across sectors either within their own organisation or in collaboration with others to address all these questions. University research departments with their sometimes ‘irrelevant’ or ‘researcher led’ research can be invaluable supporters of this process as they are likely to have done the research already.

Think tanks’ comparative advantage lies, or should lie, in translating policy questions into research questions and research question answers into policy question answers. To do this they must behave as boundary workers: simultaneously working in the policy and academic communities.

The best example of this kind of boundary work that I have come across is from the Vietnam Academy of Social Sciences VASS). We had been hired to help VASS communicate its research more effectively (apparently it didn’t before) to inform policy. VASS, like the Chinese Academy of Social Sciences, is a large organisation that hosts several institutes that cover issues as diverse as economic policy, North Korean studies, and agriculture. Its President is a member of the cabinet as is a political appointee; albeit a career academic. During one of the visits to VASS we asked the President if any of the researchers or directors of the institutes in VASS ever went directly to the senior politicians and policymakers he worked with: ‘No,’ he replied. And there is a reason for this.

The questions that he or the directors of the institute brought to the Academy were not, he explained, research questions. Through his interaction with policymakers they formulated policy questions that had to be translated into research questions that could then be assigned to several researchers, depending on the issue or challenge being addressed. In his case, in particular, the questions he received had a strong political component. He wasn’t just asked to provide advice on how to develop certain skills; he also had to take into account the country’s fiscal space, budget trade-offs with other sectors, political trade-offs with other government promises and objectives, etc.

In his mind, and I think he has a point, the power of this think tank rested in its capacity to bring together different disciplines, areas of policy research, and views to support equally multi-disciplinary, across-government, and often ideologically contradictory policies.

To address relevance we need more that my taxi driver test (which is probably more useful when it comes to thinking about the message one wants to share with the general public). What is needed is real participation in the politics of a nation, a policy sector or a community. This fad in ‘political economy analysis’ (often not even done properly) of doing one for every sector and country to inform interventions from afar is no more than that: a fad. To really understand what is relevant one needs to be ‘there’.

As an outsider at the event in South Africa, all I could do is facilitate the process by pressing on, challenging the assumptions that the researchers were making about what policymakers would or should ask, and encouraging others to do the same.

There is one more challenge. The policy question may be one that needs to be answered by may still not be the one that the policymakers want answered. In this case, jumping straight to the answer may not be the best approach. Think tank will have to get policymakers to acknowledge the importance of this or that particular question before they can offer their well researched and argued answers. And the key to this may lie in finding the right policy question again.