Previous
In my last article, I wrote about what an analytical framework for an ‘evidence context’ study might look. Here, I highlight thirteen lessons to hold in mind when conducting a study (primarily targeting people from a ‘Northern’ context). It’s based on studies of the policy process I’ve conducted jointly in Indonesia, Vietnam, India, South Africa and Nepal.
Work with local partners and negotiate your respective roles
You’ll benefit greatly by doing the research jointly with local researchers or consultants. From a developmental perspective this is important in itself. From an instrumental point of view, they are likely to know the local context and cultural sensitivities, speak the local language(s), be part of professional, formal and informal networks, enjoy legitimacy and recognition among peers and, if you’re from a high income country, will command lower costs (even if this is unfair). Without the ‘social capital’ they enjoy, you’ll have challenges in accomplishing your study.
I’ve written on the practicalities of working collaboratively with partners here. Nevertheless, local partners may be reluctant to work with you on an equal footing due to a lack of experience in undertaking such a study, a fear of their writing and approach being judged harshly by a foreign collaborator and/or a lack of interest in doing evidence studies or a study on the specific issue in question. On the latter, if you are working with a research centre or think tank, researchers are often more interested in doing a study on policy content (rather than process), especially in their areas of interest/expertise and promoting its uptake (rather than supporting government to use a broad range of evidence). But these issues may only surface – and informally – part way through your work together. Negotiating with your partners exactly what you want to, and can, do, together not just at the start of the work, but throughout, will help maintain a healthy relationship with your partners and deliver outputs on time.
Ensure that research results can be channelled back
Potential interviewees may be suffering from ‘interview fatigue’ especially if their issue is the subject of considerable attention from donors and other stakeholder groups, while officials may be more interested in the instrumental formulation of recommendations and less so in a detailed analysis of problems. Potential interviewees will also have expectations about investments or tangible outcomes that will follow from the research.
You’ll inevitably be perceived as a donor, which can be a source of tension during the interview. When it is clear these expectations are not going to be met, interviewees may refrain from being frank. As such you might find interviewees more forthcoming if research results can be directly channelled into capacity development work with their organisation afterwards. If this isn’t possible, you’ll want to give something back at least by sharing research findings with interviewees through, for example, a discussion workshop. You need to show that you’re not going to take an extractive approach. If not, interviewees may judge an interview with you as a waste of their time.
Don’t panic if your government counterpart (a key user of your research) wants to get involved in the research
You might find yourself working closely with a government official who wants to ‘get their hands dirty’. It’s good practice for users to be involved in the research itself as it promotes its use. You’ll find this unsettling, but an engaged official may give you access to key government officials (but this may be harder where officials are from a different agency). Participating in interviews will reveal useful insights which might lead to improvements in the way they view and use evidence. A government counterpart will likely see interviewees not as subjects to be studied but as collaborators, and interviews as discussions. But there are trade-offs. Non-governmental groups may be less willing to agree to be interviewed if they know a government official is present, whilst interviewees may not be frank in their responses. Research ethics codes which promise anonymity of interviewees may have to be put aside.
Use both formal and informal channels to secure interviews
You’ll want to secure interviews with officials with whom your research partners have cultivated relationships of trust. An email or text message may suffice. However, if you’re aiming to interview government officials, you may, in addition, need to send requests through proper channels, from the top down. Risk averse officials will need assurance that they are not doing something untoward when agreeing to be interviewed by you. Moreover, officials may be more likely to say yes to an interview if the request is made on a letter head for the agency which is funding the project (although this will come with expectations of a follow up investment).
Serving officials may not always be the best people to interview
Serving officials may be reluctant to be frank with you about the true nature of policymaking processes (which we describe below). As such you may want to speak to people who are knowledgeable about policy processes but at the same time have some distance from it (with less to ‘lose’). These could be recently retired or ex- government officials (who may have set themselves up as a consultant), foreigners on long term assignments working in and around government institutions, foreign academics with a long-standing interest in the country in question, and local consultants contracted into government on short term assignments. Some foreign academics will have cultivated close relationships with political insiders and will have a deep understanding of the context – although they may not always want to share this with you.
Be aware of your positionality as a foreign researcher
What interviewees say will be influenced by your positionality and perspectives which in turn will be shaped by your class, gender, nationality and other identities. Although British, my brownness and South Asian heritage opened up certain spaces and probably closed off others. In Nepal for instance, the local director of a research firm was very frank about the extractive nature of British researchers and consultants, which my white colleagues probably weren’t privy to. In some cases, I was able to develop good rapport relatively quickly with policy actors, who perhaps saw a little of themselves in me. However, when conducting focus group discussions with communities, people were confused and perhaps a little suspicious that I was brown but spoke English with a British Accent. Generally, being a ‘bi-cultural’ researcher has helped to me to avoid the tendency among US and European authors to interpret the world in terms of their own country histories.
Create a ‘safe space’ to help interviewees be frank with you
Seated in formal offices, with large desks between you and your interviewee and certificates hanging from walls, you may receive answers to your questions that are in line with the official position of the organisation. Some officials will feel that you are evaluating them, rather than providing them with an opportunity to reflect on their practice and develop new insight. They might suspect that you are planning to ‘hang their dirty laundry out to dry’ and extract data for publishing purposes or will not appreciate being asked about internal business – issues that might be considered ‘family matters’.
They’ll be reluctant to acknowledge potential gaps between formal processes and actual practices or brush over the details, overstate the importance of evidence in their work and be reluctant to discuss conflicts within and between government institutions and other policy actors. Officials may well have promotion on their mind and will want to minimise the risk of having their own reputation damaged internally (for potentially being critical of the institution). They will also be anxious in sharing details especially given the practice in some governments of axing the messengers who bring bad news, or finding scapegoats when things go wrong. Officials may subsequently choose not to answer your questions instead describing the issues the institution is working on. The use of a voice recorder can further constrain what they say.
You may find interviewees invite their colleagues to join them – achieving ‘safety’ in numbers. They might be more junior or senior. Who speaks will be shaped by gender and age with senior men dominating and younger women speaking only when asked to. To accommodate the larger number of people your interview may take place in a conference room, with larger tables and the presence of microphones, further structuring the boundaries of what is or is not said.
Rather than having interviews in officials’ offices, you may want to suggest taking a walk around the building or going for a coffee (where the context allows this). The atmosphere can change significantly and interviewees can feel freer to discuss from their own personal opinion and experience of policy discussions and evidence practices. Frame the conversation as a discussion rather than an interview. Reassure ‘participants’ that you are not evaluating them but jointly exploring with them both the ‘institutions’ potential to excel and the challenges it needs to confront’. Whilst you’ll be interested in problems and gaps, your interviewee might be more interested in good practice and lessons: you’ll need to strike a balance. Having said this, in some cases interviewees will see speaking to you as an opportunity to express criticism (as an expression of resistance), with a larger and different audience, without taking a significant personal or professional risk or hope that you will in some way influence circumstances for the good.
Be creative in getting officials to talk about policy processes and their use of evidence
Those in government may struggle with discussions about process. They may be comfortable talking about the ‘what’ (of health, education or the economy, etc), but you’ll find eliciting response to the ‘how’ questions hard, especially if they are open-ended. Policy process and the use of evidence may come across as abstract and tangential to their core work. Officials may redirect you to internal research units where they exist.
More, the word evidence can throw up unexpected reactions. For instance, in Indonesian, the word evidence directly translated is the term used for evidence that is presented in a law court, which may have made Indonesian officials reticent, especially given the scrutiny they were under from the Auditor General to account for public funds.
So rather than referring to problems, you may want to use indirect questioning or observation, or choose less controversial language by asking officials about the challenges they face. Avoid talking about ‘political-economy’, ‘interests’ and ‘incentives’ and instead use words such as motivations and relationships. And you may want to avoid asking them explicitly about evidence and instead ask them about decision making, influence and the different ‘resources’ they draw on to support their policy work.
You may have to use your interview guide flexibly, following what the officials says, asking questions tangentially, where opportunities arise, focusing on short closed questions, and asking them to elaborate where necessary. You will also have to be sensitive to what people say, the means through which they say it (with some using humour to express difficult feelings) and what is left unsaid.
Where possible draw on already written articles
You may find interviewees reluctant to provide much detail about the policy context, or you simply won’t have the time to get to the bottom of how policy unfolds. In these cases, you’ll have to draw on articles, especially academic, that have already been written. Whilst not explicitly about the use of evidence they will go a long way in helping to you understand the political and policy context in which evidence is acquired and used. In Indonesia I found the work of academics such as Edward Aspinall, John McCarthy, Marcus Mietzner and Elizabeth Pisani amongst others invaluable. International agencies may have also done tangential work. Issue specific technical reports by the World Bank came in very handy in Indonesia whilst the Asian Development Bank’s work on Making Markets Work Better for the Poor in Vietnam was very useful. Reading these sorts of reports/papers can help you ask more focussed questions during your interviews.
Interviewees can’t or won’t always speak English
Where you’re working in a context where English is not the official language, carrying out your research in English will mean you’ll miss the informal banter between people, which can be revealing – even if your interviewee has very good English. Moreover, officials may struggle to understand the subtleties of your question and in turn give relatively superficial answers.
Good simultaneous interpretation is often difficult to come by and very expensive. Your research counterparts may subsequently suggest they interpret for you. But this then undermines their ability to listen and analyse what is being said. And your counterparts may lack the specialist interpreting skills that they will need.
Where they are needed, take the time to train your interpreter
If you do work with an interpreter, this brings its own challenges. Their positionality may well create difficulties. Although they may be very familiar with the language, they may be unfamiliar with specific terms used to describe policy processes. In some cases, they may not understand your methodology and want to answer your questions themselves.
Taking the time to train interpreters to appreciate the rationale for particular methods and questioning strategies can be very useful. Where possible, interpreters should have some knowledge of how government works and the broad policy environment. They should also be able to engage with people of different backgrounds and be conscious of their own positionality. You will also need to respond to interpreters’ own interests and professional development needs by for instance, valuing their insights and allowing them the opportunity to provide input into the research process.
Be careful what issues you put into writing
During your work you and your partners will have amassed a huge amount of information. However, you may not want to write up the more unsavoury elements of your findings, especially if you are planning to work with the same officials you have interviewed. Putting into writing the more contentious issues people have expressed during interviews may provoke feelings of shame and elicit defensiveness. As such these issues, expressed subtly may be best left discussed orally during a closed-door meeting. You may need to induce a sense that one cannot not live with the current situation but also provide people with a sense of purpose. In South Africa, during a BCURE project, researchers were being invited as guests into “the kitchen” of a particular government agency. And guests were expected to behave appropriately. Findings that discredited policy makers was thus deemed inappropriate. You may have to subsequently write more ‘appreciatively’, avoiding criticism and emphasising good practice and lessons to avoid antagonising your counterpart. And you’ll have to accept that you may have to redraft your research several times.
Give contributors a chance to make sense of your findings
Finally, policymakers will have limited time to read your (or other) report(s). Produce short summaries, and/or schedule small focus group discussions (rather than large seminars) to share findings with potential users (e.g. policymakers) and enable them to comment and reflect on what it might mean for their own practice.