The issue of quality control comes up time and time again in my work. This is something think tanks are constantly asking about and some have found interesting solutions to it. Some keep it in-house and other make use of external reviewers or members of their boards. Each choice has its pros and cons and these should be taken into consideration. Here are some things that think tanks should keep in mind before deciding what is the best model for them:
- The need for speed: Whatever the process think tanks need to be fast. From idea to publication a think tanks should always try to beat its ‘competitors’. A great policy brief or opinion piece is of no use if it is published a couple of months after a decision has been made. Therefore, the process has to be fast and flexible.
- Timing: While speed is important timing is crucial. Even if the paper is ready its publication needs to be planned carefully to maximise the chances that it will have the desired effect. Therefore, the process has to be, as much as possible, planned and well managed.
- Quality is not always about being 100% right: Inevitably, speed may force some trade-offs in terms of quality. Many of the problems that think tanks will be working on will not be easy to understand or explain. Still they have to engage and communicate. It is of no use to tell the public ‘come back later when we are done’. They won’t be back. Hence they have to pay attention to the building blocks of good research and analysis to ensure that even rough numbers and preliminary findings can be communicated and used. The final answers may change but this will not necessarily damage the reputation of the think tank if they are leading the process and being open about what they are doing and learning.
- Transparency: What surprises me the most is that few think tanks explain their quality control processes to their publics. This would certainly be useful and it could strengthen their reputation. And why not invite others to use the same data or review their work publicly?
- Cost: Quality control can be expensive and so think tans must consider what is appropriate for different types of efforts. Not everything needs to be reviewed in the same way. Some things can just be someone’s opinions -with all the disclaimers that demands.
- Think tanks are not (always) universities: Whatever process is established think tanks must remember that they are not always publishing peer reviewed papers. Their publications are often ‘up for debate’ and do not need to be checked and re-checked as if they were a PhD thesis. Mistakes, specially in the assumptions, are hardly ever the end of the world and can be managed with a good communication strategy (this is not, I must stress, an excuse of poor research and analysis).
So what can think tanks do? Here are some options with their pros and cons:
- Internal quality control: Some think tanks keep the quality control responsibility within the organisation. Sometimes within a programme. They assume that programme leaders or managers are sufficiently knowledgeable about their subject of expertise to review most publications. Some, maybe key reports or publications that are seen as being from the organisation are sometimes reviewed by members of the senior management team and, even, by the executive director. The advantage of this approach is that the process can be sped up to meet windows of opportunities before they close. The disadvantage is that there may perverse incentives against criticising one’s peers may affect the depth of the reviews. In any case, if this was the preferred approach I would recommend that reviewers included a senior researcher who may be able to comment on the method, a senior researcher or expert who could check the accuracy of the statements and arguments, and a senior communicator to ensure their clarity and political relevance.
- Outsource: This option often includes using independent or third party experts to review certain publications. This can be combined with the other two options as it would be unsurprising that the think tank may eventually do research on an issue it is not entirely familiar with. In these cases, why not add the names of the reviewers to the publications? It would certainly address the issue of transparency. The main problems with this approach is that it is unlikely to be fast and could be expensive. Therefore, it should be reserved for certain publications, ideally those planned in advance to target key, and expected, policy windows: e.g. budget debate, elections, etc.
- Using the board: Another popular approach involves using the board. This is the most challenging of all. Outsourcing often involves sub-contracting and internal reviewing a degree of line-management. Hence the think tank has some control over the process. The board, however, is likely to do this voluntarily: board members are, after all, not paid to do their work (this may not be the case in all think tanks). Therefore those who take on the quality control role must be really committed to doing so properly -and fast.
In all three cases think tank should remember:
- To establish and follow a process with clear roles and responsibilities and, most importantly, time schedules. e.g. all comments must be provided within 24 hours or it will be assumed that the publication does not need to be edited.
- That the time of the staff and external parties involved must be carefully budgeted to make sure that no corners are being cut.
- That the best way to ensure quality is to hire the best people. Good researchers and communicators are less likely to need significant quality assurance efforts.