[This article was originally published in the On Think Tanks 2017 Annual Review. ]
Establishing research quality processes are critical to think tank credibility – they are like an airplane’s black box: unnoticed when everything is working well, but critical if something goes wrong.
I learned about the importance of how research quality affects credibility the hard way. I had never worked at a think tank before, but the research team I was assigned to was very successful at the time. I was involved in the production and launch of a series of popular papers. But at one point we made a big mistake, which we did not realise until we had already published the report.
Where did we go wrong?
We used inaccurate public information for our analysis, which invalidated the results. While it was not ‘technically’ our fault, think tanks cannot fall back on this sort of poor justification if they are to remain credible.
Looking back, the incident was preventable. We had become overconfident and in doing so failed to implement effective checks and balances.
Had we involved an external expert, or a public official, they would likely have either highlighted the mistake, or questioned the results we were obtaining from the data and made us dig deeper.
For me, the lesson from this experience is that think tanks need clear processes to ensure research quality. Of course, employing high-quality researchers is essential, but researchers can be fallible.
There are two reasons why these processes are particularly important. First, they can help identify problems before it is too late.
Secondly, if a credibility crisis does arise due to a research error, having clear processes in place makes it much easier to show that the mistake is due to human error, rather than a lack of capacity or integrity. During my crisis, my research team had nothing to hold on to, which made overcoming the problem even harder.
The editors of the British Journal of Pharmacology wrote an editorial recently about an article they had published, which did not meet the journal’s quality standards. While the piece had been peer reviewed, the editors included the paper without noticing the poor quality of the reviews. In the analysis of what went wrong, the editors realised that some reviews were even fake. This case highlighted serious gaps of their editorial process, which needed attention.
Think tanks who have an active communications operation can be particularly vulnerable to credibility problems, as they are the ones who can make headlines. For these sorts of organisations, having research quality processes in place – and publicly available – is very important.
Bruegel, a Brussels-based economic think tank, offers a simple and practical example that other organisations could copy: it publishes conflict of interest statements from all its researchers on their website.
In summary, establishing research quality processes improves think tank performance, while also being critical to their credibility.