The importance of high quality research is undisputed among think tanks, which is why they put a high premium on having a strong reputation. Our last post from this series looked at hiring and retaining highly capable staff members as a way of ensuring quality work. This week, we will focus on oversight mechanisms such as peer reviews in order to guarantee research that is factually correct, logically consistent, methodologically sound, grounded in literature and written in a manner that is understandable and attractive to the primary audience.
A model peer review process is provided, one that is usually implemented at third-stage think tanks in the United States and most likely in think tanks from other countries as well. It is a formal review, meaning that it is mandatory and written. The following must be mentioned:
- the range of products subject to review;
- the person responsible for designating reviewers for different products;
- the criteria to be used in the reviews—usually, high-quality methods of analysis, conclusions based on the analysis, clear and effective presentation, and ensuring that the product corresponds to what was required under a contract or grant agreement;
- the form in which comments are to be provided (e.g., written, oral, in a particular format);
- the process for resolving possible disputes between the reviewer and the author(s); and
- the extent of the responsibility of the reviewer for any problems later identified with the product.
Of course, this model varies from institution to institution. Some are less formal, and some exempt senior staff from the peer review process. Other think tanks use in-house review seminars in place of peer reviews, which can be valuable but are not as effective in ensuring the quality of the product.
The chapter also goes over peer-review practices of 10 think tanks from Eastern Europe and the Commonwealth of Independent States, organisations with full time and part-time analysts. Nine out of the ten had a peer-review policy process, but only four of the nine think tanks with said policy had a written policy statement that governed the process. This might leave the door open for different interpretations of what good quality research requires. Also, seven of the nine think tanks require a written review. As for who oversees the process, it is usually senior management. When it comes to disputes between reviewers and authors, they usually do not need a third person to intervene but work it out between them.
In short, as mentioned above, quality control is generally taken quite seriously by the think tanks studied. As always, every think tank should determine the resources that it can dedicate to quality control.