[Editor’s note: This is the third of a series of posts on a peer review mechanism pilot for think tanks. It has been edited by Andrea Ordoñez as part of the Guest Editor initiative launched by On Think Tanks last year. If you are interested in being a Guest Editor please get in touch.]
Objectives and limitations
The first critical question is why we carry out peer review processes. There is not one answer for this, so let’s explore some of them.
Gatekeeping – By filtering and screening material, the peer review system is the way in which journals have traditionally decided on what to publish or what not to publish. The reviewers usually face the following recommendation choices:
- to unconditionally accept the manuscript or the proposal,
- to accept it in the event that its authors improve it in certain ways,
- to reject it, but encourage revision and invite re-submission,
- to reject it outright.
Peer reviews also have a place in grant making processes and academic conferences presentations. At the end of the day, the objective is to distribute limited space, time and resources: not everybody can get funding, get published or present at conferences.
Maintain reputation – It has been argued that peer review processes were popularized by the Royal Society to protect its reputation. Reputation is thus based on the quality of those involved in the review process, which is why being part of a journal editorial board is a highly regarded position among researchers. Furthermore, the decision of publishing or not is not only based on the editor’s judgment, but includes that of others.
Quality control – Peer review processes are supposed to ensure that research papers are consistent: that an appropriate methodology is used, that conclusions presented are backed by the research undertaken, and that other relevant knowledge on the subject is correctly acknowledge (basically, a good literature review).
Capacity development – although this aspect is not the main concern of the world of publishing, I believe that receiving input about one’s work is a valuable opportunity to improve one’s work. This usually requires a capacity to interpret and critically analyse comments holistically. Many think tanks I know introduce peer review as a mechanism to support researchers, even when peer review is not used for the previous purposes.
Although the peer review process is a keystone in the academic world, it is not perfect. At the end of the day, the responsibility of the research paper is the authors’. Fraud and plagiarism cannot be detected by a reviewer. A peer review is no guarantee of the validity of the research presented; that responsibility lies with the authors. In fact, Springer recently retracted 120 papers that were actually computer-generated nonsense. Authors can act unethically, and reviewers may not have enough information to judge those actions.
Criticisms of the peer review processes
- Some have argued that peer review processes are biased. Researchers that cite the right authors, that maintain conservative views and follow traditional methods might get more published, some believe. It is hard to assess this, as the peer review documents are usually not public.
- Santiago Basabe recently wrote a great reflection on the bias on ‘good data’. This particularly affects countries with deficient information systems. As a result, countries with better data are more studied than those with poor data. Although we want more knowledge of these countries, this bias reduces the interest of researchers, who see their chances of getting published diminished even if their work is methodologically sound with the information currently available.
- The way peer review processes are carried out now delays publication significantly. This, some argue, is detrimental to scientific inquiry and it may have an impact on the diffusion of such knowledge, as authors move quickly into new topics.
- Peer review is highly connected to the “publish or perish” paradigm of modern science. Are the high expectations on publishing responsible for the growing manipulation of peer review processes? These high expectations on publishing are pushing the limits of the system. Researchers are encouraged to create various papers out a research project and longer term processes are discouraged. As Peter Higgs argues, today he might not have the space to do the work that involved studying the Higgs boson in the sixties.
This brief overview of the peer review processes shows how this simple concept, in practice, gets quite complex. This is not meant to discourage researchers from participating in such endeavours but to do so with a critical perspective.