Peer review: experimenting with think tanks

16 June 2014
SERIES Peer reviews for think tanks 8 items

[Editor’s note: This is the first of a series of posts on a peer review mechanism pilot for think tanks. It has been edited by Andrea Ordoñez as part of the Guest Editor initiative launched by On Think Tanks last year. If you are interested in being a Guest Editor please get in touch.]

At the end of 2013, I started a small project with the Think Tank Initiative (TTI): a pilot Peer Review System for the think tanks that are supported by the initiative in Latin America. This idea, born originally at the first global meeting of the Initiative, came to life in the last months. Behind this pilot is a spirit of maximizing the value of being part of this community of institutions.

Peer review still is the most common, valuable and credible mechanism available for researchers to assess the quality of academic work. The principle is simple: other experts in the field are the best qualified individuals to judge the relevance, originality and rigour of a peer’s work. The concept has attracted others, and there are those who suggest transferring it to other arenas, with initiatives now available for government officials to exchange reviews with other countries’ colleagues (for example, the African Peer Review Mechanism).

Given the centrality of the peer review process in academia, there is much debate on how relevant it is, if it is really improving the research process, and if other ways of assessing research should be developed. Think tanks, living in the complex settings of academia, practice and policy, often see peer review processes as the mechanism to assess the academic aspect of their work, and also a mechanism to gain credibility.

What can think tanks learn about the peer review processes for their own work? What are the specific needs of think tanks when it comes to reviews? Are there special considerations to be taken into account? How do researchers and reviewers react to a peer review process? [Editor’s note: CIPPEC has an internal peer review process that is worth looking at.]

This series, devoted to the implementation of the peer review options for think tanks, seeks to shed some light on these concerns. It is based on work with eleven think tanks in Latin America, who submitted eighteen products to be reviewed. The process involved carrying out a scoping study of what think tanks currently do to assess their work, the revision of their knowledge products, and the systematization of feedback from the researchers and the reviewers on the system. This series is a work in process; the idea is to gain as much information as possible, along with comments from others who might be interested in peer review processes to enrich future interventions.

The series includes the following posts:

  1. The donor perspective: why support a peer review system? Antonio Romero, from the TTI in LA will share the ideas behind TTI’s decision to support this pilot as well as their expectations of the process.
  2. What is the role a peer review system plays in the think tank context? This post will dissect the concept of peer review: What is it? How does it work? What are their advantages and shortcomings?
  3. Is Research from Think Tanks Really Different? This post summarizes the findings of the scoping study: what are the types of knowledge products think tanks produce, and what type of peer review, if any, do they use? At the end of the day, how different are think tanks from universities?
  4. Peer review and training of young researchers – Horacio Vera, a junior researcher at INESADshares his perspectives on the peer review process and how to improve it.
  5. (Re)Creating a culture of peer review – Patricia Ames, research director at IEP shares the challenges of peer review processes for think tanks.
  6. Perspectives on the peer review system – This post summarises the key aspects of the evaluations by authors and reviewers.
  7. Lessons from Peer Reviewing among Think Tanks– the way forward.

I am really eager to hear what researchers at think tanks have to say about these findings. I hope that some of the questions we raise throughout the series inspire us to further discuss the nature of the work think tanks carry out and if a peer review process has a role to play in these settings.