The donor perspective: why support a peer review system?
[Editor’s note: This post was written by Antonio Romero, Program Officer for Latin America in the Think Tank Initiative. It is part of the series devoted to discussing peer review processes for think tanks. It has been edited by Andrea Ordoñez as part of the Guest Editor initiative launched by On Think Tanks last year. If you are interested in being a Guest Editor please get in touch.]
Sitting at the interface between research and policy, think tanks face the question of what makes excellent research. This is not an easy question to answer. As noted in a report on research excellence evaluation by the International Development Research Centre (IDRC), there are various definitions for research excellence. There is also disagreement on the quality dimensions by which research should be evaluated. This makes it difficult to address a related question: how can think tanks enhance the quality of their research?
This is a question that the Think Tank Initiative (TTI) grapples with, as it seeks to promote research excellence among the institutions it funds. The TTI is a multi-donor program dedicated to strengthening the capacity of independent policy research organizations in the developing world. The Initiative currently provides 48 think tanks in 22 countries with core, non-earmarked funding. This support allows the institutions to attract, retain and build local talent, develop an independent research program, and invest in public outreach to ensure that research results inform and influence national and regional policy debates. This core funding is combined with dedicated capacity development in three broad areas: research methods and skills, policy engagement and communication, and general organizational effectiveness. The Initiative also supports peer-to-peer review, learning and exchange.
The peer-review challenge
In considering capacity development modalities aimed at enhancing research quality, peer review emerged as an interesting possibility. Simply put, peer review is a process through which research documents are reviewed by other researchers in order to provide authors with useful feedback. Conversations with some executive directors of TTI think tanks in Latin America suggested that peer review was considered a valuable tool for enhancing research quality. However, not everyone agreed that peer review was necessarily appropriate for think tanks. The main issues tend to be the difficulty in finding external reviewers and the time it takes for reviewers to provide feedback. The latter was particularly problematic for some executive directors. They argued that investing time in peer review was often incompatible with the urgency to publish results in line with policy timing.
Despite these challenges, Latin American grantee think tanks increasingly started to adopt different peer review arrangements:
- One think tank began to use peer review on a sub-set of its research production, and to pay external reviewers for their time. The executive director of this think tank argued that paying reviewers made it easier to enforce deadlines.
- Another institution formalized a peer review system through a research protocol that established mandatory internal peer reviews. In this case, reviewers were drawn from within the institution, with an aim to draw from external reviewers in the future. This institution developed official peer review guidelines which outlined the review process, assessment criteria, and expected outputs.
As these innovations began to take shape, TTI made available to its grantees a pilot peer review mechanism. The idea was to provide an opportunity for think tanks to send documents for review without having to worry about finding reviewers and liaising with them. An independent consultant –Andrea Ordoñez- was charged with implementing the peer-review mechanism. Her responsibilities included matching research documents to suitable reviewers, liaising with reviewers and researchers, and ensuring deadlines were met. Our expectation was that this mechanism would encourage think tanks to consider the potential benefits of establishing a formal peer review system for their institutions. The jury is still out on whether this activity will spur new peer review arrangements in participating think tanks. But the value of the reviews conducted through this exercise is reflected in participants´ feedback. The upcoming posts in this series will discuss in detail the design of this peer review mechanism and what participants had to say about it. If you want to learn more – stay tuned!