{"id":918,"date":"2014-06-25T21:21:09","date_gmt":"2014-06-25T21:21:09","guid":{"rendered":"https:\/\/onthinktanks.org\/?p=918"},"modified":"2016-01-06T18:09:10","modified_gmt":"2016-01-06T18:09:10","slug":"is-research-from-think-tanks-really-different","status":"publish","type":"post","link":"https:\/\/onthinktanks.org\/articles\/is-research-from-think-tanks-really-different\/","title":{"rendered":"Is Research from Think Tanks Really Different?"},"content":{"rendered":"<p><em>[Editor\u2019s note: This is the fourth\u00a0of a series of post on\u00a0<a href=\"http:\/\/wp.me\/pYCOD-1wM\" target=\"_blank\">a peer review mechanism pilot for think tanks<\/a>. It has been edited by Andrea Ordo\u00f1ez as part of the\u00a0<a title=\"On Think Tanks is looking for guest editors (a pilot)\" href=\"https:\/\/onthinktanks.org\/2013\/10\/18\/on-think-tanks-is-looking-for-guest-editors-a-pilot\/\" target=\"_blank\">Guest Editor initiative\u00a0<\/a>launched by On Think Tanks last year. If you are interested in being a Guest Editor please get in touch.]<\/em><\/p>\n<p>Before establishing a peer review system for think tanks, it is important to understand what types of knowledge products they prepare. After all, the goal of this project is to serve the specific needs of think tanks. The participating think tanks shared some information on their structure, products, current editorial rules and expectations of a peer review system. The twelve centres from the\u00a0<a href=\"http:\/\/www.thinktankinitiative.org\/think-tanks\">Think Tank Initiative<\/a>\u00a0in Latin America were invited to participate, out of which 11 filled out this survey. Here are the main findings of the survey and what they mean by a peer review system.<\/p>\n<h3>Key Findings<\/h3>\n<ul>\n<li>In terms of the structure, most centres have both a\u00a0<a href=\"https:\/\/onthinktanks.org\/2012\/12\/12\/new-director-of-research-post-at-grupo-faro-an-opportunity-to-learn\/\">Research Area<\/a>\u00a0and research protocols which include some sort of peer review. This means that \u2013 hopefully \u2013 a peer review process could strengthen not only the work of the author who participates but also inform the organizational processes in general.<\/li>\n<li>In relation to the\u00a0<a href=\"http:\/\/wp.me\/pYCOD-1wQ\" target=\"_blank\">peer review system<\/a>\u00a0that they use, the most common one is an open review system, where both the reviewer and the author know their identity. From my personal experience, this is not surprising: in settings with small local epistemic communities, keeping a double blinded process can be almost impossible.<\/li>\n<li>These think tanks produce \u2013 what I would call \u2013 traditional products, mainly: books, working papers and to a lesser extend policy briefs. When asked, for which products the peer review process is more relevant (on a Likert scale from 1 to 5), their scoring followed a similar trend.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/onthinktanks.files.wordpress.com\/2014\/06\/graph1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter  wp-image-5884\" src=\"http:\/\/onthinktanks.files.wordpress.com\/2014\/06\/graph1.png?w=633&amp;h=466\" alt=\"graph\" width=\"633\" height=\"466\" \/><\/a><\/p>\n<ul>\n<li>I also explored the idea of carrying out the peer review at three stages of the research process: at the proposal stage, when there is a manuscript ready for publication and after the publication has been finalized as an evaluation. All centres highly value the peer review at the manuscript stage. The proposal and final stage were less relevant.<\/li>\n<\/ul>\n<table>\n<tbody>\n<tr>\n<td width=\"105\"><\/td>\n<td width=\"128\">Scoring (Likert Scale 1-5)<\/td>\n<\/tr>\n<tr>\n<td width=\"105\">Proposal Stage<\/td>\n<td width=\"128\">3.1<\/td>\n<\/tr>\n<tr>\n<td width=\"105\">Manuscript<\/td>\n<td width=\"128\">5.0<\/td>\n<\/tr>\n<tr>\n<td width=\"105\">Final Evaluation<\/td>\n<td width=\"128\">2.5<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<ul>\n<li>Think tanks also highly valued that the peer review process takes into account the particularities of each type of output. This was agreed by many to be the most important aspect included in the review process.<\/li>\n<\/ul>\n<h3>So what\u2026<\/h3>\n<p>Although this analysis was meant to capture the specificities of think tanks (in comparison to university and others), the results do not show many particularities. These think tanks prioritize producing working papers and books, and see the peer review process as most relevant for these types of products.<\/p>\n<p>I was also testing whether they would be interested in having a peer review process in other stages, and not only the manuscript. In this sense, think tanks were also conservative, prioritizing the peer review process for manuscripts, a similar process to that of academic journals.<\/p>\n<p>Interestingly, centres value having specific evaluation for different products. In this sense, they seem to see themselves as different from other knowledge producing actors and their outputs as needing specific criteria of evaluation. Isn\u2019t this contradictory to the fact that they produce \u2018traditional products\u2019?<\/p>\n<h3>In practice<\/h3>\n<p>Based on these characteristics the peer review system carried out for think tanks followed these parameters:<\/p>\n<ul>\n<li>The selected mechanism was that of a<a href=\"http:\/\/www.nature.com\/ngeo\/journal\/v6\/n6\/full\/ngeo1853.html\" target=\"_blank\">\u00a0<strong>double blind process<\/strong>.<\/a>\u00a0This process is considered by many as the golden standard of peer review. Although I am not personally convinced of this, it was chosen as a valuable addition to the processes that the centres already had in place since many had open processes.<\/li>\n<li>Each paper was\u00a0<strong>reviewed by two researchers<\/strong>: an external reviewer and a reviewer from a TTI think tank in the region.<\/li>\n<li>The system was opened to\u00a0<strong>four types of products<\/strong>: project proposals, book chapters, working papers and policy briefs. Each type of product had a different guideline for evaluation.<\/li>\n<li>We also accepted products that were\u00a0<strong>either manuscripts or final products for evaluation<\/strong>. In the first case, the authors still had a chance to introduce changes to their documents. In the second case it was more about evaluating the final product and receiving inputs for the future.<\/li>\n<\/ul>\n<p>Although we opened up all the possibilities, think tanks stick mainly to the usual suspects. We received mostly working papers (62%), book chapters (30%) and very few policy briefs (8%). We didn\u2019t receive proposals for review. Informal conversations with Executive Directors raised the concerns of sharing a proposal, and the possibility of the reviewer \u2018stealing\u2019 the idea, noting that time frames at the proposal stage are usually very short.<\/p>\n<p>Thematically, most of the papers received were on social issues such as health, education, poverty and inequality. Economic and fiscal issues were the second most relevant with two outliers: one on environmental policy and one on judicial matters.<\/p>\n<p>This is a very small sample of think tanks indeed. But it does open questions regarding the types of products think tanks produce and how they really differentiate themselves from universities and other knowledge producers. Although I wanted to test new spheres for peer review, in practice, think tanks still value the peer review process in its most traditional form.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Andrea Ordo\u00f1ez has edited a series about a new peer review mechanism for think tanks that the Think Tank initiative has piloted recently. She has produced a series of posts that explore the approach, consider possible lessons learned, and outlines recommendations for future efforts. In this post, Andrea outlines the key findings from a survey they conducted to inform the pilot. She finds that think tanks are more traditional than one would have expected.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","footnotes":""},"tags":[182],"acf":[],"_links":{"self":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/918"}],"collection":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/comments?post=918"}],"version-history":[{"count":0,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/918\/revisions"}],"wp:attachment":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/media?parent=918"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/tags?post=918"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}