Vox has recently published an excellent piece on The 7 biggest problems facing science, according to 270 scientists. This is worth reading for anyone interested in the broader question of the future of science and research. Here I provide a brief summary with a touch of reflection on how this plays out for think tanks and policy research centres.
Vox asked 270 scientists all over the world. They heard of how perverse incentives are affecting the sector and their careers in a way that may only result in bad science.
The 7 problems they identified were:
- Academia has a huge money problem
- Too many studies are poorly designed
- Replicating results is crucial — and rare
- Peer review is broken
- Too much science is locked behind paywalls
- Science is poorly communicated
- Life as a young academic is incredibly stressful
The money problem
OTT has dedicated ample of space and time to this issue: funding and supporting think tanks is at the core of our work. The long term trend we and Vox observe is one best characterised by a projectisation of research. Whether through very large (very large) grants that demand consortia and multiple layers of sub-contracting to be handled or small (very small) funding for consultancy-like studies that limit greatly the scope that most think tanks have to operate in, funders are, by and large, placing ever greater attention to using funds to control the research and the research-to-policy process.
While we welcome the attention given to funding research and communications (and not just research), we are concerned of the manner in which this is sometimes imposed. For instance by demanding, ex-ante, theories of change that become part of schedule of deliverables, or expecting, ex-post, that a think tank’s worth is evaluated on the basis of its policy impact.
Funding that is so closely linked to impact can only attract policy research projects that seek to please the status quo or that shy away from the larger and longer term problems societies face -the difficult, complex and wicket problems that cannot be solved through a log-frame.
Furthermore, project funding is likely to be uncertain and highly volatile overtime. It does not necessarily follow a single and clear policy research funding policy but the personal and profesional interests of funders (staff). Think tanks, however, need to make long term investments if they are to develop their capacity to meet the world’s greatest challenges.
One of the consequences of this approach to funding policy research is that it demands organisational as well as individual qualifications to get it. This limits the number of organisations and, consequently, researchers that can access funds. This is further compounded by efforts from some to focus only on PhDs. It is another reason why it is often so hard for new think tanks to emerge in countries awash with (aid) money for research: funders may be willing to take risks intellectually but not financially.
Vox suggests that besides increasing the amount of funding available funding needs to be more stable and transparent. It should avoid volatile project funding fads and allow researchers and research centres to make long term investment decisions.
An interesting proposal that could be taken up domestically is that national funding bodies could allocate funding through a lottery system.
Poor research design
Too many studies are poorly designed. Funding (dis)incentives are partly to blame. Demand for impact (to promise and demonstrate it) means that researchers are increasingly designing research projects almost entirely driven by the expected result.
However, a still fundamental aspect of poor research design is poor researcher competency. And this is partly down to the weakness of developing country university systems which fail to train new graduates in basic research skills.
Not surprisingly, Vox argues that funding ought to avoid encouraging positive results and could, in fact, praise negative results. This would go a long way in addressing a publication bias towards positive results that means that some of the most interesting questions are not even asked.
Additionally, greater efforts must be made to develop research skills across the board -starting in school and through university.
Replication is rarely done -where is the debate?
This is particularly relevant for think tanks in countries with a few foreign funders or working in unattractive sectors for funders. Rather than funding two or three studies on the same policy problem, we find that funders prefer to support a single organisation and/or study on it -and expect it will have an impact on policy.
In these cases, if a think tank is funded to undertake a study on, for instance, gender gaps in primary education, no other think tank will receive funds to look at the same issue. The effect is that policy is not so much informed as it is instructed by evidence -from a single source.
But replication is further hampered by a lack of transparency. Funders and think tanks are not prone to publish their data in an effort to allow others to replicate their studies. Some think tanks do (the Center for Global Development, for instance).
Vox suggests that carrots need to be dangled in front of researchers to pursue replication studies. Funders need to pay attention to this, too: aiming to fund alternative views on the same issue, for instance (see Rohini Nilekani’s interview in which she makes this point).
But think tanks should help each other by publishing their data alongside their studies and reports. This will help others spot mistakes and foster a debate on the issues that matter to them.
Peer review is broken
I was very surprised at finding out that a highly respected think tank did not have a peer review policy -and that, in fact, most of its work was never peer reviewed at all. Many think tanks do not use peer review mechanisms to address the quality of their work. And when they do, this is often done in-house or through their own networks of support.
This makes sense, if one thinks about it. Think tank need speed. They cannot wait for a proper peer review process to finish before they publish a report that is set to inform a very short policy window.
Vox reports on a more worrying trend: that peer reviewers are simply not doing their job.
OTT published a series on a peer review pilot based on a network model. This approach can help think tanks in contexts where there may not be enough independent experts on all policy issues and could address the disincentives for peers to carryout thorough reviews.
Too much research is behind paywalls
This affects the capacity of policy researchers -and think tanks in particular- to use the latest research to inform their own studies. Even the larger think tanks often lack subscriptions to the top journals.
This is a serious problem because it means that very few think tanks (possibly those with links to universities) are drawing research from the best resourced sources.
Efforts to open up academic publishing are gaining momentum. In the meantime, partnerships with universities and support from funders (e.g. IDRC offers its grantees access to journal data bases during the duration of the grant), will go a long way.
Policy research is poorly communicated to the public
Academia faces a growing challenge to unlock the science that is hidden behind paywalls. But think tanks face the challenge to communicate their work a lot better than they have done so far.
While this should apply more to think tanks in developing than in developed countries (here a focus on communications is greatly accepted) the latter are still underperforming when it comes to reaching out to the general public.
Policy research centres have become too obsessed with policy impact and their relationships with policymakers; so much so that their efforts to reach out to the public have been affected. The Brexit debate provides an excellent illustration of what can happen: Brexiters easily dismissed the evidence based arguments of experts. Experts, think tanks, and policy wonks have been discredited in the UK. Years of playing to the policymakers have taken their toll on their relationship with the public.
Communication with the public demands a broader approach to impact. This involves possible detours from the pathway to change that may see think tanks engaged in debates with a range of people and institutions who may or may not be driven by research-based-arguments. Rejecting such encounters has driven think tanks and experts to a corner in which they talk to each other but no one else.
Life as a young thinktanker is very stressful -and few make it; especially women
Finding a job in a think tank can be an attractive prospect but few young researchers receive the support their need to nurture their skills and further their careers. The life of a young researcher in a top think tank (where they are more likely to learn the skills they need to pursue a career in policy research) is hard. We carried out a short study of young thinktankers in Peru in 2015.
This was complemented by a longer series on women in think tanks. What we found was that the business model of many think tanks works against young researchers and women in particular. To succeed they must adopt behaviours and draw support from networks that are only available to a few better off individuals.
This does not promote diversity in research centres and has negative effects on the quality of their research and their engagement with the public.
To address this think tanks must actively seek to hire and retain individuals from different backgrounds. More formal recruitment practices and avoiding hiring through private networks can help.
But they will also have to reflect on their business models to ensure that they are able to accommodate individuals with different needs -women, to begin with, who will have to carry the burden of family care at some point in their careers.
Funders, too, may be able to address this by supporting individuals over a long period of time at the start of their careers -those who already fund think tanks through programatic or core grants could earmark a portion of these to support the development of new generations of thinktankers.