Insight into factors affecting perceptions of performance in policy communities (Part 2)

14 February 2014

 [Editor’s Note: This post was written by Philippe Martin, formerly Research Awardee, Think Tank Initiative (TTI), IDRC.]

The first part of this blog post provided a short background on a study exploring key factors affecting perceptions of think tank performance in policy arenas. Empirical analysis was conducted using data from the Think Tank Initiative.

Here are some of the most interesting findings as well as questions for further investigation.

  1. Funding composition affects performance assessment. Institutions that enjoy a high share of funding from domestic sources are more likely to be rated as high performing by national policy stakeholders. The study also found that the effect of foreign funding on performance assessment varies across regions. In South Asia, foreign funding seems to negatively affect the way that think tanks are perceived. Nonetheless, in Africa and Latin America, the relationship between international funding and assessment is positive: a higher share of funding from abroad increases the likelihood of being rated as performing. But why is that so? And what’s happening in South Asia?
  2. Researchers need to better communicate their findings to policy makers. The analysis shows that the odds of policy actors providing a negative assessment of think tank performance are between 2 to 3 times higher if the organization held targeted meetings with policy makers in the last year. This intriguing and counterintuitive finding provides empirical support to claims that some think tank researchers are not particularly effective at communicating their findings in a brief, clear and convincing way to non-researchers. An interesting question would be what effect, holding research quality constant, communications efforts have on reputation.
  3. Context matters for performance assessment. An interesting finding of the study is that the data shows a negative relationship between the indicator of government effectiveness and think tank performance assessment, both within the global and the Africa samples. A possible explanation for this negative relationship is that, as the quality of the governance context improves, policy actors have heightened expectations about what think tanks ought to achieve. Yet at the same time, respondents who feel optimistic about the policy making process in their country are more likely to provide a positive assessment of the work being done by surveyed think tanks. These results warrant additional investigation.
  4. Familiarity with a think tank greatly affects respondent attitudes. For respondents with a high degree of familiarity with a think tank, the odds of providing a high performance rating are between 2 and 11 times higher. This effect is robust and positive across regions. This probably is an example of the’ revolving door, small clique’ effect in action in small policy communities where people know, help and protect each other. Although expected, this finding also points to potential response bias in the Policy Community Survey.

Limitations and Final Remarks

While these findings provide food for thought, it is important to underline the limitations of such a quantitative research design. First, statistical methods provide breadth versus depth. In this case regression analysis involved a significant dose of conceptual simplification. Second, the study is based on a non-random sample. The TTI sample is not representative of the population of think tanks/policy research organizations in the developing world, which means that findings cannot be generalized beyond the TTI cohort. And without a strong sample, it is difficult to establish cause-and-effect relationships. Third, there are also limits to making valid comparisons across countries and organizations given the diversity of organizations and country contexts considered.

Additional data and a more refined conceptual modelling would certainly further the usefulness and applicability of this study. In particular, data on organizational internal features such as leadership, management, governance, and workplace dynamics, would allow drawing a more complete picture of think tanks. However key factors, such entrepreneurship and innovativeness, are arguably hard to quantify. These challenges are inherent to the complexity of studying thinks tanks in a comparative fashion.

The full working paper is available here.