August 8, 2015

Publication

Methodology Brief: Qualitative Survey Analysis

1) Basic Information

Method: Quantitative Survey Analysis

Author: Courtney Tolmie is a Senior Program Director at the Results for Development Institute (R4D). Since joining R4D in 2007, Courtney has led the Social Accountability and Think Tank Support work at R4D. In her role in leading R4D’s think tank programs, she has been the technical lead for the Strengthening Institutions program (led by the Global Development Network, supporting fifteen think tanks globally) and advisor for the Think Tank Initiative’s Policy Engagement and Communications program for Anglophone Africa and the publication of Improving Think Tank Management (Raymond Struyk). Most recently, Courtney led the R4D-based team undertaking a mixed-method research study on the links between context, think tank decisions, and performance. She is co-author of Lives in the Balance: Improving Accountability for Public Spending in Developing Nations (Brookings Press), From the Ground Up (Brookings Press), and Using PETS to Monitor Projects and Small-Scale Programs (World Bank). Courtney graduated from Bowdoin College (B.A. Economics) and holds a Masters of Arts in Economics from the University of Virginia.

 

Contact: [email protected]

2) Description of method

Objectives. The objective of quantitative survey analysis is to leverage large comparable datasets to identify trends in think tank characteristics, actions, and exogenous factors. In this application, I use “analysis” to represent a set of methods that can be used to analyze large datasets, including regression analysis and even running summary statistics. This is an important set of methods to discuss, especially in think tank research, because there have been limited opportunities to analyze quantitative comparable data on think tanks at a large scale. Quantitative survey analysis gives researchers the potential to investigate the generalizability of findings that may initially be identified in comparative case studies or other qualitative techniques but that cannot be assessed at scale due to the limited number of cases that are often researched when using case study methodologies.

Description of Method and Practicalities. The major requirement for quantitative survey analysis is a dataset with comparable data on diverse think tanks. One of the reasons that quantitative survey analysis has been used so infrequently in think tank research is that there is the lack of representative surveys and thus large-n datasets on global think tanks. As part of our recent research Linking Think Tank Performance, Decisions, and Context, Results for Development sought to overcome this gap by designing and implementing an online survey of think tanks in low- and middle-income countries that included questions about organizational characteristics, human capital and staffing, finances, performance measurement, political context and related strategies, and exogenous factors (such as civil society, media, and NGO environment). This survey was a unique attempt to collect information from up to 400 diverse organizations that would have allowed us to test hypotheses regarding relationships between these characteristics, exogenous factors, and organization strategies. These hypotheses regarding trends are difficult to answer using other methods because statistically significant results require a large enough sample to allow for confidence that observed relationships between different factors are not due to unobserved heterogeneity or other underlying causes.

The experience of fielding this survey demonstrated many of the practical challenges related to undertaking quantitative survey analysis. While the quantitative analysis itself is potentially powerful and not incredibly challenging, it does rely on a high-quality dataset from a representative well-designed survey with a large-enough response rate. In the case of our survey (the only of its kind that we could identify), we received a response rate of approximately twenty-five percent (or 94 think tanks). Of the 94 responses received, many of the responses were incomplete, which resulted in much smaller samples for many of the regressions that we sought to run. As a result, we largely utilized tabulations and cross-tabulations as a means for analysis. Cross-tabulations allowed our team to identify where there were statistically significant relationships (and the direction of those relationships) between factors; for example, our analysis showed that think tanks based in countries with higher level of development have a lower fraction of funding that comes from unrestricted core funding sources. This is an interesting finding that could only be identified with comparable cross-country data; however, without having the data quality or sample size to be able to undertake a more rigorous regression analysis, we are unable to say anything about causation or if there are underlying factors that could explain this relationship.

As a final point, this study’s cross-country survey and quantitative analysis was part of a larger mixed methods study that allowed our research team to answer questions that are not best suited for quantitative analysis and are better suited for methods such as comparative case studies and participatory research. I would suggest two take-aways as part of our experience with this study. First, the think tank research field could benefit enormously from high-quality large-n quantitative analysis; however the challenges in developing a dataset to undertake this type of analysis are significant. Second, any single method for research can only answer a sub-set of interesting and related research questions; as such, mixed methods research provides a valuable means to holistically consider the most interesting questions regarding think tank decisions and performance.

3) Further Resources

As noted above, Linking Think Tank Performance, Decisions, and Context (Results for Development) is the only study that we know of that undertakes a mixed methods approach to think tank research that includes an analysis of cross-country quantitative survey data.