Notes on methods and collaboration: Creating an online survey to assess think tanks’ performance

SERIES Articles and Opinions

In our last meeting in Jakarta, one of our main tasks was to narrow the scope of our subject of study – performance self-assessment – and identify the most important dimensions we would like to address. We came up with the following ones:

  • knowledge management,
  • quality control of publications and events,
  • strategic direction, and
  • organizational boards as part of the governance process.

The methodology we developed included an online survey, desk research and interviews with key actors in each think tank. The methods were chosen based on their diversity and appropriateness, but also due to ease, speed and low cost of collecting and analyzing information from several respondents. Moreover, a web-based questionnaire would ensure anonymous answers, thus resulting in more honesty from the side of the respondents.

Creating the survey together

In order to create the survey, we divided the four dimensions among the members of the group. Each one of us had to study one dimension and come up with a list of questions that would be useful for the survey and also guidelines for the desk research. In this process, the most useful resources were:

All questions and guidelines were compiled in two documents- one providing the basis to the survey and the other for the desk research.

We uploaded the questionnaire to the Google Forms platform and created a first version of the survey. With the purpose of testing this initial version, we sent the survey out to a small number of staff members from our organizations and asked them to provide us with feedback.

One of the main problems identified by the respondents was that it was too long and somewhat repetitive. To tackle these issues, we worked together to cut down the number of questions, and we managed to reduce the time it takes to answer the survey in half.

We also decided to change the host of the survey, from Google to SurveyGizmo, as the later offers more options regarding layout and format. This time around, we also created a Spanish version of the survey, to be shared with the Spanish speaking organizations.

Overall, the final version of the questionnaire turned out way more user-friendly than it started, and it was ready to be implemented (you can check the English version here and the Spanish version here).

Implementing the tool

Each team member coordinated the implementation of the survey at his or her organization . We each developed a brief description of the project, and introduction to the survey and an invitation to fill it, which was used to disseminate the survey and gather responses. In parallel, we worked on the desk research and interviews that complement the perceptions gathered through the survey.

Using these various tools presented different levels of difficulty. While the guidelines developed to complete a desk research and conduct interviews with key stakeholders were very useful and straightforward, the online survey tool presented some challenges :

  • The survey focuses on the perceptions of the organization’s members around the selected dimensions. Many respondents felt that assessing the organization’s performance through their perceptions wasn’t appropriate and were skeptical about the usefulness of the tool.
  • It was very hard to motivate people to completing the online survey. We used several reminders and friendly messages about the importance of completing the survey. Explaining to the staff the reasons behind the survey and the potential benefits of getting more knowledge about the organization definitely increased participation.
  • We found out that different members of the organization have different levels of knowledge on each dimension; some didn’t feel comfortable answering some questions as they felt they didn’t know enough about that topic. Respondents were able to leave a question unanswered but this was not explicit so many felt they had to provide an answer anyway.
  • The tool we developed didn’t include a requirement to log in, in order to keep responses anonymous, however this created the inconvenience of having partial answers that couldn’t be completed and having people sending in both a partial and a complete response.
  • Some respondents found the survey too long (two sections asked details for each possibility that was marked, each quality control process and each type of board).
  • We feel that in some cases respondents were biased to give positive answers as they felt they could easily be identified and could get in trouble if they provided honest feedback.

Nonetheless, the survey enabled the group to gain access to the subjective perceptions of the staff in terms of the four dimensions of organizational performance.

Untitled-4

The results will be paired with the data gathered through the desk research and in-depth interviews, which will give us an opportunity to compare the perception of the staff and the information gathered at the organization, as well as to the opinions of the managers.