What do we rate?
Transparify rates the extent to which think tanks publicly through their websites disclose who funds them with how much to do what work.
What do we not rate?
Transparify does not rate think tanks’ transparency about their expenditures, only about their income. Transparify also does not rate the transparency of the funding sources themselves.
What does the 5 star rating system tell us about a think tank?
Transparify visits think tanks’ websites and awards up to 5 stars. Think tanks that score 5 stars are highly transparent about their funding. Think tanks that score 0 stars provide no up-to-date information on where they get their money from.
5-star: highly transparent — Think tanks that score the maximum possible 5 stars enable fellow researchers, journalists, policy makers and citizens to see clearly and in detail who funds them, how much each donor contributed, and typically details what projects or activities (if any) that money went towards. .
4-star: broadly transparent — Think tanks that score 4 stars are largely transparent, but the information they provide is less detailed or comprehensible. Outside stakeholders can infer who their main donors are.
1, 2 and 3-star: incomplete funding data — 1-3 stars mean that a think tank only provides some data. For example, a 2-star think tank may list donors and some financial information, but keep contribution levels obscure.
0-star: no funding information — 0-star think tanks do not provide any up-to-date information on where they get their money from.
How does the rating process work?
Two or more raters assess each institution independently from each other, awarding between zero and five stars according to the type and extent of financial information available on their websites. Raters enter their assessment into a Google Form. They do not know which other rater assesses the same institution, and work from different lists.
Our raters are individuals with completed university degrees. They represent the kind of engaged citizen that should be able to find funding information on a think tank website. We recruited the raters from the broad cross-section of young researchers we have worked with, trained them via a PowerPoint presentation, and provided them with a standard protocol to follow when searching for financial data online. We tested all new candidate raters on calibrated ratings to ensure that they returned reliable results.
Which rating criteria did Transparify use?
The criteria for the number of stars to award are clearly defined. The specified criteria used by the raters were:
- 5-star: all donors listed and clearly identifying funding amount for, and source of, particular projects
- 4-star: all donors above USD 5000 listed in 4+ precise funding brackets, anonymous donors no more than 15%
- 3-star: all or most donors listed in 2 or 3 broad contribution brackets [i.e. “5.000 to 15.000 USD, the following donors”)
- 2-star: all or many donors listed, but no or little financial information,
- 1-star: some donors listed, but not exhaustive or systematic
- 0-star: no relevant info
- Other: detailed financial information available but no donor list
Institutions who provided their last funding information three or more years ago (for example, 2011 reports by the end of 2014) received 0 stars because whatever information they provide is significantly out of date. We made some allowance for listing anonymous donors, as there can be pre-existing commitments. Note that we may revise criteria for future rounds of ratings, so do follow us to receive updates.
How did you deal with borderline cases?
An experienced external adjudicator was recruited by Transparify to review and resolve any discrepancies in the results returned by two raters and determine the final score.
In the ratings for 2014 and 2015, the overall gradation of categories worked well, though as in all quantitative research there can be challenges of identifying exact cutoffs, especially when relevant information is distributed over various webpages, or when it is unclear how comprehensive disclosure actually is. In such cases, adjudication contributed to the integrity of the process. In this we follow established practice in evaluation. The emphasis on process has parallels in jurisprudence which has a long tradition of reflecting on judgment.
How reliable are the rating results?
The rating system draws on the professional experience of the Transparify team, gained while overseeing 100+ research projects, including dozens of countrywide surveys, for more than 30 donors across numerous countries. The team thus has extensive previous experience in operationalizing research.
The rating system was successfully tested on a sample of prominent think tanks from around the world in 2013 before we rolled it out. The results showed that the system can be applied across a wide range of think tanks.
The use of two trained raters following a standardized protocol independently from each other, combined with the involvement of an external adjudicator to review and resolve any discrepancies between raters, gives us high confidence that our results are correct. If you think we should review a rating, let us know.
What if the raters overlooked crucial data?
It is theoretically possible for two raters and an adjudicator to miss information tucked away on a website. In this case, though, a typical engaged citizen would also struggle to identify who supports an institution. Online treasure hunts can be fun, but don’t qualify as transparency.
Can I check up on your rating results?
Yes. Anyone can visit the website of any a think tank we rated and compare the information provided there against our rating criteria. Thus, the results can be replicated by any interested third parties.
Can this system be used for other organizations as well?
The Transparify rating system can be used to assess additional think tanks currently not covered by us, or other organizations such as NGOs and advocacy organizations. The basic questionnaire is available here, and can be used freely (though we’d grateful if you let us know). Check our 2015 report for an overview of how others have used Transparify’s method to rate nearly 200 additional institutions, around the world.
What are Transparify’s future steps?
We will expand Transparify’s ratings in the future, to cover more institutions. We are also considering a review of some of the criteria. Any changes will be communicated via our blog. Sign up for regular updates via Email, Facebook or Twitter to get notified when we release our assessment results. Questions or suggestions? Let us know.
We have been rated and want to improve our ratings. How do we do that?
We have an explanation on how you can obtain a five-star rating, in a few steps. Check here.