At this year’s OTT Conference in Rabat, I co-facilitated a session on think tank impact measurement and was interviewed for the podcast of our hosts, the Policy Center for the New South. The Policy Center had sent me four questions in advance, all on impact: how the notion is evolving for think tanks, how we should adapt to a changing ecosystem of influence, which methodologies and indicators still matter, and what AI is doing to all of the above. I did not write my answers down. This reflection is my attempt to reconstruct them, with the benefit of hindsight and of the conference conversations that followed.
What counts as impact does not travel well
Any honest discussion of impact has to start with the sector’s broad typology. Think tanks come in different shapes and sizes, if you will. They operate in different contexts and environments, specialise in different domains, and run on different operating models and structures. What constitutes impact for one think tank may not translate to another.
My organisation operates in the Washington, D.C. space, arguably a unique context with its own very specific challenges and “rules of engagement.” We are a relatively young organisation, and how we think about impact has evolved over the years. I come from an international development background, with almost 18 years of experience at the World Bank and the European Commission. So when we started our organisation and began to think about monitoring, evaluation and learning (MEL), our first instinct was to go with what we knew best: the approach adopted by the World Bank and other international development organisations. Essentially, the textbook logical framework methodology. You start by defining the problems you mean to address, analysing their root causes, converting them into objectives, and then converting the objectives into a logical framework with activities, outputs, outcomes and impact. A methodical approach to identifying and codifying impact that frankly works well in international development. As it turns out, it did not translate cleanly into our work as a DC-based, boutique, niche, single-donor-funded foreign policy think tank that is also a school.
So we had to iterate and improvise. We looked at what other older and wiser think tanks do and considered a few approaches. One treated impact as a series of consecutive stages with increasing weight. First comes outreach, where your work is mostly raising awareness. Next is engagement, where audiences begin to engage with your work through events, citations and invitations to policy discussions. Next is action, where your work is included or acknowledged in relevant policy discussions. And finally, implementation, where your work directly leads to legislation or policy change. That approach was adopted at one point by an old and highly respected think tank based in the UK whose name rhymes with “Stratum Mouse”. It did not work for us at the time. But it did inform how we thought about our projects in terms of their structure, scope, and the competencies and resources needed to implement them.
If you build it, they will not necessarily come
We continued to iterate and, with nothing else to fall back on, went back to the World Bank model and made a big push to implement it. In hindsight, that was an exercise in futility for several now-obvious reasons. But at the time, our thinking was: maybe it is a matter of building the infrastructure. The templates, workflows, systems, guidelines and training materials. If you build it, they will come, and so on and so forth. So we went all in and built it all.
No one came.
The lesson was that this sort of thing has to be designed and built around actual behaviour. For a system to work, it has to emerge from existing practice and, ideally, add efficiencies and deliver results greater than the sum of the parts. We were not there yet. As a start-up organisation still grappling with our operating model, our organisational structure and our internal processes, let alone figuring out who does what, our practices were quite varied and very much in beta. So we took a few steps back, let practices settle, and then designed around them.
That is where we are now with the approach we presented in Rabat, a blend of what we have learned thus far. It is not a finished product, but it is the approach most aligned with what we do day-to-day. Perhaps other think tanks operating in similar contexts will find it useful. We have shared our presentation with the organisers. Please feel free to reach out if you would like to discuss or share your own experience.
Impact as a process rather than a point in time
Against this backdrop, a helpful way to think about impact, for us at least, is as a continuous process that often goes beyond pre-defined metrics, rather than a single point in time. For a policy think tank such as ours, a lot of upstream work has to be done before a policy report is even commissioned. This may go against conventional wisdom, and it certainly does not follow the linear project design approach described earlier, but we have to identify the specific policymakers who would be the main audience for a report before that report exists. We ask about their interests and motivations, what their sphere of influence looks like, who else is in their network, and how likely they are to support the work. We have to land on answers that align with our principles, mission, priorities, competencies and resources. Then we consider the range of products and activities to design and roll out, including events and campaigns. Then come messaging, positioning, rollout locations and platforms, media channels, languages, and timeline.
After all of that is done and the “product” has been launched, the real work begins. That is where influence and impact become a process, rather than a checkbox. In other words, the drive to impact starts downstream, and it does not stop. We keep going, check as many boxes as we can (media exclusives, policy adoption, shaping legislation, invitations to provide policy briefs to high-ranking officials and diplomats, new institutional partnerships, speaking engagements, repeat business), and then check some more.
Analysts as all-terrain vehicles
All of this has consequences for the people who do the work. A significant cross-functional skilling is now necessary for think tank analysts. Traditionally, analysts were trained as subject-matter experts who focused primarily on their areas of research and analysis. They keep their heads down, go deep on their topics, and produce publications or outputs that advance specific policy positions, make recommendations, or enhance understanding of those topics. But the domain has become increasingly non-linear. Impact and influence are no longer just about the quality of the research, or the relevance, feasibility and actionability of the policy recommendations. They are increasingly about building and managing relationships, communicating in languages and formats that appeal to wider audiences, negotiating contracts and legal agreements, and executing and steering projects across shifting terrain. On top of that, analysts continue to spend significant time raising funds and seeking institutional partnerships in a highly competitive funding environment.
Essentially, analysts are increasingly asked to become jacks of all trades, expected to be competent in, if not master, multiple roles and functions that were traditionally handled by others: the communications team, the MEL team, the development team, and the operations team. Analysts are becoming less of a two-axle vehicle and more of a 4×4 hybrid amphibious crossover copter that performs well across all terrains.
Think tanks, therefore, need to rethink their operating models, organisational structures, recruitment and retention strategies, approaches to revenue generation, and, perhaps most importantly, how they manage and develop their teams. Think tanks must become more agile, adaptive, and responsive to change because the pace and magnitude of change will likely increase over the coming years as everyone grapples with AI and continuously shifting, unpredictable political dynamics.
Strength and coverage: a network view of measurement
If think tank influence has become more networked, we might as well adopt the relevant terminology from the telecoms industry. In my view, one of the most important measures of a think tank’s impact is the strength and reach of its network, alongside the organisation’s ability to leverage it successfully and efficiently to effect the desired change. This refers to all upstream and downstream work highlighted earlier. Think tanks are different, and what works for some may not work for others. But the network metaphor can be quite helpful in thinking about impact measurement.
In that vein, I am reminded of the work of the late Jim McGann of the Lauder Institute’s Global Go To Think Tank initiative. Many years ago, when Jim was still among us, and when my organisation was still grappling with MEL, we wanted to learn from what others were doing. We found a gem of a document published by McGann’s initiative that was immensely helpful. I searched for it while writing this but could not find it. What I did find was a file I had saved that includes some of the document’s contents. I want to share it here because I think (and hope) many colleagues and friends within the OTT network will find it useful in thinking about impact and how to measure it. Much of what is in that document remains relevant today and will remain relevant for some time. The document is available here.
The human in the network
A lot is being said now about AI and how it will influence the way we work, structure operations, and measure performance. The dust is far from settled on this, because what AI can do today, which is already overwhelming, pales in comparison with what it will be able to do in a few months, let alone a few years, barring any meaningful global AI policy backlash.
One point of consensus echoed throughout this year’s conference: the importance of the human. Not only as gatekeeper, authorizer, approver and quality assurer, but as carrier, custodian and representative of the organisation’s work, output and reputation vis-à-vis the organisation’s network. Erica Schoder’s keynote made a version of this argument, and so did many of the participant reflections collected at the close of the conference. Human-to-human interaction remains the central domain where impact occurs, in its multifaceted and multidimensional forms.
I think that will remain true for some time. I am just not sure for how long. Something to revisit at next year’s OTT Conference.
Kareem Makhlouf is co-founder and Chief of Staff of the New Lines Institute for Strategy and Policy in Washington, D.C. He is also author of Thinktankering on Substack.