Devising, revising and evaluating strategies for funding organisations: 10 lessons from OTT’s work

22 July 2021

with contributions from Julie LaFrance, Simon Hearn, and Enrique Mendizabal

Over the last couple of years, OTT has provided services to large funders to develop, refresh and evaluate strategies in areas as diverse as economic justice; scaling research impact; transparency, participation and accountability; and others. It has been fulfilling work and we’ve learnt lots in the process. Here are some of those lessons, targeting both organisations who are commissioning such work and the consultants who have been selected to conduct it. These will have implications on how these efforts are budgeted for, managed, communicated and resourced – in terms of personnel.

1. Draw and regulate boundaries around the work

Developing, refreshing and evaluating large strategies can be time and resource intensive. If clear boundaries are not drawn around the work (in terms of key questions to answer, time, specific tasks), members of the consulting team will experience stress, their ability to think clearly and learn effectively will be curtailed and the work may not be completed to the required standard. Setting boundaries will require potentially difficult choices to be made early on. But once boundaries are set, they will need to be ‘enforced’ or at least negotiated on an ongoing basis. This requires having potentially difficult conversations, where both parties will need to ‘walk the talk’ of embracing differing perspectives and accept work that is ‘good enough’. Commissioning organisations with their control over resources will need to acknowledge the power they will have over the consultant during negotiations.

2. Agree on the roles that are taken up by the commissioning organisation and the consultants

In our experience, the commissioner–consultant relationship becomes strained if parties have not been explicit about the respective roles they expect each other to play and if/how this might change over the course of the assignment. Respective roles can be clarified by highlighting the purpose of the consulting relationship. For instance, will consultants provide information and generate reports? Will they help the commissioner to make decisions (e.g. about areas of focus), especially if there are divergent views within the commissioning team? Or a combination of the two (and if so, to what extent)? In addition, clarify who has the authority to make decisions, especially within the commissioning team. If decision-making authority is distributed within the commissioning organisation, sufficient time will need to be set aside to enable consensus to be reached.

3. Decide how the task is managed

A comprehensive refresh of a strategy can potentially comprise several tasks and sub-tasks featuring research, evaluation, communication and engagement on different approaches, themes, contexts, processes and stakeholder groups, which might each be led by a different person. With a potentially large team for the consulting team leader to manage, decisions will need to be made on who engages with the commissioning team about what, how many and what sort of outputs are produced and who authors them, bearing in mind that commissioning teams often have limited bandwidth for this type of engagement.

For instance, if engagement with the commissioning team is limited to the consulting team lead, conversations may be brief and focussed, but the former may miss out on potentially interesting insights, while task leads may feel a sense of exclusion. If on the other hand, all task leads engage directly with the commissioning team, there may be a greater sense of cohesion and coordination, but meetings may take up a lot of time and could result in meandering conversations. In relation to outputs, options include a range of outputs authored by different task leads, a longer multi-authored output or a focussed synthesis authored by 1 or at most 2 authors, which draws on multiple outputs produced by task leads. Trade-offs between coherence and focus on one hand and detail and complexity on the other will need to be negotiated and managed throughout the process.

4. Be flexible and adaptive

Like policy processes, strategy processes are rarely linear. Avoid spending a lot of time planning tasks in great detail. Doing so might make you feel better but plans will soon get pulled out of shape due to changing circumstances. Take up the option of extending (or shortening) particular phases of work (albeit within the set boundaries). And resist the temptation to split a strategy refresh process too rigidly into a ‘look back’ and a ‘look forward’ phase. During a look-back phase, you may need to look forward (e.g. to identify the scope of the evaluation and specific case studies), while during a look forward there may be a need to look back (e.g. to scope out particular areas of interest in more detail).

5. Be clear about decision-making criteria

Strategy-related decision making rarely takes place in a political vacuum and with a blank slate. Although you may consider technical criteria such as the potential for an area to have ‘significant impact’, to ‘address structural inequality’, or to provide ‘additionality’, you may well be under pressure to take up new areas, continue with old ones, or drop certain areas due to senior management imperatives, expertise among current staff, available funding, path dependency issues, and the need to protect existing relationships. These criteria may be more implicit. However, if these issues are not explicitly named during discussions, relations among those involved can become strained and the integrity of the process will be called into question. We suggest being clear about the criteria that inform decision making, asking whether there are any ‘sacred cows’ in terms of relationships, projects, areas, or countries that need to be honoured, and providing space for these to be openly contested by those involved.

6. Strategies should not be prescriptive

A strategy aims to set an end point and a direction, without necessarily defining how to get there. Even if there is a theory of change, it may be abstracted to the point that it is not useful in guiding the strategy’s delivery. We suggest avoiding a focus on detailed planning and rigid disbursements and instead aiming for a broad framework with soft targets. This provides grant makers (and grantees) with the flexibility to use their practical judgement, experience and intuition in the face of new information and ever evolving contexts. We suggest that developing more specific (and useful) theories of change is the responsibility, not of funders/commissioners but of grantees and recipients of funding. Given that the best plans often get pulled apart by reality and unintended consequences, commissioners should consider having more frequent but less intensive strategy planning/revision processes.

7. But this makes evaluating strategies challenging

Given the intentional vagueness of a strategy, there won’t be clear boundaries about what is in and what is out of scope. There may well be a large amount of diversity (in terms of contexts, approaches and outcomes), which makes evaluating strategy in a conventional way difficult. Evaluating the full population or even a representative sample of grants will be challenging, making it hard to answer ‘how well’, ‘how effective’ and ‘to what extent’ types of questions. For instance, you may obtain a ‘flavour’ of emerging issues and outcomes by undertaking case studies focusing on certain areas of work but this won’t necessarily help you answer general questions about effectiveness. The commissioning organisation and consultant will need to discuss how they address these limitations early on during an inception phase.

8. Focus on processes as well as outcomes

We all know about the importance of outcomes and results. However, outcomes are usually well beyond the control of any one actor, let alone grantees. During an evaluation, grantees and their partners (in and out of government) may feel the pressure to make bold claims, despite limited evidence, to ensure continued funding. So in addition to holding outcomes as a standard measure of effectiveness, we would emphasise reviewing the quality of the process pursued to achieve those outcomes. This means reflecting on the skills and qualities of personnel from the commissioning organisation and grantee. Key questions include: are staff curious and humble about what they know and don’t know? Are they reflective about how change is or is not happening and why, and reflexive about their own role? What rules of thumb do staff use to make decisions? This also means exploring whether commissioning and grantee organisational practices promote risk taking, encourage staff to experiment, and embrace new ideas and thinking outside of the box.

9. Build in spaces for dialogue and containment

Strategy refreshes and evaluations are difficult times for all concerned: funders, their grantees and hired consultants, among others. For instance, grantees will hold significant information about the context and how funds have been used. They will be concerned about whether they will continue to receive funding and the implications for their myriad partnerships. Funders will be faced with having to make difficult decisions with limited information in tight timeframes. Staff might be worried about their future in the organisation or they may well disagree with each other about a particular course of action. Consulting teams will be under pressure to familiarise themselves with the relevant content, have the right expertise, produce robust analyses and communicate findings on what might be a vast set of topics in an accessible way within short timeframes. All of this generates stress.

Frequent dialogue and communication between and among the funder, hired consultants and grantees is vital. Creating spaces for different parties to dialogue with themselves and with each other and to ‘contain’ the energy that various actors generate together during the process will be key in growing trust, identifying and addressing difficulties that arise and getting the work done on time and to a ‘good enough’ standard. But all this costs. Set aside sufficient budget, find appropriate expertise and put in the time at the start and throughout the process to think through (and act on) appropriate communication practices.

10. Communicate content and key decisions but acknowledge that this is just the beginning

Strategy planning, revision and evaluation processes will produce and result in a stream of important content and strategic funding decisions. These will need to be communicated to parties beyond the commissioning and consulting teams, to other units within the funding organisation, to other funders, to grantees, potential grantees and other important actors. Publishing long documents on a website is unlikely to engage key audiences. Summary articles may need to be published on Medium, for instance; more creative approaches to communicating content may need to be pursued; webinars and launch events may need to be facilitated and documents may need to be translated into other languages. Once again, responsibilities will need to be allocated (some of this will fall to the commissioning team to take up), budget will need to be set aside and relevant experts (such as editors, graphic designers, strategic communication specialists and translators/interpreters) will need to be hired. In any case, communicating decisions to a broader audience is just the beginning of translating those decisions into practice. New or revised strategies may mean developing new relationships while ending old ones. Either way, funders will need to be patient and acknowledge that in any case the best plans lead to a combination of the intended, unintended and unwanted.