On the KSI pilot evaluation and lessons for the programme implementation

23 April 2014

[Editor’s note: This post follows from a series on supporting think tanks originally commissioned to input into an evaluation of a series of pilots that pre-dated the Knowledge Sector Initiative in Indonesia. It is slightly long overdue.]

Ben Davies, from DFAT Australia, wrote and published a management response to the review conducted on the KSI pilots evaluation. I was asked to contribute to the evaluation via a series of think pieces (I should clarify that I was not a member of the evaluation team -more on this later).

The evaluation offered a number of key findings and recommendations. I offered some of my own via the synthesis published a few weeks ago.

The management response welcomed the findings and recommendations. So what did it agree to? (but  should it have agreed to it?)

Empower the grantees (I am not going to call them partners as in the evaluation report or how KSI refers to them. I feel that we need to be clearer about labels. A partner should reflect an equal relationship. Think tanks need clarity in their relationship with their funders (this was one of the key lessons from the think pieces). So, my labels for the KSI programme are: DFAT (Australia) is a funder, KSI is a contractor providing funding on behalf of DFAT, and the think tanks are the grantees.): empowering the grantees goes well with the KSI’s demand driven approach. But what does ‘demand-led’ really mean?

I have written many times in this blog that demand-led approaches do not fit well with international development funders and agencies with a results-driven agenda. It is hard to be patient and let (or wait for) ones’ grantees to take the initiative when there are millions (and millions) waiting to be spent. It is hard for the funders and it is even harder for the contractors who have to worry about the bottom line. The industry set-up works against this concept.

Demand-driven approaches can only be empowering if the think tanks are solely responsible for taking advantage of the opportunities provided to them by the funders (directly or via contractors). This does not mean that all grantees must be able to put together the best (or perfect) proposals for institutional funding or capacity development efforts; but they should be, at least, all capable of demonstrating a clear and heartfelt commitment and willingness to improve.

Since not all think tanks are equal, demand-led poses another challenge: will the contractors be able to manage 10, 11, … 14 different ‘demands’? Will DFAT be comfortable allowing its contractors to ‘test’ different ways of supporting think tanks in Indonesia? Will it be comfortable if 1, 2, .. 10 think tanks fail to take advantage of the opportunity and are dropped form the programme (and possibly forced to fold)? Demand-led means being open to the possibility that there won’t be any demand.

And will, and this is the tricky one, the funders and the contractors able to change their programme if their grantees ask them to? Would they be able to handle a demand for a significant change in the way the KSI is organised, how it funds and supports its grantees, and even its objectives, if such demand came from its grantees?

Improve monitoring and evaluation: The Evaluation Team recommended that each grantee should have an M&E system and should seek help from the KSI contractors. KSI responds:

The KSI Implementation Team have made sure to properly resourced M&E in KSI, with two international M&E experts overseeing the design of the system for the overall program, and a dedicated M&E Manager and two M&E Support Officers to administer the system. One of the priorities for the supply side component is assisting partners to develop simple M&E systems beginning with a self-organisational assessment in the first year of the program. Improving M&E is also a priority for KSI more broadly – the KSI Implementation Team is developing its Technical Support Pool which will also include M&E experts.

I am not sure about the kind of M&E that is expected of think tanks in developing countries. It is far too onerous on the think tanks themselves and demands efforts (skills, resources, time) that would never be expected of a think tank in a developed country by its funders.

The emphasis ought to be on monitoring for management -which is management, really, not M&E. Good governance and management allow think tanks to ‘pre-empt’ the need to develop M&E systems that burden rather than help them.

But if they will be asked to develop M&E systems then my advice is that these should be kept small and simple (with a view to growing in complexity as the organisation learns to use them and focusing on inputs and outputs first -what is the point of evaluating outcomes if the work is of poor quality?); they should help the organisation and not the funders (if the funders want to assess the effect their grants are having then they or their contractors should worry about their own M&E systems and not bother their grantees with that); and they should not rely on one-size fit all approaches or templates (I am yet to come across two think tanks that can be evaluated in an entirely identical way).

And a note on self-assessments: it is not the best idea to start with these when money is still being negotiated. It is much better to work with the grantees to encourage them to seek independent views about their organisation as a starting point of the funder/contractor/grantee relationship. It might take longer and may be more expensive but it will be more honest and worthwhile.

(It is a good thing I was not part of the Evaluation Team: It means I can change my mind and even disagree with it.)

Help think tanks to improve their knowledge products and services: This was always going to feature in an initiative like this one. The role of mentors as well as the role of workshops are highlighted by the management response. Both can be effective tools to develop a think tank’s capacity but it is important that we understand their limits:

  • Workshops are better at raising awareness about a topic or to learn and practice a particular method or technique. Trying to do both usually fails. It means that the participants are either not convinced enough or not in the position to do something with what has been learned.
  • Mentors are better when they are mentoring based on their own experience. Consultant-mentors or facilitator-mentors are really not mentors. The best think tank mentors are found in think tanks. And for developing country think tanks these are think tanks dealing with mainstream politics in more successful think tanks or in more developed think tank communities -not in, I will not get tired of this, international development think tanks or communications consultancies.

A much better approach to helping think tanks to improve their products and services is to encourage them (maybe via mentors or consultants) to develop a portfolio of channels and tools and then practice them as much as possible. This can certainly benefit from exposure to think tanks in other countries. Australia should not shy away from bringing Indonesian and Australian think tank communities together -but it could also encourage links to China, India, other regional think tank hubs and other G20 countries, to begin with.

Pay more attention to learning what works: This does not only mean ‘testing the grantees’ but also testing the hypotheses that the KSI is operating under, the KSI’s governance arrangement, its actions, etc. KSI has to be capable of honest and open critical thinking. In my experience, grantee think tanks are the first to see through the jargon and any pretence that ‘everything is alright’. If something is not be working as expected it is best to question it, publicly, right away. This is, after all, an experiment. Nobody has tried it before.

Learning what works also means seeking experiences elsewhere. The KSI has the opportunity and the responsibility to engage with other initiatives to learn from them and with them. But this will demand more open communications from the programme. The more you give the more you get.

More importantly, still, it should encourage its grantees to take the initiative in doing this kind of research themselves. It is all very well to hire third-party experts to do research for the initiative but it would be much better to give that opportunity to the KSI grantees. The Think Tank Initiative has given its grantees some opportunities to undertake research on think tanks (see for example, the Universities and Think Tanks research projects in Latin America and South Asia) but this rol has fallen, for the most part, on external researchers and consultants. I’d prefer to see more think tanks (grantees or not) in developing countries carrying out this very important supporting research.

Here, it is important to differentiate between the funder, the contractor, and the grantees. They all have something to learn and all should be encouraged and feel free to do so. It may not be possible, of course, for DFAT to be as candid as a think tank, but it should instead allow its grantees to take on that role. And it must ensure that its contractors, the KSI, feel free to challenge its own original assumptions: after all, only Gods and fools do not change.

Strengthen policy advocacy: The recommendation suggests that the grantees should explore their own approaches to influencing policy and learn from their own experiences as well as from others’. In their response, DFAT recommends:

For example, helping partners develop a policy influencing plan could be an activity that is chosen to help strengthen their ability to have their research informing policy decisions.

Think tanks should not have to develop influencing plans. It seems odd to suggest this. Think tanks are all about influencing. Everything they do should be geared towards influencing -or attempting to influence. Rather than developing plans -or strategies- then the grantees should evaluate their ‘influencing capability’. In other words, are their governance arrangement, management systems and capabilities, research agendacommunications portfolio, partnerships, resource base, etc.  adequate for an organisation that aims to influence public policy?

Maybe then they can develop an influencing plan for a specific initiative.

Help grantees to manage their context and relationships (including with their funders -and their funders’ contractors): The lessons from the think pieces commissioned in support of this evaluation coincide with this recommendation. Think tanks need to be better able to manage the relationship with their funders. This is why I insist in using the right labels: funders, contractors, grantees.

Many think tanks have developed into decentralised organisations in which each researcher manages his or her own relations with key partners, clients, funders, and audiences. Often, I find that different parts of the same think tanks engage, independently from each other, with different parts of the same policymaking body. This works for some, depending on the level of cohesion found in the think tanks’ audiences, but can be highly risky for others.

In a way, in some cases where the policy community they work with is rather incoherent itself, it may make sense to maintain a certain degree of incoherence in relation to their  audience. However, the relationship with a think tank’s funders has to be coherent and clear. It is never a good idea to let too many people manage the organisation’s relationships with a single funder.

It gets even more complicated when funders provide support via contractors. This relationship is different and it requires different skills.

DFAT and KSI would help their grantees by clarifying the difference between them. DFAT should remain as  a research funder in Indonesia (this is who I go if I have a research project idea) and KSI focus on developing research (and think tank) capacity (this is who I go to if I want to strengthen my organisation).

Finally, KSI should encourage think tanks in their cohort to get together (to unionise) before the programme delivers most of its planned support. Together, the think tanks will be better able to manage their relationship with KSI and DFAT. It will be more equal and fair.

Facilitate gradual move to full cost recovery: Funders of think tanks are always concerned that their grantees may not be self-sufficient. They expect them to move away from financial dependence and achieve some sort of financial independence. But the fact is that think tanks are not businesses. They are not supposed to make money. They serve other purposes. Someone has to decide that think tanks are worth having and worth bankrolling. Whether it is the State, the private sector, civil society, political parties or individuals. Someone has to pick up the bill and pay up.

Insisting that they become self-sufficient can lead to some of them becoming research consultancies. Some consultancy is alright -but too much can be dangerous.

What is important, however, is that think tanks are better able to monitor and manage their resources. Few can tell how much they spend on specific projects or tasks. The first step towards full cost-recovery funding is better cost allocation systems. DFAT gets this:

As such, the program will work with partners to develop better cost-allocation systems and to focus on different ways to raise funds from other sources.

Make gradual changes to core funding: How to manage core or institutional funds is an important question. However, we do not know much about how this is being done by KSI. The latest reports I have suggest that all the think tanks involved in the programme are receiving about AUS100,000 per year during the first year and AUS200,000 from the second year onwards. For some think tanks this will constitute an important sum of money but for others may be a relatively small amount -also small relative to other funds they receive from DFAT.

This means that the core funding arrangement with each think tank is in fact different. The same amount does not translate into the same deal; rather into different ones. For those for whom AUS100,000-200,000 constitutes a significant share of their budget the KSI has the potential to become a key stakeholder in the organisation’s life. In these cases, this kind of funding will feel more like project-funding than core funding.

Changes to the manner in which KSI provides core funding ought to be evidence based. They have an opportunity to test different ‘models’ during this phase of the programme: different amounts, different rhythms, different terms of reference, different degrees of freedom, etc. The evidence from these tests ought to inform their future support to think tanks in Indonesia.

Support grantees to improve performance: At the core of think tanks’ performance are their staff. Supporting think tanks to find the best staff (among researcher, communicators and managers) is a key role of the KSI. This also includes support to ‘grow their own’.

My advice is to avoid the temptation of quick fix workshops. These investments need to be made over a long period of time and ought to include:

  • Supporting the development of HR capacities within the think tanks or among the network of Indonesian think tanks;
  • Offering scholarships for promising thinktankers to work with, study abroad or spend time in think tanks in other countries as interns or associates (please, not development policy think tanks) so they can learn from their experience; and
  • Supporting Indonesian universities to ensure that the next generations of graduates respond to the needs of organisations such as think tanks.