{"id":1609,"date":"2013-01-14T12:22:00","date_gmt":"2013-01-14T17:22:00","guid":{"rendered":"https:\/\/onthinktanks.org\/articles\/\/"},"modified":"2016-01-23T12:23:39","modified_gmt":"2016-01-23T17:23:39","slug":"a-monitoring-and-evaluation-activity-for-all-think-tanks-ask-what-explains-your-reach","status":"publish","type":"post","link":"https:\/\/onthinktanks.org\/articles\/a-monitoring-and-evaluation-activity-for-all-think-tanks-ask-what-explains-your-reach\/","title":{"rendered":"A monitoring and evaluation activity for all think tanks: ask what explains your reach"},"content":{"rendered":"<p><a title=\"Contributors\" href=\"https:\/\/onthinktanks.org\/about\/contributors\/\" target=\"_blank\">Nick Scott\u00a0<\/a>has published an interesting post on the ODI blog:\u00a0<a href=\"http:\/\/www.odi.org.uk\/opinion\/7192-2012-odi-online-awards-some-insights-they-offer-into-success\" target=\"_blank\">The 2012 ODI online awards \u2013 and some insights they offer into \u2018success\u2019<\/a>. On Think Tanks was spared his critique of organisations and people who publish the lists of the most popular posts of the year (probably because our post offered some analysis into the list) but still his detailed analysis of the various reports, posts, events, etc. provides a challenge for us.<\/p>\n<p>His blog post is also an interesting example of\u00a0<a title=\"This year, instead of ranking think tanks lets think about them more\u00a0carefully\" href=\"https:\/\/onthinktanks.org\/2012\/11\/26\/this-year-instead-of-ranking-think-tanks-lets-think-about-them-more-carefully\/\" target=\"_blank\">the kind of analysis that ranking efforts ought to consider<\/a>. It&#8217;s useless to offer a simple list of &#8216;best online strategies&#8217; and not explain what is it that makes each better or worse than the other.<\/p>\n<p>Nick&#8217;s post focuses on ODI&#8217;s content so it should not be taken to be an assessment of the best out there. Rather, it is an interesting monitoring (and learning) component of any think tank. One that should be on the top of the to-do list of any communications lead. Or, if we follow<a href=\"http:\/\/blogs.cgdev.org\/globaldevelopment\/2012\/11\/the-envelope-please-seeking-a-new-way-to-rank-think-tanks.php\" target=\"_blank\">\u00a0CGD&#8217;s example<\/a>, something researchers themselves should pay attention to.<\/p>\n<p>A similar kind of analysis is offered by Kent Anderson in:\u00a0<a href=\"http:\/\/scholarlykitchen.sspnet.org\/2012\/11\/29\/metrics-and-meaning-can-we-find-relevance-and-quality-without-measurements\/\" target=\"_blank\">Metrics and Meaning \u2014 Can We Find Relevance and Quality Without\u00a0Measurements?<\/a>\u00a0In this post for the\u00a0<a href=\"http:\/\/scholarlykitchen.sspnet.org\/\">The Scholarly Kitchen<\/a>\u00a0he muses that:<\/p>\n<blockquote><p>Thinking about the goals of altmetrics \u2014 identifying content that\u2019s more relevant, more interesting, novel, or important, and doing so as quickly as possible after publication or, better yet, helping authors to find the best match for their works \u2014 made me wonder if we\u2019re missing some obvious alternatives to metrics, ones based on words rather than numbers. Not alternative metrics (or altmetrics), but alternatives to metrics (which I\u2019ll call alt2metrics).<\/p><\/blockquote>\n<p>He argues that:<\/p>\n<ul>\n<li>There is an obsession with metrics that borders on fetish and can be rather unhelpful;<\/li>\n<li>Often the people and organisations promoting them are interested in selling them to us -of making us play &#8216;their&#8217; game; and<\/li>\n<li>That intangibles can matter a great deal more than things we can measure.<\/li>\n<\/ul>\n<p>This last point he expands with the following idea:<\/p>\n<blockquote><p>Perhaps key to all this is the fact that metrics take time to assemble \u2014 they are delayed, and secondary to activity. Non-measurement-dependent signals are more important, anticipatory, and upstream from metrics. And they are what scientists rely on every day to guide them and their searches for information. It would be a shame to spend all our time on secondary, derivative measurements while primary, original signals of value are ignored or downplayed inappropriately.<\/p><\/blockquote>\n<p>His analysis is worth considering when reading through Nick&#8217;s own (and do remember that Nick has argued something quite similar).<\/p>\n<p>For the list of top ODI publications, events, blogs, etc. it&#8217;s probably best to read the blog. (Note that the most-read post was one published in this blog:\u00a0<a href=\"https:\/\/onthinktanks.org\/2012\/01\/06\/monitoring-evaluating-research-communications-digital-tools\/\">A pragmatic guide to monitoring and evaluating research communications using digital tools<\/a>; but then again, this is only among those he could measure!)<\/p>\n<p>What I am interested in, are the lessons that come out of the analysis:<\/p>\n<p><strong>The media<\/strong>\u00a0is a key audience of a think tank&#8217;s research and so media uptake should be an indicator of readership:<\/p>\n<blockquote><p>This key report by Homi Kharas of\u00a0<a href=\"http:\/\/www.brookings.edu\/\" target=\"_blank\">Brookings<\/a>\u00a0and Andrew Rogerson of ODI on the aid industry made a large splash following its release in July. Mentions of it could be found in media outlets such as\u00a0<a href=\"http:\/\/www.guardian.co.uk\/\" target=\"_blank\">The Guardian<\/a>,\u00a0<a href=\"http:\/\/www.chicagotribune.com\/\" target=\"_blank\">Chicago Tribune<\/a>, the\u00a0<a href=\"http:\/\/www.bbc.co.uk\/\" target=\"_blank\">BBC<\/a>\u00a0and\u00a0<a href=\"http:\/\/www.smh.com.au\/\" target=\"_blank\">the Sydney Morning Herald<\/a>; in blogs such as\u00a0<a href=\"http:\/\/www.aviewfromthecave.com\/\" target=\"_blank\">A View from The Cave<\/a>,\u00a0<a href=\"https:\/\/www.devex.com\/en\/\" target=\"_blank\">Devex<\/a>\u00a0and\u00a0<a href=\"http:\/\/www.huffingtonpost.co.uk\/\" target=\"_blank\">the Huffington Post<\/a>; and by development organisations including\u00a0<a href=\"http:\/\/www.worldbank.org\/\" target=\"_blank\">the World Bank<\/a>,<a href=\"http:\/\/www.worldvision.org.uk\/\" target=\"_blank\">World Vision<\/a>\u00a0and\u00a0<a href=\"http:\/\/www.savethechildren.org.uk\/\" target=\"_blank\">Save the Children<\/a>. Over the months since, it has registered 6,622 downloads \u2013 putting it well ahead of the nearest competitor in terms of downloads,\u00a0<a href=\"http:\/\/www.odi.org.uk\/publications\/6411-euro-zone-crisis-developing-countries-vulnerability\">The euro zone crisis and developing countries<\/a>, which registered 3,880.<\/p><\/blockquote>\n<p><strong>Global audiences<\/strong>\u00a0are increasingly relevant in certain occasions and so their origin should feature high in their analysis:<\/p>\n<blockquote><p>Events were broadcast around the world through video streaming and reached viewers around the globe. There were registrations to watch from as far afield as Zimbabwe, Iraq, Pakistan, the Philippines and the three countries themselves. Overall, we logged registrations from 48 different countries in all five continents<\/p><\/blockquote>\n<p><strong>VIPs<\/strong>\u00a0still matter:<\/p>\n<blockquote><p>The blog around the graphic received 2,017 views. The big story, however, is not how many times it was viewed but\u00a0<em>who<\/em>\u00a0it was viewed by. We happen to know that this info-graphic was shared among the negotiators on climate finance at the Doha summit.<\/p><\/blockquote>\n<p><strong>Alternative metrics\u00a0demand alternative analyses<\/strong>, too; things are not \u00a0always what they appears to be:<\/p>\n<blockquote><p>The ODI Centre for Aid and Public Expenditure conference saw 113 tweets mentioning the event page. However, this again has a lesson, and it is one that will become increasingly important to take on board as\u00a0<a href=\"http:\/\/altmetrics.org\/manifesto\/\" target=\"_blank\">AltMetrics<\/a>\u00a0become more important to judging academic success. The issue is what makes up a figure \u2013 the 113 tweets about the conference include a number of tweets by ODI researchers and staff promoting the conference. Given this, a more likely winner for the most tweeted piece of content would be\u00a0<a href=\"http:\/\/www.odi.org.uk\/opinion\/6954-corruption-golden-thread-development-governance\">Inconvenient truths about corruption and development<\/a>, a November 2012 blog written by Marta Foresti, which was tweeted about 99 times.<\/p><\/blockquote>\n<p><strong>Regular updating<\/strong>\u00a0has positive effects (is the opposite true as well?):<\/p>\n<blockquote><p>That is the fastest growing research site I\u2019ve seen in the six years I\u2019ve been at ODI. Perhaps that is because it has, in those months, published 207 posts &#8211; that is over a post a day. This includes posts from people at numerous organisations working on the subject. This regular updating has made it a vital source for information in the sector, a fact also evidenced by\u00a0<a href=\"https:\/\/twitter.com\/post2015\" target=\"_blank\">the 719 followers it has quickly accrued on Twitter<\/a>, including leading figures in the post-2015 debates.<\/p><\/blockquote>\n<p><strong>Third-party posting<\/strong>, and using other people&#8217;s networks, can be more effective than attempting to keep it all in-house:<\/p>\n<p>This works for publishing in other spaces:<\/p>\n<blockquote><p>This points to another of the issues with top 10 lists, especially for ODI given our \u2018<a href=\"https:\/\/onthinktanks.org\/2011\/09\/19\/digital-disruption-traditional-communications-being-there\/\" target=\"_blank\">being there\u2019 digital communications strategy<\/a>\u00a0that encourages posting to large or sector-specific blogs wherever possible to reach a wider or more directly relevant audience.<\/p><\/blockquote>\n<p>And for asking relevant outsiders to write for the think tank:<\/p>\n<blockquote><p>Through strategic use of Twitter to engage people with an interest in the subject, the\u00a0<a href=\"http:\/\/www.odi.org.uk\/projects\/2320-secure-livelihoods-research-consortium\">Secure Livelihoods Research Consortium<\/a>\u00a0team was able to interest\u00a0<a href=\"http:\/\/www.badscience.net\/about-dr-ben-goldacre\/\" target=\"_blank\">Ben Goldacre<\/a>, well-known for writing \u2018Bad Science\u2019 and the discussions kicked off from there.<\/p><\/blockquote>\n<p><strong>Most people are looking for jobs<\/strong>\u00a0(this appears to be true also for some of the most read newspapers in many countries) so we should not be fooled by &#8216;visits&#8217;:<\/p>\n<blockquote><p>around 70 percent of visitors to the ODI site come to research or look for jobs. They\u2019re not necessarily in an interacting mood.<\/p><\/blockquote>\n<p><strong>Action beyond reading<\/strong>\u00a0is an important indicator of success:<\/p>\n<blockquote><p>And my final award of the year goes to the blog that managed to get most comments on the ODI site \u2013 a key indication that the blog has generated engagement and interest.<\/p><\/blockquote>\n<p><strong>Don&#8217;t forget the old stuff<\/strong>; it is often work done many years ago that keeps getting the most downloads and the most interest:<\/p>\n<blockquote><p>For many years now, this 2003 online toolkit has been by far the most viewed piece of content. In 2012, the toolkit received 30,310 views and the downloadable version was accessed 8,410 times. Why is this still getting so much attention? One word: Google.<\/p><\/blockquote>\n<p>Beyond the lessons, too, this kind of analysis is important for any organisation. I would even argue that this is really all that is necessary. Asking questions about impact on a policy simply goes too far. It escapes the legitimate role of most think tanks and demands that they make assumptions that make any assertion highly unreliable; and useless. This, however, offers many useful ideas for immediate and future action.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Nick Scott\u00a0has published an interesting post on the ODI blog:\u00a0The 2012 ODI online awards \u2013 and some insights they offer into \u2018success\u2019. On Think Tanks was spared his critique of organisations and people who publish the lists of the most popular posts of the year (probably because our post offered some analysis into the list) [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","footnotes":""},"tags":[463,235,233,182,457],"acf":[],"_links":{"self":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/1609"}],"collection":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/comments?post=1609"}],"version-history":[{"count":0,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/1609\/revisions"}],"wp:attachment":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/media?parent=1609"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/tags?post=1609"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}