{"id":2403312,"date":"2021-01-22T10:40:28","date_gmt":"2021-01-22T15:40:28","guid":{"rendered":"https:\/\/onthinktanks.org\/articles\/\/"},"modified":"2021-03-03T12:34:00","modified_gmt":"2021-03-03T17:34:00","slug":"guiding-principle-4-dynamic-evaluation","status":"publish","type":"post","link":"https:\/\/onthinktanks.org\/articles\/guiding-principle-4-dynamic-evaluation\/","title":{"rendered":"Guiding principle 4: Dynamic evaluation\u00a0"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Scaling creates dynamic change that calls for dynamic evaluation \u2013 continuous learning before, during and after scaling. Dynamic evaluation helps us to apply the other three principles by understanding when scaling is justified, what is the optimal scale, and how to coordinate.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">An introduction to Guiding principle 4: <\/span><a href=\"https:\/\/youtu.be\/ECtF8NlWSag\"><span style=\"font-weight: 400;\">Dynamic evaluation<\/span><\/a><\/p>\n<div class=\"iframe-container\"><iframe loading=\"lazy\" title=\"Guiding principles for scaling impact: Dynamic evaluation\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/yBAQLyk-k2c?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/div>\n<h2>Guiding questions<\/h2>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">How have you incorporated dynamic evaluation into scaling processes? What have been challenges and opportunities?<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Have you used results of dynamic evaluation to inform an approach to scaling impact \u2013 if so, how? If not, what inhibited this?<\/span><\/li>\n<\/ul>\n<h2>General conclusions from the discussion<\/h2>\n<p>Each conversation leading up to the session on dynamic evaluation highlighted its importance to informing and upholding the other three guiding principles. Scaling advisors raised the importance of a learning culture within organisations to enable dynamic evaluation. In environments where evaluation is understood as mainly an accountability-oriented tool, there may be a reluctance to use approaches that will surface the information needed to identify and pursue optimal scale. There is also a need to think beyond individual projects when taking a dynamic evaluation stance, given that scaling processes typically span multiple projects\u2014dynamic evaluation means we need to consider what has been done before, and ask questions that could inform potential future efforts to scale for optimal impact. To do this requires sufficient resources for evaluation as well as flexibility around what will be assessed.<\/p>\n<div class=\"iframe-container\"><iframe loading=\"lazy\" title=\"Fishbowl 4: Dynamic evaluation\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/4kzcjFTZfJU?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/div>\n<p><i><span style=\"font-weight: 400;\">The following statements do not reflect all the opinions or reflections presented during the session. In some cases they reflect the ideas presented and shared in their own working groups. We have kept the name of the person who shared this particular idea during the session.\u00a0<\/span><\/i><\/p>\n<h2>Discussion<\/h2>\n<p><i><span style=\"font-weight: 400;\">[Tobias Sch\u00f6nwetter, University of Cape Town]<\/span><\/i><\/p>\n<p>&nbsp;<\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>With multi-cycle projects, change often happens a long time after the cycle. <\/b><span style=\"font-weight: 400;\">So we make sure we document what we did in previous cycles, to be able to think about impact.<\/span><\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><i><span style=\"font-weight: 400;\">[Enrique Mendizabal, OTT]<\/span><\/i><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Scaling interventions and evaluations may involve different approaches, methods and expertise at different stages of the process. <\/span><\/li>\n<\/ul>\n<p><i><span style=\"font-weight: 400;\">[Blanca Llorente, Fundacion Anaas]<\/span><\/i><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Dynamic evaluation requires building an evaluative and learning culture within organisations.<\/b><span style=\"font-weight: 400;\"> How do we build and nourish an evaluation culture? What incentives, what communications tools do we use to build that culture within our groups? One way is to incorporate \u2018learning\u2019 into the roles of every member of the team.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Evaluations are always political<\/b><span style=\"font-weight: 400;\">. Evaluating a scaling effort by (or with) the government can be highly political.\u00a0<\/span><\/li>\n<\/ul>\n<p><i><span style=\"font-weight: 400;\">[Johannes Linn, The Brookings Institution]<\/span><\/i><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Dynamic evaluation demands that we look beyond the \u2018what\u2019 and focus on the \u2018how\u2019<\/b><span style=\"font-weight: 400;\"> \u2013 how to justify scale, how to determine the optimal scale, how to partner or collaborate and so on. It requires a focus on learning rather than accountability.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Evaluation needs appropriate resources.<\/b><span style=\"font-weight: 400;\"> The monitoring, evaluation and learning (MEL) working group of the Scaling Up Community of Practice has concluded that about 20% of budgets for pilot projects should be devoted to MEL, rather than the common 3\u20135%.<\/span><\/li>\n<\/ul>\n<p><i><span style=\"font-weight: 400;\">[Tatiana Rinc\u00f3n, Fundaci\u00f3n Capita<\/span><\/i><i><span style=\"font-weight: 400;\">l<\/span><\/i><i><span style=\"font-weight: 400;\">]<\/span><\/i><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Dynamic evaluation requires resources<\/b><span style=\"font-weight: 400;\">. Funding is often limited for this role but we recognise that it is central to any scaling effort.\u00a0<\/span><\/li>\n<li><b>It\u2019s necessary to invest in developing more cost-effective evaluation techniques.<\/b><span style=\"font-weight: 400;\"> Evaluation resources are often focused on evaluating the impact of the pilot intervention through randomised control trials, which are resource intensive. But evaluation is needed throughout the scaling effort and other methods are sometimes more appropriate and affordable. <\/span><\/li>\n<\/ul>\n<p><i><span style=\"font-weight: 400;\">[Hayley Price-Kelly, IDRC]<\/span><\/i><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>An evaluation of scaling efforts often involves multiple projects<\/b><span style=\"font-weight: 400;\"> that may not be formally connected to each other and this presents a challenge.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Dynamic evaluation can mean a whole load of things depending on where you are in the process.<\/b> <span style=\"font-weight: 400;\">S<\/span><span style=\"font-weight: 400;\">ometimes as we move through projects and start to move towards a more specific scaling intention, it\u2019s about making sure you\u2019re doing the right things to understand justification, optimal scale, coordination.\u00a0<\/span><\/li>\n<\/ul>\n<p><i><span style=\"font-weight: 400;\">[Leslie Fierro, IDRC]<\/span><\/i><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Dynamic evaluation pushes the \u2018how\u2019 and \u2018why\u2019 rather than the \u2018what\u2019<\/b> <b>questions.<\/b><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>There\u2019s often an assumption that the effects will be visible right away<\/b><span style=\"font-weight: 400;\">. In a dynamic evaluation context, it\u2019s really important for funders to be thinking about what the lag time is between an effort and what we expect to see, and how that changes over time. We tend to think about this with outcome-focused evaluation.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Evaluative thinking is to continuously question assumptions that are part of our implementation process<\/b>, to be able to continuously learn and adapt.<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-2609476 size-full\" src=\"https:\/\/onthinktanks.org\/wp-content\/uploads\/2021\/01\/IDRC-ScalingScience-Banner_REV2.png\" alt=\"\" width=\"2480\" height=\"413\" srcset=\"https:\/\/onthinktanks.org\/wp-content\/uploads\/2021\/01\/IDRC-ScalingScience-Banner_REV2.png 2480w, https:\/\/onthinktanks.org\/wp-content\/uploads\/2021\/01\/IDRC-ScalingScience-Banner_REV2-300x50.png 300w, https:\/\/onthinktanks.org\/wp-content\/uploads\/2021\/01\/IDRC-ScalingScience-Banner_REV2-768x128.png 768w, https:\/\/onthinktanks.org\/wp-content\/uploads\/2021\/01\/IDRC-ScalingScience-Banner_REV2-1024x171.png 1024w\" sizes=\"(max-width: 2480px) 100vw, 2480px\" \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Scaling creates dynamic change that calls for dynamic evaluation \u2013 continuous learning before, during and after scaling. Dynamic evaluation helps us to apply the other three principles by understanding when scaling is justified, what is the optimal scale, and how to coordinate. An introduction to Guiding principle 4: Dynamic evaluation Guiding questions How have you [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","footnotes":""},"tags":[2689,2984],"acf":[],"_links":{"self":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/2403312"}],"collection":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/comments?post=2403312"}],"version-history":[{"count":0,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/posts\/2403312\/revisions"}],"wp:attachment":[{"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/media?parent=2403312"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/onthinktanks.org\/wp-json\/wp\/v2\/tags?post=2403312"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}