Rethinking Program Evaluation: Embracing Outcome Trajectories
Program evaluators are encouraged to shift focus from the traditional program theory of change to exploring outcome trajectories, which consider the broader context and interlinked changes over time. This approach offers a more insightful understanding of program outcomes and their evolution. An example illustrates the impact of outcome trajectories in a program aimed at controlling viral diseases in Tanzania through cassava seed certification. By starting with outcome trajectories, evaluators can gain valuable insights for future programs and stakeholders.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
I wouldnt start from here: making the case for Outcome Trajectory Evaluation Boru Douthwaite, Selkie Consulting Ltd Presentation given to Webinar on Contexts, Trajectories, and Assessing Progress Towards Goals 11 March 2021
I wouldnt start from here
As program evaluators We want to arrive at a plausible and insightful explanation of how a program has, or hasn t, had an impact To learn lessons for the future To be accountable to investors and stakeholders Usual practice is to start with the program theory of change (ToC), as laid out in the program document A model of how program activities will produce outputs, will produce outcomes, will produce impact
The program ToC is the wrong place to start! It focuses too much on the program and as a result assumes the program has more causal power than it does A case of the universe going around the earth! Does not adequately take into account the broader context in which the program operates the metaphorical bogs, lakes, mountains and bus routes that also have more causal power!
A better place to start is the broader context, specifically the outcome trajectories that have produced program outcomes Program outcomes are arrangements of interlinking and self- reinforcing changes that emerge and evolve over time Program outcomes emerge from outcome trajectories -- interacting and co-evolving system of actors, knowledge, technology and institutions.
Example of a program outcome Cassava seed certification arrangement to control two viral diseases in Tanzania To work it requires: Farmers to produce clean planting material Seed inspectors certifying it Farmers buying it and making a profit from doing so All underpinned by regulations passed into law The outcome trajectory began in 2007, if not earlier!
Development of Outcome Trajectory Evaluation First used to evaluate four policy outcomes of interest to two CGIAR research programs
Step 1: Agree on the program outcomes to evaluate These were agreed by the two programs before commissioning the evaluations. Selected on basis of significance to the program and the interest of respective teams to learn from the experience
Step 2: Select a published middle-range theory that is the best fit with the types of change under scrutiny Positioned between disciplinary theory and program theory Potentially applying to clusters of similar programs, e.g., those attempting to inform evidence-driven policy in agriculture in developing countries E.g. Policy Window theory That policy progress is made during windows of opportunity when policy champions can link two or more streams in the policy process The streams are: the way the problem is defined; the policy solution to the problem and the politics surrounding the issue
Step 3: Use middle-range theory to help identify and describe the outcome trajectories E.g., look for existence of policy champions being able to link together policy streams during windows of opportunity. Develop timelines and write out an explanatory narrative
Step 4: Use the middle-range theory as the basis of case-specific theories of change, one for each outcome trajectory Describe how the generic outcomes identified in the middle-range theory manifest themselves in each outcome trajectory, e.g., Shifting social norms Capacity development Strengthening support base (from Stachowiak, 2013) Describe the outcome-trajectory-specific strategies used by outcome- trajectory-actors to contribute to these outcomes
Step 5: Validate the outcome trajectory timelines and case-specific theory of change with key stakeholders
Step 6: Use the case-specific ToC and validated timelines to answer the evaluation questions Particularly useful at answering coherence, effectiveness, sustainability and impact questions Able to identify how two CGIAR programs had contributed to the broader outcome trajectory
Step 7: Review of final evaluation report before publication Each evaluation report reviewed by a bespoke reference group Douthwaite B (2020a) Mainstreaming of biofortification in the African Union: Evaluation of CGIAR contributions to a policy outcome trajectory. CGIAR Research Programs on Roots, Tubers and Bananas & Agriculture for Nutrition and Health: Lima, Peru. https://doi.org/10.4160/978929060-5522 Douthwaite B (2020b) Development of a cassava seed certification system in Tanzania: Evaluation of CGIAR contributions to a policy outcome trajectory. International Potato Center: Lima, Peru. https://hdl.handle.net/10568/110093 Douthwaite B (2020c) Development of a cassava seed certification system in Rwanda: Evaluation of CGIAR contributions to a policy outcome trajectory. International Potato Center: Lima, Peru. https://doi.org/10.4160/9789290605638 Douthwaite B (2020d) Control of potato purple top in Ecuador: Evaluation of CGIAR contributions to a policy outcome trajectory. International Potato Center: Lima, Peru. https://doi.org/10.4160/9789290605553
In conclusion: comparing OTE with standard program evaluation Distinguishing features OTE Standard program evaluation Starting point Program outcomes and the outcome trajectories from which they emerged ToC / results framework in program document Use of theory Use published theory as the basis for theory of change for each outcome trajectory Test whether theory of change specified in the Program Document was valid Establishing causation By investigating whether: 1) the outcomes exist; 2) they were generated by the respective trajectories; and 3) the program contributed to the trajectory By investigating whether program activities contributed to outcomes without really knowing about the synergistic contribution of others. This makes establishing causation harder Harder without a common framework Learning from one evaluation to the next Possible because middle-range theory provides a common framework to do so
Some questions to finish Does starting from somewhere else make sense? Does standard program theory assume too much causal power? Are outcome trajectories a better place to start? Should we be making more use of middle-range theories to help accumulate learning from one evaluation to the next?
