Evaluation Synthesis in Changing Contexts: Enhancing Knowledge for Development Effectiveness
Evaluation synthesis is crucial for promoting learning, reflection, and decision-making in development work. This process involves bringing together diverse knowledge sources to generate strategic insights and facilitate wider use of evaluation findings. The Independent Office of Evaluation of IFAD emphasizes the importance of synthesis as a knowledge product to address current and future challenges effectively. This approach involves answering key knowledge questions based on robust methodologies and testing the external validity of findings across various stakeholders. The document highlights four evaluation syntheses on gender, partnerships, rural finance, and scaling up, illustrating the importance of rich evidence, stakeholder interest, and context-specific practices in evaluation processes.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Knowledge-focused Evaluation in Changing Contexts Evaluation synthesis Independent Office of Evaluation of IFAD Johanna Pennarz EES 2018
Why evaluation synthesis? Knowledge product - to enhance the general understanding of a particular topic To promote learning, collective reflection and improve development effectiveness To highlight strategic implications of findings, raise strategic issues for further consideration by management and governing bodies To facilitate wider use of evaluation findings To contribute to decision-making processes Effective when there is limited resources or time *Independent Office of Evaluation of IFAD. Evaluation Manual. 2015. 2
Whose knowledge? Global audience Technical experts, practitioners, beneficiaries etc. Community of practice Evaluation associations, UN organisations, IFIs etc. Member countries Oversight bodies, country governments etc. IFAD management and staff Project management, evaluation, technical specialists etc. 3
Synthesis as knowledge product For a synthesis to be useful For a synthesis to be useful knowledge questions knowledge questions It has to answer It has to answer Based on a robust Based on a robust methodology methodology What has worked in the past? Relevant enough to address up-to- date issues What may work in the future? Findings sufficiently grounded in context Sufficient evidence for broad coverage and scope And under what conditions? External validity of findings tested 4
Four evaluation syntheses Gender Partnerships Rural finance Scaling up 5
Four evaluation syntheses Rich evidence Little evidence Broad topic Gender - - - - - Partnerships - Broad stakeholder interest - Broad, but less dynamic topic - Broad range of practices, but not well described across different contexts Broad stakeholder interest Broad and dynamic topic Broad Range of practices Dynamic topic Well documented for a large number of contexts Narrowly defined topic Rural finance - Topic owned by fewer stakeholders - Narrow but dynamic topic - Broad range of practices well described - Limited number of cases for each practice Scaling up - Topic owned by fewer stakeholders - Narrow and less dynamic topic - Limited range of practices; not well described 6
Breadth or depth? Practices documented elsewhere Going into Breadth Practices documented in evaluations Going into Depth Sample of practices - 7 -
Strategy search process Cover larger number of cases Rural finance Large range of practices described Focus on smaller sample Gender Range of practices Limited range of practices described Cover existing sample Scaling up Cover large number of issues Partnerships Broadly defined topic Synthesis strategy Focus on selected issues Gender Narrowly defined topic Focus on pre- defined issues Rural finance Introduce state of the art concepts Topic Gender More dynamic topic Expand timespan Rural finance Less dynamic topic Use evaluation time span Partnerships, Scaling Up 8
Gender synthesis Systematises large number of existing practices Uses representative sample of reports Uses case studies to contextualise findings Applies state-of-the-art concept for a forward- looking perspective 9
Sample selection (ESR Gender) Layered process for screening evaluation products 1st layer of screening: strong evidence-base criteria for ratings of 4 and 6 Screening criteria sum* 3 0 1 0 4 4 9 2 5 10 4 1 Total 14 14 3 4 5 6 Total 1 15 15 31 14 3 17 Screened total * Sample filtered to include products that at least report outcomes, strategies, and evidence Sample for review: programme sub-sector against screening criteria sum Sub-sector Corrected sample to reflect overall population characteristics 1 1 0 0 0 2 1 0 1 0 3 5 4 1 1 4 9 7 5 0 5 7 1 2 1 AGRIC MRKTG 0 0 0 1 0 Screening criteria sum RURAL AGRIC CREDI RSRCH IRRIG LIVST FISH MRKTG Total* 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 3 2 11 23 12 Total 23 12 9 2 1 1 1 1 1 51 10
Gender transformation index IFAD GEWE Transformation Index 11
Outlier analysis (ESR Gender) Distribution of practices of the 10 projects with highest and lowest transformation index score 12 Source: compiled from data in table 2 below.
Partnership synthesis Covers large number of partnership arrangements Uses existing sample of country programme evaluations Uses case studies to contextualise findings Uses categories of partnerships for comparative analysis 13
Partnership categories Importance of different partnership categories Source: CSPE review, compiled from data in Annex V.1 14
Partnership results Outcomes reported for MICs and LICs Source: CSPE review, see data in annex V.3 tables 2 5 15
Nature of Analysis Broad Analysis Covers a large number of situations, contexts and issues Can help to understand what happens and under what conditions Can provide a systematic overview of broader patterns Can inform broader strategic thinking Deep analysis Covers a limited number of situations, contexts and issues Can help to understand why and how things happen Can provide a systematic assessment of specific practices etc. Can inform future practices Scaling up Partnerships Gender Rural finance 16
Breadth vs depth Scope of a synthesis is a challenge Stakeholders always want broad range of issues covered Time frame constrained by evaluative evidence; some topics might be more dynamic Nature of evidence base may require broader search strategy Broad topics with limited evidence are particular challenging (Partnerships) Rich evidence allows deeper analysis (Gender) 17
Nature of findings Generalised findings Able to identify general patterns and relations beyond individual context Likely to abstract across a number of situations General lessons can be learned Findings embedded in context Able to explain why things work under certain conditions Likely to provide a number of specific cases Specific lessons can be learned Scaling up Gender Partnerships Rural finance 18
External validity vs context Rigorous sampling approach possible in some cases; enables external validity (Gender) Additional strategies required to enable generalisation of findings (e.g. theory of Change, typologies, hypothesis testing) Diversity of contexts and time periods covered presents challenge Case studies indispensable to understand importance of context (Gender, Partnerships) Practices with limited cases are challenging (Partnerships) 19
Conclusion Methodology Nature of analysis (broad or deep) Knowledge-focused evaluation synthesis (Whose knowledge? What knowledge? Knowledge for what?) Stakeholder interest Sampling Data collection & aggregation Analytical framework Validation Timeliness Nature of findings (generalised or embedded) Available evidence base 20
