Comprehensive Course on Evaluation Methods and Impact Measurement
This comprehensive course covers topics such as evaluation methodologies, impact measurement, randomization, threats analysis, sampling techniques, cost-effectiveness analysis, and more. It also delves into the theory of change, log frame objectives, and practical applications through case studies and lectures. Additionally, the course explores the role of women as policymakers through a detailed case study.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Measurement Marc Shotland J-PAL Global
Course Overview 1. What is evaluation? 2. Measuring impacts (outcomes, indicators) 3. Why randomize? 4. How to randomize 5. Threats and Analysis 6. Sampling and sample size 7. Cost Effectiveness Analysis and Scaling Up 8. RCT: Start to Finish
Course Overview 1. What is evaluation? 2. Measuring impacts (outcomes, indicators) 3. Why randomize? 4. How to randomize 5. Threats and Analysis 6. Sampling and sample size 7. Cost Effectiveness Analysis and Scaling Up 8. RCT: Start to Finish
Theory of Change Less Diarrhea Contaminated water is primary source of illness Drink Clean water Have access to clean water at come Choose to drink only clean water Access to clean water at source Choose to collect only clean water Understand benefits of clean water Know which water is clean No recontamination Clean method of extracting water Hygiene practices Sufficient water
Log Frame Objectives Hierarchy Lower rates of diarrhea Indicators Sources of Verification Household survey Assumptions / Threats Waterborne disease is primary cause of diarrhea Impact (Goal/ Overall objective) Outcome (Project Objective) Rates of diarrhea Households drink cleaner water ( in) drinking water source; E. coli CFU/100ml E. coli CFU/100ml; Household survey, water quality test at home storage Shift away from dirty sources. No recontamination Outputs Source water is cleaner; Families collect cleaner water Source protection is built Water quality test at source continued maintenance, knowledge of maintenance practices Sufficient materials, funding, manpower Inputs Protection is present, functional Source visits/ surveys (Activities) Source: Roduner, Schlappi (2008) Logical Framework Approach and Outcome Mapping, A constructive Attempt of Synthesis,
Lecture Overview 1. Case Study Review and follow-up 2. Theory of Change, Outcomes, Indicators 3. Indicators: Use in analysis 4. Sources of Data 5. Collecting Data
Women as Policymakers CASE STUDY
What was the main purpose of the 73rd Amendment of India s constitution A. To reserve leadership positions for women (and caste minorities) B. To formalize local institutions of leadership C. To give women the right to vote in local elections
Theory of Change Public goods reflect Women s preferences Women have different preferences Investments reflect women s preferences Pradhan s preferences matter More female pradhans Imperfect democracy Some democracy Reservations for Women Women are empowered
Log Frame Objectives Hierarchy Lower rates of diarrhea Indicators Sources of Verification Household survey Assumptions / Threats Waterborne disease is primary cause of diarrhea Impact (Goal/ Overall objective) Outcome (Project Objective) Rates of diarrhea Households drink cleaner water ( in) drinking water source; E. coli CFU/100ml E. coli CFU/100ml; Household survey, water quality test at home storage Shift away from dirty sources. No recontamination Outputs Source water is cleaner; Families collect cleaner water Source protection is built Water quality test at source continued maintenance, knowledge of maintenance practices Sufficient materials, funding, manpower Inputs Protection is present, functional Source visits/ surveys (Activities) Source: Roduner, Schlappi (2008) Logical Framework Approach and Outcome Mapping, A constructive Attempt of Synthesis,
Data used Sources of Measurement Household (HH) Survey Indicators Declared HH preferences HH perceptions of quality of public goods and services Political experience Investments undertaken Village infrastructure + investments Perception of public good quality Participation of men and women Issues Budgets Balance sheets Who speaks and when (gender) Issues raised Village Leader Interview Village PRA Administrative Data Transcript from village meeting
Results West Bengal Rajasthan Issue W 31% 17% 31% 25% Issue W 54% 13% Reserved Investment 9.09 0.18 Reserved Investment 2.62 -0.08 M M Issue Drinking Water Road Improvement Irrigation Education Investment # facilities Road Condition (0-1) # facilities Informal education center 49% 23% 4% 6% -0.38 -0.06 2% 5% 4% 13% -0.02 20% 12%
Use in analysis INDICATORS
The main challenge in measurement Accuracy Precision
The main challenge in measurement Validity Reliability
Validity How well does the indicator map to the outcome? Standardized Tests, Income, empowerment Are you getting unbiased answers? Social desirability bias (response bias) Framing effect Recall bias Anchoring bias
Reliability The measure is very noisy, but on average, correct
Which is worse? A. Poor Validity B. Poor reliability C. Equally bad D. Depends E. Don t know/can t say
Validity When there s a baseline When there s a comparison group When bias is correlated with treatment
Consistently Biased Baseline Endline Difference truthestimates Treatment Control
Bias is correlated with treatment Baseline Endline Difference truthestimates Treatment Control
Reliability AND Validity Question wording Surveyor training/quality Data entry Translation Length, fatigue
Reliability and error Question wording Definitions Relationships Recall period Answer choice Open/Closed Single v. Multiple option Likert Ranked response Units
How frequently do you drive your car? A. Never B. 1-2 C. 3-5 D. 6-10 E. More than 10 F. Don t know/can t say
The problem With the following questions
Outcome: annual consumption Indicator: food expenditure in last week A. Validity B. Reliability C. Both D. Neither
Outcome: annual consumption Indicator: food expenditure in last three months A. Validity B. Reliability C. Both D. Neither
Question: have you had sex in the past week? [if yes] Did you use protection? A. Validity B. Reliability C. Both D. Neither
Other important considerations Ethics Might affect compliance
Why do you need data? Outcomes Sub-groups Covariates Predictors of compliance
Where can we get data? Administrative Data Other Secondary Data Primary Data
Primary Data Collection Self-reported Surveys Exams, tests, etc Games Vignettes Direct Observation Diaries/Logs
Modules Income, consumption, expenditure Perceptions, expectations, aspirations Bargaining power Patience, risk Behavior (time use) Anthropometric Cognitive, Learning Yields
Public good investments Public goods reflect Women s preferences Women have different preferences Investments reflect women s preferences Pradhan s preferences matter More female pradhans Imperfect democracy Some democracy Reservations for Women Women are empowered
Women empowerment? Women empowerment Women have different preferences Investments reflect women s preferences Pradhan s preferences matter More female pradhans Imperfect democracy Some democracy Reservations for Women Women are empowered
Outcome: Gender Bias Question: How effective is your leader? (ineffective, somewhat effective, effective, very ) A. Validity B. Reliability C. Both D. Neither
Perceptions and Attitudes How effective is your leader? (ineffective, somewhat effective, effective, very ) Listen to a Vignette (Male v. Female) Revealed preference voting behavior Implicit Association tests
Enter question text... Academic or Good Practitioner or Bad Happiness
Enter question text... Academic or Bad Practitioner or Good Pleasure
Enter question text... Practitioner or Bad Academic or Good Professor
Enter question text... Practitioner or Good Academic or Bad Happiness
Enter question text... Practitioner or Good Academic or Bad Research
Considerations DATA COLLECTION
Data Collection Considerations Quality Control Surveyor training Surveyor (gender) composition Human subjects Data Security Electronic v paper Costs
When to collect Data Baseline During the intervention Endline Scale-up, intervention
Results from Women Empowerment Significant electoral gains for women in subsequent unreserved elections Changed perceptions of women s ability to lead effectively Heightened career aspirations of adolescent girls and increased level of educational attainment
