Evaluability Assessment in Program Evaluation
Evaluability Assessment
critical for successful
Evaluations
By Sylvester Kato
Technical M&E in the BMAU
UEA Conference, February 2019
Kampala Uganda
Annual
Evaluation
conference
February 11
th
to 15
th
2019
E
v
a
l
u
a
b
i
l
i
t
y
A
s
s
e
s
s
m
e
n
t
Presentation outline
◦
Introduction to EA
◦
Where is Evaluability lost
◦
Key steps in evaluability Assessment
◦
Key EA Questions
◦
Crucial Steps
◦
Conclusion
Evaluability Assessment
Evaluability assessment is ‘‘a pre-evaluation appraisal of whether a program’s
performance can be evaluated and if so whether it should be’’ (Rossi, Freeman, &
Lipsey 1999).
Systematic process to determine whether program evaluation is feasible.
Decide whether an evaluation is useful to provide timely, relevant findings for
decision makers.
Evaluation research will be pointless if the program itself cannot be evaluated.
Yes, some type of study is always possible, but a study specifically to identify the
effects of a particular program may not be possible within the available time and
resources.
So researchers may conduct an evaluability assessment to learn this in advance,
rather than expend time and effort on a fruitless project.
Evaluability Assessment
Assesses:
Underlying program logic
Current state of program implementation
Feasibility of conducting rigorous outcomes-focused evaluation or other sorts of evaluation
Evaluability Assessment
L
o
s
s
o
f
E
v
a
l
u
a
b
i
l
i
t
y
Review of Dutch Private Sector Programs
2/3 of executed ‘evaluations’ cannot be used
Major reasons:
- Evaluation agency not fully independent
- stated objectives too broad/vague
- no clear indicators defined
- data at too
aggregate
level
- absence of baseline data
- no representative sampling
- too general intervention theory
Key steps in evaluability Assessment
•
Smith (1989): ten steps procedure for
Evaluability Assessment
initial diagnostics
◦
Identify relevant stakeholders
◦
Define boundaries of the program
◦
Analyze available program documents
◦
Clarify intervention theory (goals, resources,
activities, outcomes)
Key steps in evaluability Assessment
◦
Analyze stakeholders perceptions of the
program
◦
Assess target population(s)
◦
Discuss differences in outcome perceptions
◦
Determine plausibility of intervention model
◦
Discuss validity of the program
◦
Decide about continuation (= full evaluation)
Key EA Questions
1.
Does the program serve the population for whom it was designed?
2.
Does the program have the resources (available/used) as scheduled in the
program design?
3.
Are the program activities being implemented as designed?
4. Does the program have the capacity to provide data for
an evaluation?
Crucial Steps
EA has five crucial tasks that an evaluator must successfully complete:
• Task 1.
Study the program history, design, and operation;
• Task 2.
Watch the program in action (participants)
• Task 3.
Determine the program’s capacity for data collection,
management and analysis;
• Task 4.
Assess the likelihood that the program will reach its
goals and objectives (
consistency
)
• Task 5.
Show whether an evaluation is feasibly (will help the
program and its stakeholders).
Conclusion
The overall purpose of an Evaluability Assessment is to inform the timing of an
evaluation and to improve the prospects for an evaluation producing useful
results.
Evaluability assessments can be configured to cast new light on developments in
the field families or clusters of policies and programs.
Although the most common focus of an Evaluability Assessment is a single
project, Evaluability Assessments should also been carried out on sets of
projects, policy areas, country strategies, strategic plans, work plans, and
partnerships to increase chances of utilization
Evaluability Assessment is a critical pre-evaluation appraisal to determine if a program's performance can be evaluated. It assesses the program logic, implementation status, and feasibility for conducting rigorous evaluations. Loss of evaluability can occur due to various factors, as highlighted in the review of Dutch private sector programs. Key steps in evaluability assessment involve initial diagnostics, stakeholder identification, defining program boundaries, analyzing program documents, and clarifying intervention theory.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Evaluability Assessment critical for successful Evaluations By Sylvester Kato Technical M&E in the BMAU UEA Conference, February 2019 Kampala Uganda Annual Evaluation conference February 11th to 15th2019
Evaluability Assessment Evaluability Assessment
Presentation outline Introduction to EA Where is Evaluability lost Key steps in evaluability Assessment Key EA Questions Crucial Steps Conclusion
EvaluabilityAssessment Evaluability assessment is a pre-evaluation appraisal of whether a program s performance can be evaluated and if so whether it should be (Rossi, Freeman, & Lipsey 1999). Systematic process to determine whether program evaluation is feasible. Decide whether an evaluation is useful to provide timely, relevant findings for decision makers. Evaluation research will be pointless if the program itself cannot be evaluated. Yes, some type of study is always possible, but a study specifically to identify the effects of a particular program may not be possible within the available time and resources. So researchers may conduct an evaluability assessment to learn this in advance, rather than expend time and effort on a fruitless project.
EvaluabilityAssessment Assesses: Underlying program logic Current state of program implementation Feasibility of conducting rigorous outcomes-focused evaluation or other sorts of evaluation
Loss of Evaluability Loss of Evaluability Review of Dutch Private Sector Programs 2/3 of executed evaluations cannot be used Major reasons: - Evaluation agency not fully independent - stated objectives too broad/vague - no clear indicators defined - data at too aggregate level - absence of baseline data - no representative sampling - too general intervention theory
Key steps in evaluability Assessment Smith (1989): ten steps procedure for Evaluability Assessment initial diagnostics Identify relevant stakeholders Define boundaries of the program Analyze available program documents Clarify intervention theory (goals, resources, activities, outcomes)
Key steps in evaluability Assessment Analyze stakeholders perceptions of the program Assess target population(s) Discuss differences in outcome perceptions Determine plausibility of intervention model Discuss validity of the program Decide about continuation (= full evaluation)
Key EA Questions 1.Does the program serve the population for whom it was designed? 2.Does the program have the resources (available/used) as scheduled in the program design? 3.Are the program activities being implemented as designed? 4. Does the program have the capacity to provide data for an evaluation?
Crucial Steps EA has five crucial tasks that an evaluator must successfully complete: Task 1. Task 2. Task 3. Study the program history, design, and operation; Watch the program in action (participants) Determine the program s capacity for data collection, management and analysis; Assess the likelihood that the program will reach its goals and objectives (consistency) Show whether an evaluation is feasibly (will help the program and its stakeholders). Task 4. Task 5.
Conclusion The overall purpose of an Evaluability Assessment is to inform the timing of an evaluation and to improve the prospects for an evaluation producing useful results. Evaluability assessments can be configured to cast new light on developments in the field families or clusters of policies and programs. Although the most common focus of an Evaluability Assessment is a single project, Evaluability Assessments should also been carried out on sets of projects, policy areas, country strategies, strategic plans, work plans, and partnerships to increase chances of utilization
