The Basic Principles of Program Evaluation
Basic Principles
of Program Evaluation
Corinne Datchi-Phillips, Ph.D.
CEBP Learning Institute
May 26
th
, 2010
Breakout Session Objectives
1.
Participants will increase their understanding of the
purpose and goals of program evaluation.
2.
Participants will be able to identify what questions
they may ask to improve their correctional
programming.
3.
Participants will become knowledgeable about the
goals of different types of program evaluation.
4.
Participants will learn about existing resources they
may use to guide the evaluation of their services.
5.
Participants will be introduced to the
recommendations of the CEBP regarding data
collection in program evaluation.
Basic Principles
of Program Evaluation
The goal and purpose
of program evaluation
Program Evaluation
•
Purpose
–
To provide administrators with
information about program activities and
outcomes
–
To contribute to the delivery of quality
service to people in need
Program Evaluation
•
Goals:
–
Improve correctional treatment and
supervision programs (enhance
effectiveness and quality of service);
–
Inform program-level decisions;
–
Help public officials meet the obligation
of using resources effectively by
identifying and providing cost-effective
programs.
Program Evaluation
•
Goals
–
To inform decisions by identifying
•
The need for specific services
•
The likelihood that these services will be
utilized
•
What amount of service is necessary to meet
the needs that have been identified
Program Evaluation
•
Goals
–
To inform decisions about the selection and
implementation of correctional treatment
and supervision programs
•
Whether to continue offering a program if it is
successful
•
Whether to discontinue a program because it fails
to meet a need in the community
•
How to alter a program to make it more useful to
participants (Continuous quality improvement)
–
What are the limitations of the program? How can
these limitations be addressed?
Program Evaluation
•
Important questions about program
implementation:
–
Is a program implemented as planned?
•
Deviations from the original plan of a
program may have harmful effects or may
result in no or minimal benefit.
–
What are the outcomes of a program?
•
Reduction of criminogenic needs
•
Increased public safety (reduction of
recidivism rates)
Program Evaluation
•
Important question about resource
utilization:
–
What programs produce the most
favorable outcomes?
•
When programs are equally effective with
this type of client with this type of problem in
this type of setting, the least expensive
program may be selected.
Basic Principles
of Program Evaluation
Common types
of program evaluation
Program Evaluation
•
Different types
1.
Need Evaluation
2.
Implementation Evaluation
3.
Outcome Evaluation
4.
Cost-effectiveness Analysis
Need Evaluation
•
Basic first step that precedes program
planning and implementation
•
Goal
–
To identify and measure critical unmet
needs within a community
•
Nature and magnitude of social,
psychological, and behavioral problems
•
Community-based resources and services
already available to address the problems
Implementation Evaluation
•
After a program has been developed
and begun
–
What parts of the program have been
implemented or not
–
What are barriers to implementation
–
Who is being served by the program
–
Match between program implementation
and program plan
Outcome Evaluation
•
After satisfactory assessment of program
implementation
–
Different levels of complexity:
•
Are program participants performing or
functioning better ?
•
Are program participants performing or
functioning better than those not receiving the
services?
•
Does the program cause a positive change in the
performance or functioning of the participants?
(Causal relationships are more difficult to establish)
Cost-effectiveness Analysis
•
After it has been established that a
program has been implemented
effectively with good outcomes
–
Cost-effectiveness analysis answers the
question of efficiency.
•
A successful program that requires a large
amount of resources may not be a viable option.
•
Among the programs that have a similar positive
impact on a specific social and behavioral
problem, which ones cost less?
–
Only compare programs that are designed to achieve
the same outcomes with the same group of clients
Basic Principles
of Program Evaluation
Resources
Resources
•
Robert Wood Johnson Foundation’s
Guide to Evaluation Primers
–
http://www.rwjf.org/pr/product.jsp?id
=18657
•
W.K. Kellogg Foundation’s Evaluation
Handbook
–
http://www.wkkf.org/knowledge-
center/resources/2010/W-K-Kellogg-
Foundation-Evaluation-Handbook.aspx
Resources
•
The Evaluation Exchange
periodical,
Harvard University
–
http://www.hfrp.org/evaluation/the-
evaluation-exchange/overview
•
Center for Disease Control’s
Framework for Program Evaluation in
Public Health
–
http://www.cdc.gov/eval/framework.ht
m
Basic Principles
of Program Evaluation
Integrating program evaluation with
routine practices:
The importance of ongoing data
collection
Integrating evaluation into routine practices
•
Program evaluation as a part of program
management
–
Requires the consistent, ongoing collection and
analysis of information that will inform program
decision making
•
Detailed description of the program
(target population;
problem to be addressed; goals and objectives;
program activities; mechanisms of change)
•
After determining which questions you would like to
answer about a specified program, identify what
information you will need to answer these questions
•
Develop a plan for collecting the information
–
Collect the information from multiple sources and
perspectives (clients, staff, documents, observations)
Data Gathering
•
CEBP recommendations (2009)
–
Basic data elements necessary to perform
program outcome evaluations
•
A survey of current community corrections
practices in the IDOC, recommendations p.
57, Appendix B p. 71
http://education.indiana.edu/Default.aspx?a
lias=education.indiana.edu/cebp-ri
Data Gathering
•
CEBP recommendations (2009)
–
Basic data elements that will help answer
the question “what works?” (effects of
community corrections programs)
•
With whom?
–
Type and severity of problems
–
Human differences (age, sex, etc.)
–
Cultural differences
•
Under what conditions?
–
Treatment dosage
–
Combination of services/programs
Data Gathering
•
CEBP recommendations (2009)
–
Specification of community corrections
programming
•
Target population; problem to be addressed;
goals and objectives; program activities;
mechanisms of change
–
Effectiveness of community corrections
programming
•
Program completion rates (criteria)
•
Measures of behavioral change (improvement in
performance and functioning)
•
Recidivism rates
Basic Principles
of Program Evaluation
Conclusions
& Questions
Program Evaluation
•
Essential to evaluate correctional practices
–
To identify practices that are useful, effective,
and feasible under particular conditions
–
To improve programming that address the
needs of community corrections clients and
communities
–
To support accountability in community
corrections
•
Using scientific methods and knowledge to inform
decisions and actions
Program Evaluation
•
Common concerns about program evaluation
–
Belief that the evaluation will limit providers’
capacity to be flexible in their implementation of
programs
•
Flexibility within broad structure of the program
–
Belief that negative evaluation will result in
termination of a program
•
Will more likely inform refinement of the program
–
Belief that information will be misused (e.g.,
threat to promotion of staff)
•
Program evaluation is distinct from individual staff
assessment
Program Evaluation
•
Common concerns about program
evaluation
–
Belief that staff’s experiential knowledge will
not be taken into consideration
•
Staff’s observations constitute data that can
inform questions about program implementation.
–
Belief that staff and administrators will no
longer be able to make decisions about the
way the program is offered
•
Program evaluation provides information that
staff and administrators can use to make changes
in the way the program is offered to improve
delivery.
Program Evaluation
Questions & Comments
This informative content discusses the fundamental principles, goals, and purposes of program evaluation in correctional settings. It covers the objectives of program evaluation, such as improving treatment programs, informing decisions, and enhancing service quality. The key goals include identifying the need for specific services, evaluating program effectiveness, and making decisions on program continuation or modifications for continuous improvement. The content emphasizes the importance of utilizing evaluation data to enhance correctional treatment and supervision programs effectively.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Basic Principles of Program Evaluation Corinne Datchi-Phillips, Ph.D. CEBP Learning Institute May 26th, 2010
Breakout Session Objectives 1. Participants will increase their understanding of the purpose and goals of program evaluation. Participants will be able to identify what questions they may ask to improve their correctional programming. Participants will become knowledgeable about the goals of different types of program evaluation. Participants will learn about existing resources they may use to guide the evaluation of their services. Participants will be introduced to the recommendations of the CEBP regarding data collection in program evaluation. 2. 3. 4. 5.
Basic Principles of Program Evaluation The goal and purpose of program evaluation
Program Evaluation Purpose To provide administrators with information about program activities and outcomes To contribute to the delivery of quality service to people in need
Program Evaluation Goals: Improve correctional treatment and supervision programs (enhance effectiveness and quality of service); Inform program-level decisions; Help public officials meet the obligation of using resources effectively by identifying and providing cost-effective programs.
Program Evaluation Goals To inform decisions by identifying The need for specific services The likelihood that these services will be utilized What amount of service is necessary to meet the needs that have been identified
Program Evaluation Goals To inform decisions about the selection and implementation of correctional treatment and supervision programs Whether to continue offering a program if it is successful Whether to discontinue a program because it fails to meet a need in the community How to alter a program to make it more useful to participants (Continuous quality improvement) What are the limitations of the program? How can these limitations be addressed?
Program Evaluation Important questions about program implementation: Is a program implemented as planned? Deviations from the original plan of a program may have harmful effects or may result in no or minimal benefit. What are the outcomes of a program? Reduction of criminogenic needs Increased public safety (reduction of recidivism rates)
Program Evaluation Important question about resource utilization: What programs produce the most favorable outcomes? When programs are equally effective with this type of client with this type of problem in this type of setting, the least expensive program may be selected.
Basic Principles of Program Evaluation Common types of program evaluation
Program Evaluation Different types 1. Need Evaluation 2. Implementation Evaluation 3. Outcome Evaluation 4. Cost-effectiveness Analysis
Need Evaluation Basic first step that precedes program planning and implementation Goal To identify and measure critical unmet needs within a community Nature and magnitude of social, psychological, and behavioral problems Community-based resources and services already available to address the problems
Implementation Evaluation After a program has been developed and begun What parts of the program have been implemented or not What are barriers to implementation Who is being served by the program Match between program implementation and program plan
Outcome Evaluation After satisfactory assessment of program implementation Different levels of complexity: Are program participants performing or functioning better ? Are program participants performing or functioning better than those not receiving the services? Does the program cause a positive change in the performance or functioning of the participants? (Causal relationships are more difficult to establish)
Cost-effectiveness Analysis After it has been established that a program has been implemented effectively with good outcomes Cost-effectiveness analysis answers the question of efficiency. A successful program that requires a large amount of resources may not be a viable option. Among the programs that have a similar positive impact on a specific social and behavioral problem, which ones cost less? Only compare programs that are designed to achieve the same outcomes with the same group of clients
Basic Principles of Program Evaluation Resources
Resources Robert Wood Johnson Foundation s Guide to Evaluation Primers http://www.rwjf.org/pr/product.jsp?id =18657 W.K. Kellogg Foundation s Evaluation Handbook http://www.wkkf.org/knowledge- center/resources/2010/W-K-Kellogg- Foundation-Evaluation-Handbook.aspx
Resources The Evaluation Exchange periodical, Harvard University http://www.hfrp.org/evaluation/the- evaluation-exchange/overview Center for Disease Control s Framework for Program Evaluation in Public Health http://www.cdc.gov/eval/framework.ht m
Basic Principles of Program Evaluation Integrating program evaluation with routine practices: The importance of ongoing data collection
Integrating evaluation into routine practices Program evaluation as a part of program management Requires the consistent, ongoing collection and analysis of information that will inform program decision making Detailed description of the program (target population; problem to be addressed; goals and objectives; program activities; mechanisms of change) After determining which questions you would like to answer about a specified program, identify what information you will need to answer these questions Develop a plan for collecting the information Collect the information from multiple sources and perspectives (clients, staff, documents, observations)
Data Gathering CEBP recommendations (2009) Basic data elements necessary to perform program outcome evaluations A survey of current community corrections practices in the IDOC, recommendations p. 57, Appendix B p. 71 http://education.indiana.edu/Default.aspx?a lias=education.indiana.edu/cebp-ri
Data Gathering CEBP recommendations (2009) Basic data elements that will help answer the question what works? (effects of community corrections programs) With whom? Type and severity of problems Human differences (age, sex, etc.) Cultural differences Under what conditions? Treatment dosage Combination of services/programs
Data Gathering CEBP recommendations (2009) Specification of community corrections programming Target population; problem to be addressed; goals and objectives; program activities; mechanisms of change Effectiveness of community corrections programming Program completion rates (criteria) Measures of behavioral change (improvement in performance and functioning) Recidivism rates
Basic Principles of Program Evaluation Conclusions & Questions
Program Evaluation Essential to evaluate correctional practices To identify practices that are useful, effective, and feasible under particular conditions To improve programming that address the needs of community corrections clients and communities To support accountability in community corrections Using scientific methods and knowledge to inform decisions and actions
Program Evaluation Common concerns about program evaluation Belief that the evaluation will limit providers capacity to be flexible in their implementation of programs Flexibility within broad structure of the program Belief that negative evaluation will result in termination of a program Will more likely inform refinement of the program Belief that information will be misused (e.g., threat to promotion of staff) Program evaluation is distinct from individual staff assessment
Program Evaluation Common concerns about program evaluation Belief that staff s experiential knowledge will not be taken into consideration Staff s observations constitute data that can inform questions about program implementation. Belief that staff and administrators will no longer be able to make decisions about the way the program is offered Program evaluation provides information that staff and administrators can use to make changes in the way the program is offered to improve delivery.
Program Evaluation Questions & Comments
