Health Program Development and Evaluation Overview

King Saud University

King Saud University

College of Business Administration

College of Business Administration

Department of Health Administration

Department of Health Administration

Masters` Program

Masters` Program

PA 519–

Health Programs Planning and Evaluation

Second Semester 1439/ 1440

Mohammed S. Alnaif, Ph D.

1

04/04/1446

Dr. Mohammed Alnaif

Learning Objectives



Describe 

Program development, program

planning and program evaluation

.



Understand the cycle of 

program planning and

program evaluation

.



Introduces types of program 

evaluation.



Use of the public health pyramid in 

program

planning and evaluation

.

2

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

The Public Health System (S

takeholders)

Source:  Centers for Disease Control and Prevention

04/04/1446

Dr. Mohammed Alnaif

3

All Programs/Projects have (implicitly or explicitly):



Objectives



Expected outcomes



Target population



Mechanism(s) to deliver services (the

intervention)



Criteria for participating in the program



A conceptual framework that provides rationale

for program existence (sometimes called the

Development Hypothesis

).

4

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

All Programs/Projects have (implicitly or explicitly):



Project planning 

involves a series of steps that

determine how to achieve a particular community

or organizational goal or set of related goals. This

goal can be identified in a community plan or a

strategic plan. Project plans can also be based on

community goals

.



A 

Program

 is a group of clear, related,

complementary activities that are intended to

achieve a desired outcome among the target

group(s).

5

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

All Programs/Projects have (implicitly or explicitly):



When organizations set out to make the world a

better place, they develop 

programs

 to effect changes

for individuals or groups.



Programs

 vary, but the basic components of

programs are essentially the same.



All 

programs

 have objectives and a set of activities

carried out by staff, volunteers and participants that

are intended to achieve those objectives.



Programs

 operate within an organization and often

within a broader system of similar programs.

6

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

All Programs/Projects have (implicitly or explicitly):



Program

 design is the process that an

organization uses to develop a program.



It is most often an iterative process involving

research, consultation, initial design, testing and

redesign.



A 

program

 design is the plan of action that results

from that process.

7

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Five reasons to do program design and development 

You want to make sure that: 

1.

There is really a need for the 

program

; 

2.

The 

design

 you are considering actually meets the

needs of the intended participants; 

3.

The 

program

 you are considering has been shown to

be effective; 

4.

The 

program

 has been appropriately adapted to fit

your context; and 

5.

The 

program

 is actually working for the participants.

8

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Stakeholder consultation



Programs

 can have many stakeholders with many

different points of view, depending on whether they

are staff, participants, funders or other community

service providers.



Balancing the values and principles of a 

program

 with

the practicalities of implementation often means

compromise. But what compromises are likely to

undermine the core values of the program?



Stakeholder consultation 

can help to determine what

is important, particularly to those implementing and

those using the program.



It can also contribute to adapting a 

model

.

9

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

What a program design includes:



Need for the program 

– assesses what a

population needs, what is available in community

and the gap between what is needed and what is

available (e.g. an organization serving youth notes

that many of its participants need mental health

supports, but need first to determine whether

there is a broader need and whether there are

existing programs).



Program objectives and outcomes 

– what the

program is intended to accomplish for the

particular target population

.

10

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

What a program design includes:



Program principles 

– these are the fundamental norms, rules and values

that guide the program (e.g. being inclusive, client-focused).



Program components 

– a specific set of inter-connected activities.



provided. Budget/cost 

– what it will cost to implement the program.



Funding strategy 

– how the program will be funded. Can it be done

within the current organizational budget? Usually not, so organizations

look to funders like the municipality, province, federal government. The

funding strategy needs to include start-up and long-term funding.



Governance 

– some programs are set within an existing organization or

some programs may be collaboration among two or more organizations.

The design needs to include the process for decision-making (e.g. an

organization may have a board program committee that has

responsibility for design of new programs as well as redesign of existing

programs)

.

11

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

12

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

13

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Steps for designing and developing a program

1.

Identify a potential need 

- This is not evidence of the

need for a program; it is only evidence of the needs of

people within the community or a population. This

may create an opportunity for new program which

will usually require additional resources or.

2.

Conduct a needs assessment 

– in order to determine

the needs of a specific target population gather and

analyze data taken from different sources in a

systematic way, looking at the needs of the population

and the existing programs in order to determine

whether there are gaps.

14

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Steps for designing and developing a program

3.

Research potential programs for filling need 

– deciding on how

best to address an identified need requires searching for and

reviewing existing programs. Contextual issues related to the

demographic population, location and magnitude of the need are

considered before selecting an appropriate program.

4.

Select an evidenced-based design 

– after a thorough review of

potential programs, select one that shows strong or promising

evidence of being effective. This means the design has been

previously implemented and evaluated and has demonstrated some

ability to produce a desired outcome for the target group. If such a

program does not exist or is not suitable for the context,

components of a program may be selected and adapted or a new

program may be designed. If the program is new or innovative,

that should be noted with reasons given for trying something new.

15

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Steps for designing and developing a program

5.

Develop the program design 

– using a logic

model (or theory of change) as a tool, create a

detailed plan that specifies program activities,

resource allocations and expected outcomes.

The theory of change 

specifies the underlying

assumptions of the program. 

Program design

consolidates all of the information gathered

during the development process and sets out

the 

program’s outcomes, outputs, activities

and resources

.

16

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Steps for designing and developing a program

6.

Pilot the program 

– before implementing a

full-scale 

program

, it is important to test

whether it works in your particular context, if

possible. 

Piloting

 includes developing the

program

 on a small scale so you can gauge the

level of readiness to deliver the program,

determine whether it is making a difference

for program users, and assess what, if any,

changes are needed. A good pilot sets the stage

for ongoing 

monitoring

.

17

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Steps for designing and developing a program

7.

Roll out the program 

– if the 

pilot

 indicates

the 

program

 works, you need to select a site,

hire and train program staff, establish

administrative processes, start service

delivery, and carry out ongoing 

monitoring

and quality control.

8.

Monitor and evaluate 

– while testing or

implementing the program, it is important to

learn what’s working well and what isn’t. It is

also helpful to review the program’s original

goals.

18

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Steps for designing and developing a program

8.

Monitor and evaluate 

– 

By using a systematic

process that gathers and analyzes data from a variety

of sources, steps can be taken to modify and improve

the program before rolling it out or as it gets

implemented. This also supports accountability to

funders.

Consult with stakeholders 

– speaking with and gathering

information from all people involved with the program

generates buy-in and surfaces diverse perspectives during

the 

planning process

. Stakeholder consultation must be

genuine and responsive to be effective, and should take

place throughout the 

planning process

.

Program planning and planning for a program

evaluation



Recently 

planners

 have been attempting to foster

connections with public health professionals and

improve health through planning actions.



As 

issues

 such as obesity, asthma, venomous

pollutants, and mental health concerns have

grown in prominence, there is increasing interest

in the potential for changes in the built

environment to alleviate such problems.

19

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Program planning and planning for a program

evaluation



We need to distinguish between 

program

planning and planning for a program evaluation

,

which can take place at any program stage of

development.



Health Planning 

is the identification and

elaboration (within existing resources) of means

and methods for providing in the future, effective

health care relevant to identified health needs for

a defined population.

20

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Program planning and planning for a program

evaluation



Aim of 

health planning 

is to maintain and improve

health status of a given community. It is achieved

through provision of health services which are

accessible, effective, and equitable and of a quality to

ensure their appropriate utilization

.



A 

plan of operations 

is the (written) summary of the

implementation plan specifying sequence and timing

of, and responsibilities for, implementation of main

objectives, (expected) results and activities.

21

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Rationale for Health Planning

Planning of delivery of effective health services to the

population with in resources provided



Translation

 of “

new policy

” statement into operational

plan



Re-planning

 on the basis of an already existing plan for the

purpose of reviewing existing health problems and needs

and rendering services more effective and efficient



Emergence

 of a new problem (e.g. SARS, natural disaster)



Planning

 of health services for a population where no

organized health care delivery system as yet, or where an

existing one is being extremely revised or re-organized.

22

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Interdependent and Cyclic Nature of Planning and

Evaluation



The current thinking about programs and evaluation

is that the activities constituting program planning

and program evaluation are cyclical and

interdependent

(FIGURE 1-1)



The 

stages

 are cyclical to the extent that the end of

one program or stage is flows almost seamlessly into

the next program or planning activity



Interdependence

 of activities and stages ideally result

from information and data feedback loops that

connect the stages

23

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

24

(FIGURE 1-1) HEALT H PROGRAM DEVLOPMENT and EVALUATION

Health Program

Planning

Assessment of

Community needs and

Assets

Program and

Evaluation Planning

Effects Evaluation

Participant/Recipient

Health Outcome and

Impact

Process Evaluation

Program

Implementation of

Process and Effect

Theory

Priority

established

Statement of

health

problems

Health status

changes

Findings from

the evaluation

Findings from

the evaluation

Evaluation

design and

methodology

Process

theory and

effect theory

delineated

Intervention effect

04/04/1446

Dr. Mohammed Alnaif

Interdependent and Cyclic Nature of Planning

and Evaluation



The cycle begin with a trigger event, such as



awareness of health problems;



a periodic strategic planning effort;



a process required by a stakeholder, such as a

grant renewal or a newly available fund for a

health program.



Failure of a health program



The trigger might also be a news media expose or

a legal action.

25

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Interdependent and Cyclic Nature of Planning

and Evaluation



The trigger event or situation leads to the

collection of data about the problem, the

characteristics of the people affected, and their

perception of the health problem.



Based on the data from the needs assessment;



Program development begin,



Problems and their solutions are prioritized

.

26

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Major Components of  the Planning Phase



Developing the program theory;



Assessment for organizational and infrastructure

resources for implementing the program;



Setting goals and objectives that are derived from the

program theory;



Implementation of the program;



Evaluation plan and cycle.

27

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

04/04/1446

Dr. Mohammed Alnaif

28

1

Define

Issues

7

Evaluation

2

Situational

Assessment

6

Implementation

4

Approve

Decision

5

Allocate

Resources

3

Discussion &

Recommendation

Evidence and Practice Based Planning Framework

Major Components of  the Planning Phase



Developing the program theory;



Assessment for organizational and infrastructure

resources for implementing the program;



Setting goals and objectives that are derived from the

program theory;



Implementation of the program;



Evaluation plan and cycle.

29

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Program life cycle



A program may begin as a pilot;



The program may not rely on any existing format

or theory;



Model programs;



Regardless of the stage in a  program’s life cycle then

major planning and evaluation stages of community

assessment and evaluation are carried out;



The life cycle of a health program is reflected in the

evolution of hospice care.

30

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

The Fuzzy Aspects of planning



Several 

Paradoxes

 pervade health planning;



That planning is shaped by the same forces that created

the problems that planning is supposed to correct.



Prosperity in our modern world has its own associated

health risks.



What may be easier and more effective may be less

acceptable 

(active protection, passive protection).



Those in need ideally but rarely trigger the planning of

health programs.



The fact is that planning is intended to be successful.



Most planning is for making changes not for creating

stability.

31

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Assumption, 

Assumptions also influence the

effectiveness of planning.



The first 

is that a solution, remedy, or appropriate

intervention can be identified or developed and

provided. Without this assumption, planning would be

pointless.



It is fundamentally an optimistic assumption about

the capacity of the planners, the stakeholders, and the

state of the science to address the health problem.



The assumption of adequate capacity and knowledge

is actually tested through the process of planning

.

32

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Assumption, 

Assumptions also influence the effectiveness

of planning.



Planning

 leads to the allocation of resources needed to address the

health problem.



This 

assumption

 is challenged by the reality that 

four groups 

of

stakeholders

 have interests in the decision making regarding

health resources and each group exists in all program planning.

1.

Members of the target audience for the health program

2.

Health payers, such as insurance companies and philanthropic

funding agencies

3.

Individual healthcare providers and healthcare organizations

and networks

4.

The general public are effected by how resources are allocated

for health programs

33

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Assumption, 

Assumptions also influence the

effectiveness of planning.

Another 

assumption

 about those involved is that they

share similar views on how to plan health programs.



During the planning process their points of view and

cultural perspectives will likely come into contrast.



Planners need to know what is relevant and important

for the problem at hand.



There are competing purposes and values.



Effective planning requires tolerance, freedom, and

fairness

34

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Assumption, 

Assumptions also influence the

effectiveness of planning.



The resulting 

conflicts

 among the 

stakeholders

 for the

limited resource apply whether they are allocating

resources across the healthcare system or among

programs for specific health problems.



Limited resources, whether real or not, raise 

ethical

questions

 of what to do when possible gains from

needed health programs or policies are likely to be

small, especially when the health program addresses

serious health problems.

35

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Assumption, 

Assumptions also influence the

effectiveness of planning.



Another assumption about the planning process is that

it occurs in an orderly fashion and that a rational

approach is best.



Four key elements are inherent in planning;

uncertainty, ambiguity, risk

, and 

control

.



The presence of each of these elements contradicts the

assumption of rational approach, and each generates its

own paradoxes.

36

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Uncertainty, Ambiguity, Risk, and Control



The process of 

planning

 is affected by the limits of both

scientific rationality and the usefulness of data to cope

with the 

uncertainties

, 

ambiguities

, and 

risks

 being

addressed by the planning process (see 

TABLE 1-3

).



Uncertainty

 is the unknown likelihood of a possible

outcome.



Four type of 

uncertainty

 are identified:



types and amount of resources

,



technological

,



market receptivity to the product

, and



organizational

.

37

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

04/04/1446

Dr. Mohammed Alnaif

38

Uncertainty, Ambiguity, Risk, and Control



Ambiguity

 is doubt about a course of action

stemming from awareness that known and

unknown factors exist that can decrease the

possibility of certainty.



In this sense, 

ambiguity

 results in 

uncertainty



Both 

uncertainty

 and 

ambiguity

 pervade the

planning

 process because it is impossible to

know and estimate the effect of all relevant

factors.

39

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Uncertainty, Ambiguity, Risk, and Control



Ambiguity

 is the characteristic of not having a

clear or single meaning.



Change, or the possibility of change, is a

possible source of 

ambiguity

.



When 

ambiguity

 is ignored, the resulting

differences in interpretation can lead to

confusion and conflict among stakeholders.



Openly and constructively addressing the

ambiguity

 and any associated conflict can lead

to innovations in the program.

40

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Uncertainty, Ambiguity, Risk, and Control



Risk

 is the perceived possibility or uncertain

probability of an adverse outcome in a given

situation.



Risk

 is not just about taking chances but it is also

about uncertainty and ambiguity ( as in the case

with estimates of cure rates and projections about

future health conditions).



Risk

 is pervasive and inherent throughout the

planning process in terms of deciding  who to

involve and how, which planning approach to use,

which intervention to use, and I estimating which

health problem deserves attention.

41

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Uncertainty, Ambiguity, Risk, and Control



Control

, as in being in charge of managing, is a

natural reaction to the presence of 

ambiguity

,

conflict

, and 

risk

.



It can take the form of directing attention and

allocating resources or of exerting dominance over

others.



Addressing the 

ambiguity

, 

uncertainty

, and 

risk

that might have been the trigger for the 

planning

process require less—not more—

control

.



Those who preside over and influence the 

planning

process are often thought of having 

control

 over

solutions to the health problem or condition.

42

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Some think of 

evaluations

 as falling into one of two

broad categories: 

formative

 and 

summative

evaluations.



Formative evaluations 

are conducted during

program development and implementation and

are useful if you want direction on how to best

achieve your goals or improve your program.



Summative evaluations 

should be completed once

your programs are well established and will tell

you to what extent the program is achieving its

goals.

43

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Several major types of activities are classified as

evaluations

. each type of activity requires a specific focus,

purpose, and set of skills

.



Which of these 

evaluations

 is most appropriate depends on

the stage of your program:



A community needs assessment



Process evaluations



Effect evaluations



Outcome evaluations



Impact evaluations



Cost evaluations



Comprehensive evaluations



Meta-evaluations

44

HEALT H PROGRAM DEVLOPMENT and

EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Unfortunately, contrasting 

summative evaluations 

with

formative evaluations 

as the 

two broad categories 

adds

more confusion to the evaluation terminology

.



Summative evaluations

, in the strictest sense, are done

at the conclusion of a program to provide a conclusive

statement regarding program effects.



The term 

summative evaluation 

is sometimes used to

refer to either an 

outcome

 or 

impact

 evaluation.



The term 

formative evaluation 

is used to refer to

program assessments that are performed early in the

implementation of the program and used to make

changes to the program.

45

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Community needs assessment 

(also known as

community health assessment) is  a type of

evaluation that is performed to collect data

about the health problems of a particular

group.



Needs Assessment, 

determines who needs the

program, how great the need is, and what can

be done to best meet the need.

46

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



The data collected for this purpose are then

used to tailor the health program to the need

and distinctive characteristics of that group.



A community needs assessment 

is a major

component of program 

planning

 because it is ,

done at an early stage in the 

program planning

and 

evaluation

cycle

.

47

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Process evaluations

, which begins at the same

time that the program starts,  and focus on the

degree to which the program has been

implanted as planned and on the quality of the

program implementation.



Process evaluations 

are known by a variety of

terms, such as monitoring evaluations,

depending on their focus and characteristics.

48

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



The underlining framework for designing a

process evaluation 

comes from the 

process theory

component of the overall program theory

developed during the planning stage.



The 

process theory 

delineates the logistical

activities, resources, and interventions needed to

achieve the health change in program participants

or recipients.



Information from 

process evaluation 

is used to

plan, revise, or improve the program.

49

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Effect evaluations 

answer a key question: did

the program make a difference?



Evaluators

 seek to use the most rigorous and

robust designs, methods, and statistics possible

and feasible when conducting an effect

evaluation.



Findings from 

effect evaluations 

are used to

revise the program and may be used in

subsequent initial program planning activities.

50

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Effect evaluations 

may be referred to as

outcome or impact evaluations

.



Outcome evaluations 

focus on more immediate

effects of the program, whereas 

impact

evaluations 

may have a more long-term focus.



Program planners and evaluators should

clarify meanings and address misconceptions

or misunderstandings.

51

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



A fourth type of evaluation focuses on

efficiency

 and 

costs

 associated with the

program.



Cost evaluations 

encompass a variety of more

specific cost-related evaluations

.



Cost effectiveness evaluations

, 

cost benefit

evaluations, and cost utility evaluations

.



Cost benefit evaluations, and cost utility

evaluations 

require expertise in economics.

52

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Nonetheless, small-scale and simplified 

cost

effectiveness evaluations

 can be done if good cost

accounting has been maintained by the program and

more sophisticated 

outcome or impact evaluation 

has

been conducted.



Because 

cost evaluations 

are performed late in the

planning and evaluation cycle

, their results are not

likely to be available in time to make program

improvements or revision.



Such 

evaluations

 are generally used during subsequent

planning stages to gather information for prioritizing

program options.

53

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Comprehensive evaluations

, the fifth type of evaluation,

involve analyzing 

needs assessment 

data, 

process

evaluation 

data, 

effect evaluation 

data, and 

cost

evaluation 

data as a set of data.



An integrated analysis of various types of data to draw

conclusions about the 

effectiveness

 and 

efficiency

 of

the program, 

comprehensive evaluations 

are relatively

uncommon.



A sixth type of evaluation is 

meta-evaluation

. A 

meta-

evaluation

 is done by combining the findings from

previous outcome evaluations 

of various programs for

the same health problem.

54

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



The purpose of 

meta-evaluation

 is to gain insights

into which of the various programmatic

approaches have had on the problem.



This type of 

evaluation

 relies on the availability of

existing information about evaluations and on the

use of specific set of 

methodological

 and 

statistical

procedures



Meta-evaluations

 are less likely to be done by

program personnel; instead, they are generally

carried out by evaluation researchers.

55

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Introduction to the Types of Evaluation



Meta-evaluation 

that are published are

extremely useful in program planning because

they indicate which programmatic

interventions are more likely to succeed in

having an effect on the participants.



Published 

meta-evaluations 

can also be

valuable in influencing health policy and

health funding decisions.

56

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Mandated and Voluntary Evaluation



Evaluations are not spontaneous events. Rather,

they are either 

mandated or voluntary

.



 A 

mandate to evaluate 

a program is always linked

in some way to the funding agencies, whether a

governmental body or a foundation.



If an 

evaluation is mandated

, then the contract for

receiving the program funding will include

language specifying the parameters and time line

for the mandated evaluation; whether the

evaluation will be done by project staff members

or external evaluators or both.

57

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Mandated and Voluntary Evaluation



Other evaluations may be linked to 

accreditation

that is required for reimbursement of services

provided, making the de facto mandated

evaluation.



For example, to receive 

accreditation

 from the

Joint Commission, a health service organization

must collect data over time on patient outcomes.

These data are then used to develop ongoing

quality improvement efforts.



Some 

accreditation

 agencies require that provider

organizations conduct a 

self-evaluation

 as an early

step in the accreditation process.

58

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

Mandated and Voluntary Evaluation



Completely 

voluntary evaluations 

are initiated,

planned, and completed by the project staff

members in an effort to make improvements.



However, given the relatively low reward from,

and cost associated with, doing and evaluation

when it is not required, these 

evaluations

 are likely

to be small with low scientific rigor.



Programs that engage in 

voluntarily evaluations

may have good intentions but they often lack the

skills and knowledge required to conduct an

appropriate evaluation

.

59

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif

When Not to Evaluate

There are some situations where 

evaluation

 may not

be a good idea. It is not advisable to attempt an

evaluation

 under the following four circumstances:



When there are no questions about the program,



When the program has no clear direction,



When stakeholders cannot agree on the program

objectives, and



When there is not enough money to conduct a

sound evaluation.

60

HEALT H PROGRAM DEVLOPMENT

and EVALUATION

04/04/1446

Dr. Mohammed Alnaif



Pyramids

 tend to be easy to understand

and work well to capture tiered concepts.



For these reasons, 

pyramids

 have been

used to depict the tiered nature of

primary health care, secondary

healthcare, and tertiary healthcare

services, the inverse relationship of effort

needed and health impact of different

interventions, and nutrition

recommendations.

61

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid

Health Impact 

Pyramid

04/04/1446

Dr. Mohammed Alnaif

62

Nutrition Recommendations

Pyramid

04/04/1446

Dr. Mohammed Alnaif

63



The 

public health pyramid 

is divided into

four sections 

(FIGURE 1-2)

.



The top, or the first, section 

of the

pyramid

 contains direct healthcare

services, such as medical care,

psychological counseling, hospital care,

and pharmacy services.



At this level 

of the pyramid, programs

are delivered to individuals, whether

patients, clients, or even students.

64

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid



Generally

, programs at the direct services

level have a direct, and often relatively

immediate, effect on individual participants

in the health program.



Direct services of these types appear at the

tip of the 

pyramid

 to reflect that, overall,

the smallest proportion of a population

receives them.



These interventions, according to the

Health Impact Pyramid

, require

considerable effort, with minimal

population effects.

65

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid



At the second level 

of the 

pyramid

 are enabling services,

which are those health and social services that support or

enhance the health of aggregates.



Aggregates

 are used to distinguish between individuals and

populations; they are groups of individuals who share a

defining characteristic, such as mental illness or a terminal

disease.



Examples

 of enabling services include mental health drop-in

centers, hospice programs, financial assistance programs

that provide transportation to medical care, community-

based case management for patients with acquired immune

deficiency syndrome (AIDS), low income housing, nutrition

education programs provided schools, and workplace child

care centers.

66

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid



As this list of programs demonstrates, the

services at this level 

may directly or

indirectly contribute to the health of

individuals, families, and communities and

are provided to aggregates.



Enabling services 

can also be thought of as

addressing some of the consequences of

social determinants of health.

67

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid



The next, 

more encompassing level 

of the public

health pyramid is population-based services.



At the 

population level 

of the pyramid, services are

delivered to an entire population, such as all persons

residing in a city, region, or country.



Examples

 of 

population services 

include

immunization programs for all children in the

country, newborn screening for all infants born in a

city, food safety inspections carried out under the

auspices of state regulations, workplace safety

programs, and nutrition labeling on food.

68

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid



As this list reflects, the distinction between an

aggregate

 and a 

population

 can blurry.



Programs at this level 

typically are intended to

reach an entire population, sometimes without

the conscious involvement of individuals.



In this sense, individuals receive a 

population-

based

 health program, such as water

fluoridation, rather than participating in the

program, as they would in a smoking-cessation

class.

69

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid



Interventions and programs 

aimed at

changing the socioeconomic context within

which populations live would be included at

this population level of the pyramid.



Such programs are directed at changing one

or more 

social determinants 

of health.



Population-level programs 

contribute to the

health of individuals and, cumulatively, to the

health status of the population.

70

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid

Infrastructure Building 

Services



Supporting the pyramid 

at its base is the

infrastructure of the healthcare system and the

public health system.



The health services at other 

pyramid levels 

would not

be possible unless there were skilled, knowledgeable

health professionals; laws and regulations pertinent

to the health of the people; quality assurance and

improvement programs; leadership and managerial

oversight; health planning and program evaluation;

information systems; and technological resources.

71

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid

Infrastructure Building 

Services



The planning and evaluation 

of health programs at

the direct, enabling, and population services levels is

itself a component of the infrastructure; these are

infrastructure activities.



In addition

, planning programs to address problems

of the infrastructure, as well as to evaluate the

infrastructure itself, are needed to keep the health

and public health system infrastructure strong,

stable, and supportive of the myriad of health

programs.

72

04/04/1446

Dr. Mohammed Alnaif

The Public Health Pyramid

04/04/1446

Dr. Mohammed Alnaif

73

Enabling

Services

Population-Based

Services

Infrastructure Building

Services

Examples: Medical Care, psychological counseling,

hospital care, and pharmacy services

Examples: Hospice programs, financial

assistance programs, transportation to medical

care

Examples: Planning and

evaluations, policy

development, quality control,

information systems

Examples: Immunization programs for

all children, food safety, workplace

safety

Direct

Healthcare

Services

(FIGURE 1-2) 

Use of the Public Health Pyramid in

Program Planning and Evaluation



Health programs exist across the 

pyramid levels

, and

evaluations of these programs are needed.



At each level of the 

pyramid

, certain issues unique to

that level must be addressed in developing health

programs.



The types of health professionals and the types of

expertise needed vary by 

pyramid level

.



Each level of the 

pyramid

 is characterized by unique

challenges for evaluating programs.



This is why, the 

public health pyramid

, as a

framework, helps illuminate those differences, issues,

and challenges.

74

04/04/1446

Dr. Mohammed Alnaif

Use of the Public Health Pyramid in

Program Planning and Evaluation



The 

public health pyramid 

provides reminders that

various aggregates of potential audiences exist for any

health problem and program and that health programs

are needed across the pyramid.



Depending on the healthy discipline and the

environment in which the 

planning

 is being done,

direct service programs may be the natural or only

inclination.



The 

public health pyramid 

provides a framework for

balancing the level of the program with meeting the

needs of the broadest number of people with a given

need.

75

04/04/1446

Dr. Mohammed Alnaif

Use of the Public Health Pyramid in

Program Planning and Evaluation



The 

public health pyramid 

also serves as a reminder

that stakeholder alignments and allegiances may be

specific to a level of the pyramid.



The 

savvy program planner 

considers not only the

potential program but also the stakeholders who are

likely to make themselves known during planning

process.



The 

public health pyramid 

has particular relevance

for public health agencies concerned with addressing

the three functions of public health; assessment,

assurance, and policy.

76

04/04/1446

Dr. Mohammed Alnaif

Use of the Public Health Pyramid in

Program Planning and Evaluation

Individual behavior 

and health are now understood to be

influenced by the social and physical environment of

individuals.

This recognition is reflected in the growing use of the

ecological approach 

to health services and public health

programs.

The 

ecological approach

, which stems from systems

theory  applied to individuals and families, postulates that

individuals can be influenced by factors in their

immediate social and physical environment.

77

04/04/1446

Dr. Mohammed Alnaif

The Public Health pyramid as an

Ecological Model



The perspective has been expanded into the 

social

determinants perspective

 in public health, which has

wide acceptance.



The individual is viewed as  a member of an 

intimate

social network

, usually a family, which is a member of

a larger social network, such as a neighborhood or

community.



The way in which individuals are nested within these

social networks 

has consequences for the health of the

individual..

78

04/04/1446

Dr. Mohammed Alnaif

The Public Health pyramid as an

Ecological Model

The 

Social - Ecological Model 

as a Framework for

Prevention



The social - ecological model considers four

levels of influence when describing health,

identifying public health issues, and designing

interventions.



These four levels are the individual level, the

relationship level, the community level, and the

societal level. These four spheres of influence

overlap and interact, as depicted in 

Figure 2.2

.

79

The Public Health pyramid as an

Ecological Model

04/04/1446

Dr. Mohammed Alnaif

FIGURE 2.2 Levels of Influence

80

Society

Community

Interpersonal

Individual

04/04/1446

Dr. Mohammed Alnaif

Four Levels of Influence

Individual Level



Our health is largely determined by personal

factors such as our genetic predisposition,

behavior, attitude toward health, motivation,

beliefs, and family history.



Prevention strategies at the individual level

might include mentoring and education to

positively change personal influences on health

and illness

.

81

The Public Health pyramid as an

Ecological Model

04/04/1446

Dr. Mohammed Alnaif

Four Levels of Influence

Relationship Level (Interpersonal)



Our health is also greatly influenced by relationships with

peers, partners, and family members.



Prevention strategies at the 

relationship level 

often include

education and peer or family programs to promote

relationships that support a positive health outcome.



For example, a program designed at the relationship level

might focus on diet and lifestyle education for both those

with diabetes and their families, recognizing that close family

members can provide moral and practical support in making

the important diet and exercise changes that are critical for

the appropriate management of diabetes and the prevention

of serious complications

.

82

The Public Health pyramid as an

Ecological Model

04/04/1446

Dr. Mohammed Alnaif

Four Levels of Influence

Community Level



Our health is influenced by our experience in our social

environment, such as our neighborhood, school, and

place of work.



Prevention strategies at the community level often seek

to change policy and the system as a whole through

means such as awareness campaigns or local programs.



For example, a program to promote physical activity

might aim to make neighborhoods more pedestrian

friendly by establishing well - lighted and convenient

walking paths and by educating the public about the

importance of building exercise into the daily routine

.

83

The Public Health pyramid as an

Ecological Model

04/04/1446

Dr. Mohammed Alnaif

Four Levels of Influence

Societal Level



Finally, our health is shaped by macro - level

factors in society as a whole, such as religious and

cultural beliefs, economic policies, gender or

racial inequalities, and social norms.



Prevention strategies at the societal level may

employ many approaches in combination, such as

creating new policies, awareness campaigns, and

programs, and they are often carried out by

multiple tiers of government and private or

nonprofit entities.

84

The Public Health pyramid as an

Ecological Model

04/04/1446

Dr. Mohammed Alnaif

Four Levels of Influence



The 

social - ecological model 

can be used to inform

theory, design research studies, create community

programs, develop policy, and evaluate existing

interventions.



Several versions of the model exist, some of which

include an additional level of influence called the

institutional 

or 

organizational 

level.



This level of influence fits between the relationship

and community levels and allows research and

interventions to focus more directly on the dynamic

within organizations such as schools, mosques, or the

workplace.

85

The Public Health pyramid as an

Ecological Model

04/04/1446

Dr. Mohammed Alnaif

Four Levels of Influence



The 

individual

 is viewed as a member of an intimate

social network, usually a family, which is a member of a

larger social network, such as a neighborhood or a

community.



The way in which 

individuals

 are nested within these

social networks has consequences for the health of the

individual.



Because it distinguishes and recognizes the importance

of 

enabling

 and 

population services

, the public health

pyramid can be integrated with an ecological view of

health and health problems.

86

The Public Health pyramid as an

Ecological Model

04/04/1446

Dr. Mohammed Alnaif

Four Levels of Influence



If one were to look down on the 

pyramid

 from above, the

levels would appear as concentric circles 

(FIGURE 2-2)

direct services for 

individuals

 nested within enabling services

for 

families

, 

aggregates

, and neighborhoods, which are in

turn nested within 

population

 services for all residents of

cities or countries.



This is similar to individuals being nested within the

enabling environment of their family, workplace setting, on

neighborhood, all of which are nested within the population

environment of factors such as social norms and economic

and political environments.

87

The Public Health pyramid as an

Ecological Model

04/04/1446

Dr. Mohammed Alnaif

Four Levels of Influence



The infrastructure of the healthcare system

and public health system is foundation and

supporting environment for promoting health

a d preventing illnesses and diseases.



This feature reinforces the message that each

level of the pyramid has value and importance

to health program planning and evaluation.



In addition, certain unique challenges are

specific to each level of the pyramid.

88

The Public Health pyramid as an

Ecological Model

04/04/1446

Dr. Mohammed Alnaif

Three Levels of Prevention



The underlying goal of all public health efforts is

prevention

. Preventing disease before it begins reduces

unnecessary suffering and makes the best use of health

care resources.



Prevention works on multiple levels and in some cases

may even apply to certain subsets of the population who

already have disease.



There are three levels at which 

prevention

 efforts can be

focused, known as the 

primary

, 

secondary

, and 

tertiary

levels.

89

MODERN PUBLIC HEALT H SYSTEMS

04/04/1446

Dr. Mohammed Alnaif

Three Levels of Prevention



At the 

primary prevention level

, we seek to prevent disease

before it begins. Activities at this level include health

promotion and education, as well as provision of primary

health care services.



At the 

secondary prevention level

, our goal is to detect disease

while it is still in its early stages and reduce its progression

and effects. This level includes screenings and case finding,

followed up by early intervention and control of risk factors.



At the 

tertiary prevention level

, advanced disease is already

present, so we seek to reduce complications and mortality. At

this level, efforts are focused on disease management and

continuing care

.

90

MODERN PUBLIC HEALT H SYSTEMS

04/04/1446

Dr. Mohammed Alnaif

Three Levels of Prevention



The 

primary level of prevention 

is the most

effective for many public health issues, and

ideally, the majority of resources should be

focused at this level.



However, people who are already ill can still

benefit from prevention efforts at the secondary

and tertiary level, as outlined in Table 2.1 Below

are two examples of how activities or

interventions can target each level of prevention

for a public health problem

.

91

MODERN PUBLIC HEALT H SYSTEMS

04/04/1446

Dr. Mohammed Alnaif

Table 2.1 the Three Levels of Prevention

92

04/04/1446

Dr. Mohammed Alnaif

Three Levels of Prevention



Motor vehicle safety 

is another example of a public

health issue for which we can intervene at multiple

prevention levels.



Primary prevention 

of motor vehicle crashes includes

improving road safety, for example by adding medians

or barriers to prevent motorists from crossing into

oncoming traffic or running off the road. In addition,

installing traffic signs and lights are examples of

primary prevention interventions because their purpose

is to prevent crashes from occurring

.

93

MODERN PUBLIC HEALT H SYSTEMS

04/04/1446

Dr. Mohammed Alnaif

Three Levels of Prevention



Secondary prevention 

includes efforts to reduce injury

or severity when a crash occurs. To accomplish this, we

can install and require the use of seat belts in cars as

well as car seats or booster seats for children. We also

can establish and enforce speed limits to minimize

damage done in a crash.



Finally, 

tertiary prevention 

in motor vehicle safety

involves minimizing disability or injury caused by a

crash. This may be accomplished by ensuring that an

adequate and responsive Emergency Medical Services

(EMS) system exists and that there is access to trauma

centers for rapid medical care of crash survivors

.

94

MODERN PUBLIC HEALT H SYSTEMS

04/04/1446

Dr. Mohammed Alnaif

Three Levels of Prevention



For virtually any public health problem, there are

strategies to prevent an outcome from occurring

(

primary prevention

),



to limit the negative impact of an event (

secondary

prevention

), and



to reduce long - term disability or morbidity associated

with the event (

tertiary prevention

).



It is important to recognize and identify action steps at

all levels of prevention.

95

MODERN PUBLIC HEALT H SYSTEMS

04/04/1446

Dr. Mohammed Alnaif

THANK YOU

THANK YOU

04/04/1446

Dr. Mohammed Alnaif

96