Health Program Development and Evaluation Overview
This course covers program development, planning, and evaluation in the context of public health systems. Topics include program objectives, outcomes, target populations, intervention mechanisms, and criteria for participation. The course emphasizes the importance of conceptual frameworks and project planning steps in achieving community goals. Through various stakeholders and components within the public health system, organizations aim to make positive changes for individuals and groups through structured programs.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
King Saud University College of Business Administration Department of Health Administration Masters` Program PA 519 Health Programs Planning and Evaluation Second Semester 1439/ 1440 Mohammed S. Alnaif, Ph D. alnaif@ksu.edu.sa 1 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Learning Objectives Describe Program development, program planning and program evaluation. Understand the cycle of program planning and program evaluation. Introduces types of program evaluation. Use of the public health pyramid in program planning and evaluation. 2 04/04/1446 Dr. Mohammed Alnaif
The Public Health System (Stakeholders) Police Home Health Corrections Community Centers Mosques EMS MCOs Health Department Parks Schools Elected Officials Hospitals Mass Transit Doctors Nursing Homes Philanthropist Environmental Health Civic Groups CHCs Fire Tribal Health Economic Development Laboratory Facilities Employers Drug Treatment Mental Health Source: Centers for Disease Control and Prevention 04/04/1446 Dr. Mohammed Alnaif 3
HEALT H PROGRAM DEVLOPMENT and EVALUATION All Programs/Projects have (implicitly or explicitly): Objectives Expected outcomes Target population Mechanism(s) to deliver services (the intervention) Criteria for participating in the program A conceptual framework that provides rationale for program existence (sometimes called the Development Hypothesis). 4 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION All Programs/Projects have (implicitly or explicitly): Project planning involves a series of steps that determine how to achieve a particular community or organizational goal or set of related goals. This goal can be identified in a community plan or a strategic plan. Project plans can also be based on community goals. A Program is a group of clear, related, complementary activities that are intended to achieve a desired outcome among the target group(s). 5 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION All Programs/Projects have (implicitly or explicitly): When organizations set out to make the world a better place, they develop programs to effect changes for individuals or groups. Programs vary, but the basic components of programs are essentially the same. All programs have objectives and a set of activities carried out by staff, volunteers and participants that are intended to achieve those objectives. Programs operate within an organization and often within a broader system of similar programs. 6 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION All Programs/Projects have (implicitly or explicitly): Program design is the process that an organization uses to develop a program. It is most often an iterative process involving research, consultation, initial design, testing and redesign. A program design is the plan of action that results from that process. 7 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Five reasons to do program design and development You want to make sure that: There is really a need for the program; The design you are considering actually meets the needs of the intended participants; The program you are considering has been shown to be effective; The program has been appropriately adapted to fit your context; and The program is actually working for the participants. 1. 2. 3. 4. 5. 8 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Stakeholder consultation Programs can have many stakeholders with many different points of view, depending on whether they are staff, participants, funders or other community service providers. Balancing the values and principles of a program with the practicalities of implementation often means compromise. But what compromises are likely to undermine the core values of the program? Stakeholder consultation can help to determine what is important, particularly to those implementing and those using the program. It can also contribute to adapting a model. 9 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION What a program design includes: Need for the program assesses what a population needs, what is available in community and the gap between what is needed and what is available (e.g. an organization serving youth notes that many of its participants need mental health supports, but need first to determine whether there is a broader need and whether there are existing programs). Program objectives and outcomes what the program is intended to accomplish for the particular target population. 10 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION What a program design includes: Program principles these are the fundamental norms, rules and values that guide the program (e.g. being inclusive, client-focused). Program components a specific set of inter-connected activities. provided. Budget/cost what it will cost to implement the program. Funding strategy how the program will be funded. Can it be done within the current organizational budget? Usually not, so organizations look to funders like the municipality, province, federal government. The funding strategy needs to include start-up and long-term funding. Governance some programs are set within an existing organization or some programs may be collaboration among two or more organizations. The design needs to include the process for decision-making (e.g. an organization may have a board program committee that has responsibility for design of new programs as well as redesign of existing programs). 11 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION 12 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program 1. Identify a potential need - This is not evidence of the need for a program; it is only evidence of the needs of people within the community or a population. This may create an opportunity for new program which will usually require additional resources or. 2. Conduct a needs assessment in order to determine the needs of a specific target population gather and analyze data taken from different sources in a systematic way, looking at the needs of the population and the existing programs in order to determine whether there are gaps. 13 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program 3. Research potential programs for filling need deciding on how best to address an identified need requires searching for and reviewing existing programs. Contextual issues related to the demographic population, location and magnitude of the need are considered before selecting an appropriate program. 4. Select an evidenced-based design after a thorough review of potential programs, select one that shows strong or promising evidence of being effective. This means the design has been previously implemented and evaluated and has demonstrated some ability to produce a desired outcome for the target group. If such a program does not exist or is not suitable for the context, components of a program may be selected and adapted or a new program may be designed. If the program is new or innovative, that should be noted with reasons given for trying something new. 14 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program 5. Develop the program design using a logic model (or theory of change) as a tool, create a detailed plan that specifies program activities, resource allocations and expected outcomes. The theory of change specifies the underlying assumptions of the program. Program design consolidates all of the information gathered during the development process and sets out the program s outcomes, outputs, activities and resources. 15 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program 6. Pilot the program before implementing a full-scale program, it is important to test whether it works in your particular context, if possible. Piloting includes developing the program on a small scale so you can gauge the level of readiness to deliver the program, determine whether it is making a difference for program users, and assess what, if any, changes are needed. A good pilot sets the stage for ongoing monitoring. 16 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program 7. Roll out the program if the pilot indicates the program works, you need to select a site, hire and train program staff, establish administrative processes, start service delivery, and carry out ongoing monitoring and quality control. 8. Monitor and evaluate while testing or implementing the program, it is important to learn what s working well and what isn t. It is also helpful to review the program s original goals. 17 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Steps for designing and developing a program 8. Monitor and evaluate By using a systematic process that gathers and analyzes data from a variety of sources, steps can be taken to modify and improve the program before rolling it out or as it gets implemented. This also supports accountability to funders. Consult with stakeholders speaking with and gathering information from all people involved with the program generates buy-in and surfaces diverse perspectives during the planning process. Stakeholder consultation must be genuine and responsive to be effective, and should take place throughout the planning process. 18 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Program planning and planning for a program evaluation Recently planners have been attempting to foster connections with public health professionals and improve health through planning actions. As issues such as obesity, asthma, venomous pollutants, and mental health concerns have grown in prominence, there is increasing interest in the potential for changes in the built environment to alleviate such problems. 19 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Program planning and planning for a program evaluation We need to distinguish between program planning and planning for a program evaluation, which can take place at any program stage of development. Health Planning is the identification and elaboration (within existing resources) of means and methods for providing in the future, effective health care relevant to identified health needs for a defined population. 20 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Program planning and planning for a program evaluation Aim of health planning is to maintain and improve health status of a given community. It is achieved through provision of health services which are accessible, effective, and equitable and of a quality to ensure their appropriate utilization. A plan of operations is the (written) summary of the implementation plan specifying sequence and timing of, and responsibilities for, implementation of main objectives, (expected) results and activities. 21 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Rationale for Health Planning Planning of delivery of effective health services to the population with in resources provided Translation of new policy statement into operational plan Re-planning on the basis of an already existing plan for the purpose of reviewing existing health problems and needs and rendering services more effective and efficient Emergence of a new problem (e.g. SARS, natural disaster) Planning of health services for a population where no organized health care delivery system as yet, or where an existing one is being extremely revised or re-organized. 22 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Interdependent and Cyclic Nature of Planning and Evaluation The current thinking about programs and evaluation is that the activities constituting program planning and program evaluation are cyclical and interdependent (FIGURE 1-1) The stages are cyclical to the extent that the end of one program or stage is flows almost seamlessly into the next program or planning activity Interdependence of activities and stages ideally result from information and data feedback loops that connect the stages 23 04/04/1446 Dr. Mohammed Alnaif
(FIGURE 1-1) HEALT H PROGRAM DEVLOPMENT and EVALUATION Priority established Health Program Planning Statement of health problems Assessment of Community needs and Assets Health status changes Program and Evaluation Planning Findings from the evaluation Evaluation design and methodology Effects Evaluation Process theory and effect theory delineated Findings from the evaluation Participant/Recipient Health Outcome and Impact Process Evaluation Program Intervention effect Implementation of Process and Effect Theory 24 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Interdependent and Cyclic Nature of Planning and Evaluation The cycle begin with a trigger event, such as awareness of health problems; a periodic strategic planning effort; a process required by a stakeholder, such as a grant renewal or a newly available fund for a health program. Failure of a health program The trigger might also be a news media expose or a legal action. 25 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Interdependent and Cyclic Nature of Planning and Evaluation The trigger event or situation leads to the collection of data about the problem, the characteristics of the people affected, and their perception of the health problem. Based on the data from the needs assessment; Program development begin, Problems and their solutions are prioritized. 26 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Major Components of the Planning Phase Developing the program theory; Assessment for organizational and infrastructure resources for implementing the program; Setting goals and objectives that are derived from the program theory; Implementation of the program; Evaluation plan and cycle. 27 04/04/1446 Dr. Mohammed Alnaif
Evidence and Practice Based Planning Framework 1 Define Issues 2 7 Situational Assessment Evaluation 3 6 Discussion & Recommendation Implementation 5 4 Allocate Resources Approve Decision 04/04/1446 Dr. Mohammed Alnaif 28
HEALT H PROGRAM DEVLOPMENT and EVALUATION Major Components of the Planning Phase Developing the program theory; Assessment for organizational and infrastructure resources for implementing the program; Setting goals and objectives that are derived from the program theory; Implementation of the program; Evaluation plan and cycle. 29 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Program life cycle A program may begin as a pilot; The program may not rely on any existing format or theory; Model programs; Regardless of the stage in a program s life cycle then major planning and evaluation stages of community assessment and evaluation are carried out; The life cycle of a health program is reflected in the evolution of hospice care. 30 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION The Fuzzy Aspects of planning Several Paradoxes pervade health planning; That planning is shaped by the same forces that created the problems that planning is supposed to correct. Prosperity in our modern world has its own associated health risks. What may be easier and more effective may be less acceptable (active protection, passive protection). Those in need ideally but rarely trigger the planning of health programs. The fact is that planning is intended to be successful. Most planning is for making changes not for creating stability. 31 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. The first is that a solution, remedy, or appropriate intervention can be identified or developed and provided. Without this assumption, planning would be pointless. It is fundamentally an optimistic assumption about the capacity of the planners, the stakeholders, and the state of the science to address the health problem. The assumption of adequate capacity and knowledge is actually tested through the process of planning. 32 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. Planning leads to the allocation of resources needed to address the health problem. This assumption is challenged by the reality that four groups of stakeholders have interests in the decision making regarding health resources and each group exists in all program planning. 1. Members of the target audience for the health program 2. Health payers, such as insurance companies and philanthropic funding agencies 3. Individual healthcare providers and healthcare organizations and networks 4. The general public are effected by how resources are allocated for health programs 33 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. Another assumption about those involved is that they share similar views on how to plan health programs. During the planning process their points of view and cultural perspectives will likely come into contrast. Planners need to know what is relevant and important for the problem at hand. There are competing purposes and values. Effective planning requires tolerance, freedom, and fairness 34 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. The resulting conflicts among the stakeholders for the limited resource apply whether they are allocating resources across the healthcare system or among programs for specific health problems. Limited resources, whether real or not, raise ethical questions of what to do when possible gains from needed health programs or policies are likely to be small, especially when the health program addresses serious health problems. 35 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Assumption, Assumptions also influence the effectiveness of planning. Another assumption about the planning process is that it occurs in an orderly fashion and that a rational approach is best. Four key elements are inherent in planning; uncertainty, ambiguity, risk, and control. The presence of each of these elements contradicts the assumption of rational approach, and each generates its own paradoxes. 36 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control The process of planning is affected by the limits of both scientific rationality and the usefulness of data to cope with the uncertainties, ambiguities, and risks being addressed by the planning process (see TABLE 1-3). Uncertainty is the unknown likelihood of a possible outcome. Four type of uncertainty are identified: types and amount of resources, technological, market receptivity to the product, and organizational. 37 04/04/1446 Dr. Mohammed Alnaif
TABLE 1-3 Fuzzy Aspect Throughout the Planning and Evaluation cycle Stages in the Planning and Evaluation Cycle Community Assessment Unknown likelihood of selecting an effective intervention, Unknown likelihood of the intervention being effective Unclear about the process, who is leading planning process, or what it is intended to accomplish Effect Evaluation Planning Implementation Unknown likelihood of the intervention being provided as designed and planned Unknown likelihood of finding key health determinants Unknown likelihood of intervention being effective Uncertainty Unclear about the boundaries of the program, who ought to participate, or who ought to deliver the program Unclear about meaning of the evaluation results Unclear about who is being assessed or why Ambiguity Unknown possibility of adverse effect from the evaluation design, or from misinterpretation of the findings Directing the process of data collection, analysis and interpretation Unknown possibility of planning touching on politically sensitive issues Unknown possibility of the intervention having an adverse effect on participants Unknown possibility of the assessment causing harm Risk Directing the process of gathering and interpreting data about the health problem Directing the decisions about the program Directing the manner in which the program is provided Control 04/04/1446 Dr. Mohammed Alnaif 38
HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control Ambiguity is doubt about a course of action stemming from awareness that known and unknown factors exist that can decrease the possibility of certainty. In this sense, ambiguity results in uncertainty Both uncertainty and ambiguity pervade the planning process because it is impossible to know and estimate the effect of all relevant factors. 39 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control Ambiguity is the characteristic of not having a clear or single meaning. Change, or the possibility of change, is a possible source of ambiguity. When ambiguity is ignored, the resulting differences in interpretation can lead to confusion and conflict among stakeholders. Openly and constructively addressing the ambiguity and any associated conflict can lead to innovations in the program. 40 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control Risk is the perceived possibility or uncertain probability of an adverse outcome in a given situation. Risk is not just about taking chances but it is also about uncertainty and ambiguity ( as in the case with estimates of cure rates and projections about future health conditions). Risk is pervasive and inherent throughout the planning process in terms of deciding who to involve and how, which planning approach to use, which intervention to use, and I estimating which health problem deserves attention. 41 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Uncertainty, Ambiguity, Risk, and Control Control, as in being in charge of managing, is a natural reaction to the presence of ambiguity, conflict, and risk. It can take the form of directing attention and allocating resources or of exerting dominance over others. Addressing the ambiguity, uncertainty, and risk that might have been the trigger for the planning process require less not more control. Those who preside over and influence the planning process are often thought of having control over solutions to the health problem or condition. 42 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Some think of evaluations as falling into one of two broad categories: formative and summative evaluations. Formative evaluations are conducted during program development and implementation and are useful if you want direction on how to best achieve your goals or improve your program. Summative evaluations should be completed once your programs are well established and will tell you to what extent the program is achieving its goals. 43 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Several major types of activities are classified as evaluations. each type of activity requires a specific focus, purpose, and set of skills. Which of these evaluations is most appropriate depends on the stage of your program: A community needs assessment Process evaluations Effect evaluations Outcome evaluations Impact evaluations Cost evaluations Comprehensive evaluations Meta-evaluations 44 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Unfortunately, contrasting summative evaluations with formative evaluations as the two broad categories adds more confusion to the evaluation terminology. Summative evaluations, in the strictest sense, are done at the conclusion of a program to provide a conclusive statement regarding program effects. The term summative evaluation is sometimes used to refer to either an outcome or impact evaluation. The term formative evaluation is used to refer to program assessments that are performed early in the implementation of the program and used to make changes to the program. 45 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Community needs assessment (also known as community health assessment) is a type of evaluation that is performed to collect data about the health problems of a particular group. Needs Assessment, determines who needs the program, how great the need is, and what can be done to best meet the need. 46 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation The data collected for this purpose are then used to tailor the health program to the need and distinctive characteristics of that group. A community needs assessment is a major component of program planning because it is , done at an early stage in the program planning and evaluation cycle. 47 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Process evaluations, which begins at the same time that the program starts, and focus on the degree to which the program has been implanted as planned and on the quality of the program implementation. Process evaluations are known by a variety of terms, such as monitoring evaluations, depending on their focus and characteristics. 48 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation The underlining framework for designing a process evaluation comes from the process theory component of the overall program theory developed during the planning stage. The process theory delineates the logistical activities, resources, and interventions needed to achieve the health change in program participants or recipients. Information from process evaluation is used to plan, revise, or improve the program. 49 04/04/1446 Dr. Mohammed Alnaif
HEALT H PROGRAM DEVLOPMENT and EVALUATION Introduction to the Types of Evaluation Effect evaluations answer a key question: did the program make a difference? Evaluators seek to use the most rigorous and robust designs, methods, and statistics possible and feasible when conducting an effect evaluation. Findings from effect evaluations are used to revise the program and may be used in subsequent initial program planning activities. 50 04/04/1446 Dr. Mohammed Alnaif
