Policy Advocacy Monitoring and Evaluation
Monitor and Evaluate Policy
Advocacy
Step 8: Develop Monitoring and Evaluation Plan
•
Purpose: demonstrate progress toward your
goals and objectives and define success for
advocacy strategy
•
Consider the factors that make M&E of
advocacy different
•
Decide on methods for collecting data and
measuring success
•
Develop indicators for your advocacy strategy
M
o
n
i
t
o
r
i
n
g
a
n
d
E
v
a
l
u
a
t
i
o
n
B
a
s
i
c
s
C
o
m
m
o
n
M
&
E
T
e
r
m
s
•
Goals
•
Objectives
•
Indicators
•
Targets
•
Data/collection
methods
•
Logic model
•
Inputs
•
Activities
•
Outputs
•
Outcomes
•
Impact
•
Performance
monitoring plan
(PMP)
“
M
”
i
n
M
&
E
•
Monitoring is the systematic process of
collecting, analyzing, and using information/
data on specified project indicators to:
–
Track progress of project implementation
–
Track progress to reaching objectives
–
Account for the use of allocated funds
–
Guide management decisions
•
The information is used by project staff,
management, and stakeholders.
“
E
”
i
n
M
&
E
•
Evaluation is the systematic collection and analysis of
information about the characteristics and outcomes
of programs to:
–
Make judgments, improve effectiveness, and inform
decisions about current and future programming
–
Compare expected and achieved results
–
Examine contextual factors and causality to
understand achievements or lack of achievement
•
Involves measuring changes in knowledge, attitudes,
behaviors, skills, community norms, and use of
services at the program or population level
B
a
s
i
c
L
o
g
i
c
M
o
d
e
l
S
t
r
u
c
t
u
r
e
L
o
g
i
c
M
o
d
e
l
E
x
a
m
p
l
e
—
A
W
e
d
d
i
n
g
Inputs
Couple
Budget
Venue
Family and
friends
Vendors
Activities
Hire
Vendors
Write vows
Get married
Celebrate
with friends
and family
Go on a
honeymoon
Outputs
600 photos
taken
1 marriage
certificate
signed
200 guests
fed
2 vows
exchanged
75 gifts
received
Outcomes
Couple:
•
Is married
•
Enjoyed
time with
family
and
friends
•
Had fun
on their
special
day
Impact
Stronger
family
bonds for
couple
Stronger
family
bonds for
the families
I
n
d
i
c
a
t
o
r
s
•
M&E requires the use of
indicators.
•
Indicators define the data
that you are collecting.
E
x
a
m
p
l
e
:
n
u
m
b
e
r
o
f
p
e
o
p
l
e
s
e
n
s
i
t
i
z
e
d
o
n
G
B
V
a
n
d
G
B
V
s
e
r
v
i
c
e
s
•
An indicator is a variable (its value changes) that
measures the components of a program or project
(inputs, activities, outputs, outcomes ).
•
Indicators can be quantitative or qualitative.
C
h
a
r
a
c
t
e
r
i
s
t
i
c
s
o
f
G
o
o
d
I
n
d
i
c
a
t
o
r
s
•
Valid:
accurate measure of a behavior,
practice, or task that is the expected output
or outcome of the activity
•
Precise:
operationally defined in clear terms
•
Measurable:
quantifiable using available
tools and methods
•
Reliable:
consistently measurable in the
same way by different observers
C
h
a
r
a
c
t
e
r
i
s
t
i
c
s
o
f
G
o
o
d
I
n
d
i
c
a
t
o
r
s
(
c
o
n
t
i
n
u
e
d
)
•
Comparable:
can be measured in different
contexts or time periods
•
Time-bound:
requires completion by a
specific date
•
Programmatically important:
Why do we
want to know this? Is the information useful?
M
o
n
i
t
o
r
i
n
g
&
E
v
a
l
u
a
t
i
o
n
f
o
r
A
d
v
o
c
a
c
y
W
o
r
k
Routine tracking of inputs and
outputs
Measuring the changes resulting from
program activities over time
Evaluation
Is the advocacy strategy being
executed according to plan and on
budget?
Are the activities still appropriate
given external changes?
Is the advocacy having the intended
effect on target audiences?
What are the successes and failures?
What is happening that you didn’t
expect?
Source: Building on the Easy Wins: A Framework for Planning and Evaluating Proactive Long-Term Policy Advocacy. Health Policy Project; 2016.
Monitoring
T
y
p
e
s
o
f
I
n
d
i
c
a
t
o
r
s
f
o
r
M
&
E
o
f
A
d
v
o
c
a
c
y
•
Process/output indicators:
numbers/types of activities and
quality determinants—what you will do and how you will do it
E
x
a
m
p
l
e
:
X
d
i
f
f
e
r
e
n
t
o
r
g
a
n
i
z
a
t
i
o
n
s
c
o
n
t
a
c
t
e
d
t
o
p
a
r
t
i
c
i
p
a
t
e
i
n
t
h
e
c
o
a
l
i
t
i
o
n
u
s
i
n
g
b
e
s
t
p
r
a
c
t
i
c
e
s
i
n
e
n
g
a
g
e
m
e
n
t
.
•
Interim outcome indicators:
how you will know that attitudes,
knowledge, and behavior of target audiences and allies have
changed in the short and medium terms
E
x
a
m
p
l
e
:
X
n
e
w
p
a
r
t
n
e
r
s
h
a
v
e
s
h
a
r
e
d
c
o
a
l
i
t
i
o
n
-
a
p
p
r
o
v
e
d
m
e
s
s
a
g
e
s
w
i
t
h
t
a
r
g
e
t
s
.
•
Long-term outcome indicators:
the policy change for which
you have advocated
E
x
a
m
p
l
e
:
T
h
e
l
o
c
a
l
c
o
u
n
c
i
l
a
p
p
r
o
v
e
d
p
o
l
i
c
y
c
h
a
n
g
e
X
w
i
t
h
8
5
p
e
r
c
e
n
t
o
f
o
u
r
p
r
e
f
e
r
r
e
d
p
o
l
i
c
y
e
l
e
m
e
n
t
s
.
Source: Building on the Easy Wins: A Framework for Planning and Evaluating Proactive Long-Term Policy Advocacy, Health Policy Project, 2016.
L
o
g
i
c
M
o
d
e
l
w
i
t
h
S
a
m
p
l
e
I
n
d
i
c
a
t
o
r
s
Inputs
Activities
Outputs
Outcomes
Impact
Indicators:
# of partners
contacted
# of partners
engaged in
coalition
# of new partners
sharing messages
of the coalition
# of PLHIV
accessing
services
D
i
s
t
i
n
c
t
i
v
e
F
e
a
t
u
r
e
s
o
f
A
d
v
o
c
a
c
y
M
&
E
•
Time frames can be unpredictable
•
Strategies and milestones shift
•
Assessing progress and interim outcomes is
important; not just impact
•
Context may define progress and outcomes
•
Demonstrate contribution to results, not
simply attribution
Source: Advocacy Toolkit, UNICEF, 2010.
M
e
t
h
o
d
s
t
o
E
v
a
l
u
a
t
e
A
d
v
o
c
a
c
y
Source: Building on the Easy Wins: A Framework for Planning and Evaluating Proactive Long-Term Policy Advocacy. Health Policy Project; 2016.
M
e
t
h
o
d
:
P
o
l
i
c
y
m
a
k
e
r
R
a
t
i
n
g
s
Support
1=Not at all supportive
2=Interested
3=Somewhat supportive
4=Supportive
5=Extremely supportive
1=Not very influential
2=Somewhat influential
3=Influential
4=Extremely influential
1=Not very confident in
the rating
2=Somewhat confident
3=Extremely confident
NOTE: You must identify
observable characteristics or behaviors
that differentiate
each “level” on the scale for consistency between raters and ratings.
Influence
Confidence
Developed by: Julia Coffman, Harvard Family Research Project.
M
e
t
h
o
d
:
S
y
s
t
e
m
s
M
a
p
p
i
n
g
•
Useful when trying to track
systems change
•
Mapping of the parts and
relationships of a system
•
Can involve mapping of a
system, key informant
interviews, or network
analysis
Adapted from: Measuring Up, A Guide for Learners. AIDS Alliance; 2010.
E
x
a
m
p
l
e
1
Problem Statement:
PLHIV in Malawi denied access to care, treatment, and support
Advocacy Goal:
Equal access to high-quality health services for all PLHIV in Malawi
Advocacy Objective:
MOH commits to adopting a policy of providing high-quality health
services to all PLHIV
I
n
d
i
c
a
t
o
r
:
M
O
H
r
e
s
o
l
u
t
i
o
n
p
u
b
l
i
s
h
e
d
b
y
M
a
r
c
h
2
0
2
0
o
n
p
r
o
v
i
s
i
o
n
o
f
h
i
g
h
-
q
u
a
l
i
t
y
s
e
r
v
i
c
e
s
f
o
r
a
l
l
P
L
H
I
V
Activities
1:
Analyze data on current quality of health care services for PLHIV
2:
Develop briefing paper on current status of care and potential impact of changed
policy
I
n
d
i
c
a
t
o
r
:
B
r
i
e
f
i
n
g
p
a
p
e
r
w
r
i
t
t
e
n
o
n
t
h
e
s
t
a
t
u
s
a
n
d
q
u
a
l
i
t
y
o
f
h
e
a
l
t
h
c
a
r
e
s
e
r
v
i
c
e
s
f
o
r
P
L
H
I
V
b
a
s
e
d
o
n
c
u
r
r
e
n
t
d
a
t
a
,
a
n
d
p
o
t
e
n
t
i
a
l
i
m
p
a
c
t
o
f
c
h
a
n
g
e
d
p
o
l
i
c
y
3:
Present data and briefing paper to MOH technical working group at quarterly meeting
I
n
d
i
c
a
t
o
r
:
P
r
e
s
e
n
t
a
t
i
o
n
d
e
v
e
l
o
p
e
d
a
n
d
d
e
l
i
v
e
r
e
d
t
o
M
O
H
d
u
r
i
n
g
q
u
a
r
t
e
r
l
y
m
e
e
t
i
n
g
Adapted from: A2 Analysis and Advocacy, Advocacy Training Manual, Health Policy Initiative, 2007.
E
x
a
m
p
l
e
2
Problem Statement:
Lack of government commitment (policy and resources) to the
national HIV and AIDS program, which has limited HIV services for the general
population
Advocacy Goal:
Adoption of national HIV and AIDS policy and allocation of appropriate
resources for its implementation
Advocacy Objective:
National AIDS Council (NAC) will draft and submit a National HIV
and AIDS Policy and Operational Plan for approval within one year
I
n
d
i
c
a
t
o
r
:
N
a
t
i
o
n
a
l
H
I
V
a
n
d
A
I
D
S
P
o
l
i
c
y
a
n
d
O
p
e
r
a
t
i
o
n
a
l
P
l
a
n
s
u
b
m
i
t
t
e
d
t
o
M
O
H
f
o
r
a
p
p
r
o
v
a
l
Activities
1:
Conduct stakeholder meeting to gather input about activities to include in the plan
I
n
d
i
c
a
t
o
r
:
D
r
a
f
t
l
i
s
t
o
f
a
c
t
i
v
i
t
i
e
s
d
e
v
e
l
o
p
e
d
a
s
a
r
e
s
u
l
t
o
f
s
t
a
k
e
h
o
l
d
e
r
m
e
e
t
i
n
g
2:
With research partners, review plans of other countries and develop a briefing note
on important considerations for the NAC
3:
Develop a website to track commitments by the NAC of what will be included in the
plan and progress toward completion
I
n
d
i
c
a
t
o
r
:
T
r
a
c
k
i
n
g
w
e
b
s
i
t
e
d
e
v
e
l
o
p
e
d
w
i
t
h
i
n
i
t
i
a
l
c
o
m
m
i
t
m
e
n
t
s
o
f
t
h
e
N
A
C
Adapted from: A2 Analysis and Advocacy, Advocacy Training Manual. Health Policy Initiative; 2007.
M
e
a
s
u
r
e
m
e
n
t
I
n
d
i
c
a
t
o
r
s
H
a
n
d
o
u
t
A
c
t
i
v
i
t
y
—
D
r
a
f
t
i
n
g
I
n
d
i
c
a
t
o
r
s
•
Develop indicators for your strategies
(small groups,
30 minutes):
–
1 indicator for the objective
–
1 indicator for at least 2 activities
Consult your handout for guidance to ensure that
indicators are suitable.
•
Identify the data source and individual(s)
responsible for collecting data.
•
Provide justification for selecting each indicator
(related to characteristics of good indicators).
•
Share indicators with the large group.
L
e
a
r
n
i
n
g
O
b
j
e
c
t
i
v
e
s
—
S
e
s
s
i
o
n
1
2
•
Define methods used to track success and
evaluate advocacy
•
Use logic models as a tool to show progress
toward advocacy goals and objectives
•
Describe methods for evaluating advocacy
•
Develop indicators for your advocacy strategy
Develop a monitoring and evaluation plan to track progress, measure success, and guide decision-making for policy advocacy efforts. Monitoring involves collecting and analyzing project data to track implementation and progress towards objectives, while evaluation focuses on assessing program outcomes and effectiveness to inform future strategies. Utilize logic models and examples to structure your advocacy strategies and outcomes effectively.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Monitor and Evaluate Policy Advocacy
Step 8: Develop Monitoring and Evaluation Plan Purpose: demonstrate progress toward your goals and objectives and define success for advocacy strategy Consider the factors that make M&E of advocacy different Decide on methods for collecting data and measuring success Develop indicators for your advocacy strategy
M in M&E M in M&E Monitoring is the systematic process of collecting, analyzing, and using information/ data on specified project indicators to: Track progress of project implementation Track progress to reaching objectives Account for the use of allocated funds Guide management decisions The information is used by project staff, management, and stakeholders.
E in M&E E in M&E Evaluation is the systematic collection and analysis of information about the characteristics and outcomes of programs to: Make judgments, improve effectiveness, and inform decisions about current and future programming Compare expected and achieved results Examine contextual factors and causality to understand achievements or lack of achievement Involves measuring changes in knowledge, attitudes, behaviors, skills, community norms, and use of services at the program or population level
Basic Logic Model Structure Basic Logic Model Structure Resources/ Inputs Activities Outputs Outcomes Impact 1 2 3 4 5 Your Planned Work Your Intended Results
Logic Model Example Logic Model Example A Wedding A Wedding Inputs Activities Outputs Outcomes Impact Stronger family bonds for couple 600 photos taken Hire Vendors Couple: Is married Enjoyed time with family and friends Had fun on their special day Couple 1 marriage certificate signed Budget Write vows Stronger family bonds for the families Venue Get married 200 guests fed Family and friends Celebrate with friends and family 2 vows exchanged Vendors 75 gifts received Go on a honeymoon
Indicators Indicators M&E requires the use of indicators. Indicators define the data that you are collecting. Example: Example: number of people sensitized on GBV and GBV services number of people sensitized on GBV and GBV services An indicator is a variable (its value changes) that measures the components of a program or project (inputs, activities, outputs, outcomes ). Indicators can be quantitative or qualitative.
Characteristics of Good Indicators Characteristics of Good Indicators Valid: accurate measure of a behavior, practice, or task that is the expected output or outcome of the activity Precise: operationally defined in clear terms Measurable: quantifiable using available tools and methods Reliable: consistently measurable in the same way by different observers
Characteristics of Good Indicators Characteristics of Good Indicators (continued) (continued) Comparable: can be measured in different contexts or time periods Time-bound: requires completion by a specific date Programmatically important: Why do we want to know this? Is the information useful?
Monitoring & Evaluation for Advocacy Work Monitoring & Evaluation for Advocacy Work Monitoring Evaluation Measuring the changes resulting from program activities over time Routine tracking of inputs and outputs Is the advocacy strategy being executed according to plan and on budget? Is the advocacy having the intended effect on target audiences? What are the successes and failures? Are the activities still appropriate given external changes? What is happening that you didn t expect? Source: Building on the Easy Wins: A Framework for Planning and Evaluating Proactive Long-Term Policy Advocacy. Health Policy Project; 2016.
Types of Indicators for M&E of Advocacy Types of Indicators for M&E of Advocacy Process/output indicators: numbers/types of activities and quality determinants what you will do and how you will do it Example: Example: X different organizations contacted to participate in the coalition X different organizations contacted to participate in the coalition using best practices in engagement. using best practices in engagement. Interim outcome indicators: how you will know that attitudes, knowledge, and behavior of target audiences and allies have changed in the short and medium terms Example: Example: X new partners have shared coalition X new partners have shared coalition- -approved messages with targets with targets. Long-term outcome indicators: the policy change for which you have advocated Example: Example: The local council approved policy change X with 85 percent of The local council approved policy change X with 85 percent of our preferred policy elements. our preferred policy elements. approved messages Source: Building on the Easy Wins: A Framework for Planning and Evaluating Proactive Long-Term Policy Advocacy, Health Policy Project, 2016.
Logic Model with Sample Indicators Logic Model with Sample Indicators Impact Outputs Outcomes Inputs Activities - New partners actively sharing coalition messages - Policy review - Partners engaged in coalition - Engage new partners in coalition Funding Enabling environment increases access and use of services by PLHIV - Community dialogue held; topics listed Staff - Positive feedback from dialogues - Conduct community dialogues Plans - Improved attitude of providers - Providers trained and scored on attitudes Training materials - Health provider sensitization trainings - Friendly services # of partners engaged in coalition # of new partners sharing messages of the coalition # of PLHIV accessing services # of partners contacted Indicators:
Distinctive Features of Advocacy M&E Distinctive Features of Advocacy M&E Time frames can be unpredictable Strategies and milestones shift Assessing progress and interim outcomes is important; not just impact Context may define progress and outcomes Demonstrate contribution to results, not simply attribution Source: Advocacy Toolkit, UNICEF, 2010.
Methods to Evaluate Advocacy Methods to Evaluate Advocacy Traditional Unique Bellwether methodology Champion tracking Interviews Intense period debriefs Media scorecards Surveys Critical incident timelines 360-degree critical incident reviews Focus groups Systems mapping Research panels Event observation Social media tracking Policymaker ratings Polling Intercept interviews Rapid response research Case studies Source: Building on the Easy Wins: A Framework for Planning and Evaluating Proactive Long-Term Policy Advocacy. Health Policy Project; 2016.
Method: Policymaker Ratings Method: Policymaker Ratings Support Influence Confidence 1=Not at all supportive 2=Interested 3=Somewhat supportive 4=Supportive 5=Extremely supportive 1=Not very influential 2=Somewhat influential 3=Influential 4=Extremely influential 1=Not very confident in the rating 2=Somewhat confident 3=Extremely confident NOTE: You must identify observable characteristics or behaviors that differentiate each level on the scale for consistency between raters and ratings. Developed by: Julia Coffman, Harvard Family Research Project.
Method: Systems Mapping Method: Systems Mapping Useful when trying to track systems change Step 1 Mapping of the parts and relationships of a system Step 2 Can involve mapping of a system, key informant interviews, or network analysis Step 3 Adapted from: Measuring Up, A Guide for Learners. AIDS Alliance; 2010.
Example 1 Example 1 Problem Statement: PLHIV in Malawi denied access to care, treatment, and support Advocacy Goal: Equal access to high-quality health services for all PLHIV in Malawi Advocacy Objective: MOH commits to adopting a policy of providing high-quality health services to all PLHIV Indicator: Indicator: MOH resolution published by March 2020 on provision of high MOH resolution published by March 2020 on provision of high- -quality services for all PLHIV services for all PLHIV Activities 1: Analyze data on current quality of health care services for PLHIV 2: Develop briefing paper on current status of care and potential impact of changed policy Indicator: Indicator: Briefing paper written on the status and quality of health care services Briefing paper written on the status and quality of health care services for PLHIV based on current data, and potential impact of changed policy for PLHIV based on current data, and potential impact of changed policy 3: Present data and briefing paper to MOH technical working group at quarterly meeting Indicator: Indicator: Presentation developed and delivered to MOH during quarterly meeting Presentation developed and delivered to MOH during quarterly meeting quality Adapted from: A2 Analysis and Advocacy, Advocacy Training Manual, Health Policy Initiative, 2007.
Example 2 Example 2 Problem Statement: Lack of government commitment (policy and resources) to the national HIV and AIDS program, which has limited HIV services for the general population Advocacy Goal: Adoption of national HIV and AIDS policy and allocation of appropriate resources for its implementation Advocacy Objective: National AIDS Council (NAC) will draft and submit a National HIV and AIDS Policy and Operational Plan for approval within one year Indicator: Indicator: National HIV and AIDS Policy and Operational Plan submitted to MOH for National HIV and AIDS Policy and Operational Plan submitted to MOH for approval approval Activities 1: Conduct stakeholder meeting to gather input about activities to include in the plan Indicator: Indicator: Draft list of activities developed as a result of stakeholder meeting Draft list of activities developed as a result of stakeholder meeting 2: With research partners, review plans of other countries and develop a briefing note on important considerations for the NAC 3: Develop a website to track commitments by the NAC of what will be included in the plan and progress toward completion Indicator: Tracking website developed with initial commitments of the NAC Tracking website developed with initial commitments of the NAC Adapted from: A2 Analysis and Advocacy, Advocacy Training Manual. Health Policy Initiative; 2007.
Measurement Indicators Handout Measurement Indicators Handout
Activity Activity Develop indicators for your strategies (small groups, 30 minutes): 1 indicator for the objective 1 indicator for at least 2 activities Consult your handout for guidance to ensure that indicators are suitable. Identify the data source and individual(s) responsible for collecting data. Provide justification for selecting each indicator (related to characteristics of good indicators). Share indicators with the large group. Drafting Indicators Drafting Indicators
Learning Objectives Learning Objectives Session 12 Session 12 Define methods used to track success and evaluate advocacy Use logic models as a tool to show progress toward advocacy goals and objectives Describe methods for evaluating advocacy Develop indicators for your advocacy strategy
