Contribution Analysis for Project Improvement
Contribution Analysis
Made Do-Able
Rod Laird Organisation
18 September 2007
Euston, London
Mike Hendricks, Ph.D.
Contribution Analysis
Made Do-Able
Rod Laird Organisation
24 September 2007
Manchester
Mike Hendricks, Ph.D.
What is “Contribution Analysis”?
•
Many people care about the performance of your project
•
Two questions : (1) Did things improve? and (2) Did you cause it?
•
The second question is much harder to answer
•
But audiences typically don’t care – they still want your best
answer
•
“Proof”, “conclusive evidence”, and “causal attribution” aren’t the
appropriate standard; too scientific, rigorous
•
A more appropriate, real-world standard of evidence is emerging
•
“Reasonable confidence”, “convincing evidence”, “plausible
association”
•
Amassing this standard of evidence is called contribution analysis
How Will We Learn
About Contribution Analysis Today?
* Explore these cutting-edge issues together, as colleagues
* Consider six steps, with multiple options for each:
1. Ensure that your project has been fully
implemented
2. Gather quantitative data from participants (
only
)
3. Gather quantitative data from participants
and others
4. Gather
qualitative
data about your project
5. Deliberately seek “plausible
alternative
explanations”
6.
Assess
your project’s contribution
* Discuss each step and option, with examples when possible
* You plan a contribution analysis for your own situation
* Presentations, discussions, feedback on your ideas
Step #1:
Ensure that your project has been fully implemented
An essential first step = Is your project really there?
Describe in detail how your project is
supposed
to be operating:
* Detailed narrative
* Logic model
Describe in detail how your project is
actually
operating:
* Inputs
* Activities
* Outputs
* Be able to report, in some detail, everyone’s experiences
Compare the
actual
reality to the original
plans
Essential Components of a Project
Inputs, Activities, Outputs, and Outcomes
•
An
INPUT
is often expressed as a NOUN – the name of a person, place, thing, or
concept. For example,
staff, volunteers, clients, facility, food, money, time
.
•
An
ACTIVITY
is often expressed as a GERUND – an “-ing” verb . For example,
feeding homeless people, training people for jobs, counseling pregnant women,
mentoring teenagers
.
•
An
OUTPUT
is often expressed as the VOLUME of work accomplished by the
project. For example,
the number of people fed, the number of training sessions
held, the number of persons trained, the number of women counseled, the number
of teenagers mentored
.
•
An
OUTCOME
often describes a CHANGE in persons you work with. For
example,
new knowledge, increased skills, changed attitudes, modified behaviors
.
Inputs through Outcomes:
The Conceptual Chain
Step #2:
Gather quantitative data from participants (only)
Can we find persuasive evidence that:
1.
Outcomes changed
after
your efforts
2.
Outcomes changed at the
appropriate time
after your efforts
3.
Outcomes changed a
lot
after your efforts
4.
Outcomes changed as your efforts
ebbed and flowed
5.
The
relevant
outcomes changed, but not others
6.
Statistical analyses
show that your efforts were important
7.
Outcomes changed in
different sites
where you worked
8.
Outcomes changed each time you
phased-in
your efforts
9.
Outcomes changed most where your efforts were most
intense
Step #3:
Gather quantitative data from participants and others
Can we find persuasive evidence that:
1.
Outcomes changed
after
your efforts, but not elsewhere
2.
Outcomes changed at
appropriate times
, but only after your
efforts
3.
Outcomes changed as you switched your efforts
on and off
in
different sites
4. For a
group
, outcomes departed from what was expected
5. For
individuals
, outcomes departed from what was expected
T
h
e
g
r
a
p
h
b
e
l
o
w
c
o
m
p
a
r
e
s
t
h
e
r
a
t
i
n
g
s
g
i
v
e
n
b
y
r
e
s
p
o
n
d
e
n
t
s
w
h
o
d
i
d
s
e
e
a
S
p
e
c
i
a
l
i
s
t
D
o
m
e
s
t
i
c
V
i
o
l
e
n
c
e
P
o
l
i
c
e
O
f
f
i
c
e
r
a
n
d
t
h
o
s
e
w
h
o
d
i
d
n
’
t
,
i
n
r
e
l
a
t
i
o
n
t
o
t
h
e
s
u
p
p
o
r
t
t
h
e
y
r
e
c
e
i
v
e
d
f
r
o
m
t
h
e
P
o
l
i
c
e
f
o
l
l
o
w
i
n
g
t
h
e
i
n
i
t
i
a
l
r
e
p
o
r
t
o
f
a
n
i
n
c
i
d
e
n
t
.
Step #4:
Gather qualitative data about your project
1.
Gather
rich descriptions
of your project and participants:
* In-depth journalism, detective work, medical diagnosis
* Case studies, modus operandi, most significant change,
success case methods
2.
Examine the pattern of results within your
logic model
(program theory/logical framework/theory of change/outcome
map, outcome chain, outcome sequence chart):
* Study data on inputs, activities, outputs, and outcomes
* Do the patterns support or dispute a plausible association?
Step #5:
Deliberately seek plausible alternative explanations
•
“Other reasons the outcomes may have changed”
•
May seem strange, but very important for your credibility
•
And we
do
see other possibilities fairly often
•
Four main steps:
1. Identify the most likely
possibilities
2. Seek evidence
for
the contribution of each one
3. Seek evidence
against
the contribution of each one
4. Compare and draw
conclusions
•
Ideally, design your research up-front to test other possibilities
Step #6:
Assess your project’s contribution
•
Pull together
multiple
lines of evidence
* About your project
* About plausible alternative explanations
•
Include both quantitative
and
qualitative information
•
Seek
consensus
of informed outsiders
* Jury deliberations
* NIH consensus-building panels
* Foundation allocation committees
* Others, especially in the UK context?
Contribution analysis evaluates project performance by answering key questions on improvement and causation. It emphasizes gathering qualitative and quantitative data to establish a reasonable level of evidence for project contributions. Learn about the process, essential steps, and components involved in conducting a successful contribution analysis.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Contribution Analysis Made Do-Able Rod Laird Organisation 18 September 2007 Euston, London Mike Hendricks, Ph.D.
Contribution Analysis Made Do-Able Rod Laird Organisation 24 September 2007 Manchester Mike Hendricks, Ph.D.
What is Contribution Analysis? Many people care about the performance of your project Two questions : (1) Did things improve? and (2) Did you cause it? The second question is much harder to answer But audiences typically don t care they still want your best answer Proof , conclusive evidence , and causal attribution aren t the appropriate standard; too scientific, rigorous A more appropriate, real-world standard of evidence is emerging Reasonable confidence , convincing evidence , plausible association Amassing this standard of evidence is called contribution analysis
How Will We Learn About Contribution Analysis Today? * Explore these cutting-edge issues together, as colleagues * Consider six steps, with multiple options for each: 1. Ensure that your project has been fully implemented 2. Gather quantitative data from participants (only) 3. Gather quantitative data from participants and others 4. Gather qualitative data about your project 5. Deliberately seek plausible alternativeexplanations 6. Assessyour project s contribution * Discuss each step and option, with examples when possible * You plan a contribution analysis for your own situation * Presentations, discussions, feedback on your ideas
Step #1: Ensure that your project has been fully implemented An essential first step = Is your project really there? Describe in detail how your project is supposed to be operating: * Detailed narrative * Logic model Describe in detail how your project is actually operating: * Inputs * Activities * Outputs * Be able to report, in some detail, everyone s experiences Compare the actual reality to the original plans
Essential Components of a Project OUTCOMES ACTIVITIES INPUTS OUTPUTS Resources dedicated to or consumed by the project What the project does with inputs to fulfill its mission The volume of work accomplished by the project Benefits or changes for participants during or after project activities money staff & staff time volunteers & volunteer time facilities equipment & supplies feeding and sheltering homeless families providing job training educating teachers about signs of child abuse counseling pregnant women number of classes taught number of counseling sessions conducted number of educational materials distributed number of hours of service delivered number of participants served new knowledge increased skills changed attitudes or values modified behavior improved condition altered status
Inputs, Activities, Outputs, and Outcomes An INPUT is often expressed as a NOUN the name of a person, place, thing, or concept. For example, staff, volunteers, clients, facility, food, money, time. An ACTIVITY is often expressed as a GERUND an -ing verb . For example, feeding homeless people, training people for jobs, counseling pregnant women, mentoring teenagers. An OUTPUT is often expressed as the VOLUME of work accomplished by the project. For example, the number of people fed, the number of training sessions held, the number of persons trained, the number of women counseled, the number of teenagers mentored. An OUTCOME often describes a CHANGE in persons you work with. For example, new knowledge, increased skills, changed attitudes, modified behaviors.
Inputs through Outcomes: The Conceptual Chain
Step #2: Gather quantitative data from participants (only) Can we find persuasive evidence that: 1. Outcomes changed after your efforts 2. Outcomes changed at the appropriate time after your efforts 3. Outcomes changed a lot after your efforts 4. Outcomes changed as your efforts ebbed and flowed 5. The relevant outcomes changed, but not others 6. Statistical analyses show that your efforts were important 7. Outcomes changed in different sites where you worked 8. Outcomes changed each time you phased-in your efforts 9. Outcomes changed most where your efforts were most intense
Step #3: Gather quantitative data from participants and others Can we find persuasive evidence that: 1. Outcomes changed after your efforts, but not elsewhere 2. Outcomes changed at appropriate times, but only after your efforts 3. Outcomes changed as you switched your efforts on and off in different sites 4. For a group, outcomes departed from what was expected 5. For individuals, outcomes departed from what was expected
The graph below compares the ratings given by respondents who did see a Specialist Domestic Violence Police Officer and those who didn t, in relation to the support they received from the Police following the initial report of an incident. Did not see DV Officer Long term support 25.00% 25.00% 15.63% 25.00% 9.38% Saw DV Officer 30% 30% 20% 20% 0% 20% 40% 60% 80% 100% Very Good Good Fair Poor Very Poor
Step #4: Gather qualitative data about your project 1. Gather rich descriptions of your project and participants: * In-depth journalism, detective work, medical diagnosis * Case studies, modus operandi, most significant change, success case methods 2. Examine the pattern of results within your logic model (program theory/logical framework/theory of change/outcome map, outcome chain, outcome sequence chart): * Study data on inputs, activities, outputs, and outcomes * Do the patterns support or dispute a plausible association?
Step #5: Deliberately seek plausible alternative explanations Other reasons the outcomes may have changed May seem strange, but very important for your credibility And we do see other possibilities fairly often Four main steps: 1. Identify the most likely possibilities 2. Seek evidence for the contribution of each one 3. Seek evidence against the contribution of each one 4. Compare and draw conclusions Ideally, design your research up-front to test other possibilities
Step #6: Assess your project s contribution Pull together multiple lines of evidence * About your project * About plausible alternative explanations Include both quantitative and qualitative information Seek consensus of informed outsiders * Jury deliberations * NIH consensus-building panels * Foundation allocation committees * Others, especially in the UK context?
