Evaluating a Behavioural Science Approach for Increasing Advice Service Uptake

 
Evaluating a behavioural science approach
to increasing the take up of advice services
offered to residents in economic hardship
 
Vicky Powers
Derbyshire County Council
 
Opportunity and challenge
 
Opportunity
Derbyshire’s Local Welfare Assistance Fund (Derbyshire Discretionary Fund) is a k
ey
touchpoint - speaks with Derbyshire’s most vulnerable residents, at a critical point.
27k calls, ½ callers from 20% most deprived areas, ~40% of calls could be prevented with
better financial management.
Improve signposting and referrals to support
Preventative opportunity - shift from ‘head in the sand’ to ‘engagement’
Build relationships with partners (big picture) - joined up approach between agencies.
Challenge
DDF was signposting to Citizens Advice / other agencies - but not tracking data or
outcomes 
‘off into the ether’
Time pressured call, the callers were focused on the immediate decision (present
biased), information wasn’t landing (cognitive load, linked with debt).
Welfare Rights internal referral data showed that contact was not always made, (38%
dropped), despite repeat calls.
 
L
o
c
a
l
 
G
o
v
e
r
n
m
e
n
t
A
s
s
o
c
i
a
t
i
o
n
 
(
L
G
A
)
B
e
h
a
v
i
o
u
r
a
l
 
I
n
s
i
g
h
t
s
 
P
r
o
j
e
c
t
 
A
pply behavioural insights to a range of local government challenges.
&
Introduce and embed a behavioural science approach.
DDF Project
1.
T
o
 
a
p
p
l
y
 
b
e
h
a
v
i
o
u
r
a
l
 
i
n
s
i
g
h
t
s
 
t
o
 
i
n
c
r
e
a
s
e
 
c
a
l
l
e
r
 
r
e
f
e
r
r
a
l
s
 
t
o
 
s
u
p
p
o
r
t
 
a
g
e
n
c
i
e
s
.
I
n
c
r
e
a
s
i
n
g
 
s
u
p
p
o
r
t
 
a
c
c
e
s
s
e
d
 
b
y
 
v
u
l
n
e
r
a
b
l
e
 
r
e
s
i
d
e
n
t
s
.
 
2.
T
o
 
u
s
e
 
a
n
 
e
v
i
d
e
n
c
e
 
b
a
s
e
d
 
a
p
p
r
o
a
c
h
 
t
o
 
e
v
a
l
u
a
t
e
 
t
h
e
 
i
m
p
a
c
t
 
o
f
 
a
n
y
i
n
t
e
r
v
e
n
t
i
o
n
s
 
-
 
 
 
R
a
n
d
o
m
i
s
e
d
 
C
o
n
t
r
o
l
l
e
d
 
T
r
i
a
l
s
 
(
R
C
T
)
.
 
3.
T
o
 
h
e
l
p
 
p
r
o
v
e
 
t
h
e
 
v
a
l
u
e
 
a
n
d
 
e
f
f
e
c
t
i
v
e
n
e
s
s
 
o
f
 
a
 
b
e
h
a
v
i
o
u
r
a
l
 
s
c
i
e
n
c
e
a
p
p
r
o
a
c
h
.
 
 
P
r
o
j
e
c
t
 
O
v
e
r
v
i
e
w
 
Scoping Phase - To establish feasibility and initial concepts
COVID-19 - Paused and reviewed feasibility
Delivery Phase
 
Identify & build partnership relationships
 
Project Management - GDPR / Ethics sign off
Design and launch 3 trials
 
Citizens Advice - Completed
 
Welfare Rights - Completed
 
Money Sorted - Low referral numbers
Analysis & Write up
Final Report - Draft 1 being reviewed
R
e
f
e
r
r
a
l
 
p
r
o
c
e
s
s
 
-
 
b
e
h
a
v
i
o
u
r
c
h
a
n
g
e
 
Mosaic System. Call
management
 
system
Provide prompts  in
the call process
Screening and Decision
Support Materials.
Cheat sheets
Increasing DM agency
Only referring ‘engaged’
callers, DM judgement
is important.
Increased focus and
awareness on
referrals. Through 
m
eetings, raffles
Providing Feedback to DMs 
Case Studies on impact
Reinforces behaviour
REFERRAL TRIALS
 
R
a
n
d
o
m
i
s
e
d
 
C
o
n
t
r
o
l
 
T
r
i
a
l
s
 
W
h
y
 
R
C
T
?
Gold standard in terms of evaluation
One of the aims of the funding stream
 
S
c
r
e
e
n
i
n
g
 
T
r
i
a
l
s
 
-
 
I
m
p
r
a
c
t
i
c
a
l
:
 
r
a
n
d
o
m
i
s
i
n
g
 
a
n
d
 
c
o
n
t
r
o
l
 
o
f
 
t
h
e
 
i
n
t
e
r
v
e
n
t
i
o
n
s
.
 
(
P
r
e
 
-
 
P
o
s
t
)
R
e
f
e
r
r
a
l
 
h
a
n
d
o
v
e
r
 
t
r
i
a
l
s
Citizen’s Advice
Welfare Rights
R
a
n
d
o
m
i
s
i
n
g
 
-
 
U
s
i
n
g
 
t
h
e
 
l
a
s
t
 
d
i
g
i
t
 
o
f
 
t
h
e
 
N
a
t
i
o
n
a
l
 
I
n
s
u
r
a
n
c
e
 
N
u
m
b
e
r
M
o
n
i
t
o
r
i
n
g
 
/
 
T
r
a
c
k
i
n
g
 
r
e
s
u
l
t
s
Through Mosaic
Data from partner organisation (contact and issues) (anonymised and aggregated)
 
C
i
t
i
z
e
n
s
 
A
d
v
i
c
e
 
T
r
i
a
l
 
Project Set Up
Defined the profile of caller CA wanted.
No complex issues
Broad range of issues
Qualified / Engaged applicants
 
Trial
A commitment device to
reduce the drop out of referrals
to reduce staff time spent ‘chasing’
referrals
 
The referral was conditional on applicants
responding to the SMS
SMS - sent +24 hours after the call
 
Sample of 610 was required to conduct an
RCT
.
 
C
i
t
i
z
e
n
s
 
A
d
v
i
c
e
 
t
r
i
a
l
 
r
e
s
u
l
t
s
 
The CAB trial was run from the 11th May 2021 to the 9th November 2021.
 
Stopped early as the data showed SMS intervention was acting as a barrier to referral take up.
 
T
h
e
 
S
M
S
 
i
n
t
e
r
v
e
n
t
i
o
n
 
r
e
d
u
c
e
s
 
t
h
e
l
i
k
e
l
i
h
o
o
d
 
o
f
 
c
a
l
l
e
r
s
 
m
a
k
i
n
g
c
o
n
t
a
c
t
 
w
i
t
h
 
C
A
B
 
a
t
 
a
 
9
9
.
9
%
c
o
n
f
i
d
e
n
c
e
 
l
e
v
e
l
.
 
The value of z is 4.6614. The value
of p is < .00001. The result is
significant at p < .01. Two tailed z-
test
 
R
e
s
p
o
n
d
i
n
g
 
p
o
s
i
t
i
v
e
l
y
 
t
o
 
t
h
e
 
S
M
S
t
e
x
t
 
i
n
c
r
e
a
s
e
s
 
t
h
e
 
l
i
k
e
l
i
h
o
o
d
 
o
f
c
a
l
l
e
r
s
 
t
a
k
i
n
g
 
u
p
 
t
h
e
 
r
e
f
e
r
r
a
l
 
w
a
s
m
a
r
g
i
n
a
l
l
y
 
s
i
g
n
i
f
i
c
a
n
t
 
w
i
t
h
 
9
0
%
c
o
n
f
i
d
e
n
c
e
.
 
The value of z is 1.3617. The value
of p is .08692. The result is
significant at p < .10. One tailed z-
test.
 
W
e
l
f
a
r
e
 
R
i
g
h
t
s
 
T
r
i
a
l
 
W
o
r
k
i
n
g
 
i
n
 
c
o
n
c
e
r
t
 
w
i
t
h
 
C
i
t
i
z
e
n
s
 
A
d
v
i
c
e
T
r
i
a
l
 
Same set up as CA.
 
The SMS was used as a reminder,
recipients do not need to confirm to be
referred.
 
SMS Sent after the referral. Response was
recorded.
 
Wider access to data as an internal service
 
W
e
l
f
a
r
e
 
R
i
g
h
t
s
 
t
r
i
a
l
 
r
e
s
u
l
t
s
 
The WR trial was run from the 22nd April 2021 to the 22nd Feb 2022.
The trial was stopped early again because of the negative impact of the SMS on referral
take up.
 
T
h
e
 
S
M
S
 
i
n
t
e
r
v
e
n
t
i
o
n
 
r
e
d
u
c
e
s
 
t
h
e
l
i
k
e
l
i
h
o
o
d
 
o
f
 
c
a
l
l
e
r
s
 
m
a
k
i
n
g
c
o
n
t
a
c
t
 
w
i
t
h
 
W
R
 
a
t
 
a
 
9
5
%
c
o
n
f
i
d
e
n
c
e
 
l
e
v
e
l
.
 
The value of z is -2.223. The value
of p is .02642. The result is significant
at p < .05.
 
R
e
s
p
o
n
d
i
n
g
 
p
o
s
i
t
i
v
e
l
y
 
t
o
 
t
h
e
 
S
M
S
t
e
x
t
 
i
n
c
r
e
a
s
e
s
 
t
h
e
 
l
i
k
e
l
i
h
o
o
d
 
o
f
c
a
l
l
e
r
s
 
t
a
k
i
n
g
 
u
p
 
t
h
e
 
r
e
f
e
r
r
a
l
.
 
T
h
i
s
w
a
s
 
m
a
r
g
i
n
a
l
l
y
 
s
i
g
n
i
f
i
c
a
n
t
 
w
i
t
h
9
0
%
 
c
o
n
f
i
d
e
n
c
e
.
 
The value of z is 1.6341. The value of
p is .05155. The result is significant at
p < .10. One tailed z-test.
 
S
M
S
 
t
r
i
a
l
 
c
o
n
c
l
u
s
i
o
n
s
 
The SMS acts as a barrier to the take up of the referrals.
Correct
 to stop the trials - Given 
DDF goals and the goals of the project
 -  increasing
support 
to
 vulnerable residents.
 
We believe the reasons for these results could include:
1.
Any extra step requiring effort results in drop-out. Complexity adds confusion.
2.
Increase in SMS fraud during the period of the trial - reduces trust
 (increase 700%)
3.
We changed the screening process / DM 
- C
allers are only being referred when
engaged. The need for a commitment device has been reduced.
 
SMS has proved challenging for DCC in other contexts, for school 
h
oliday food vouchers,
15% of payments made by SMS were initially unclaimed, despite this being a direct
payment to recipients.
 
BUT for those who did respond to the CAB SMS it did act as a commitment device -
increasing referral take up. Marginally significant, if the trial had gone on probably a
significant finding
 
R
e
f
l
e
c
t
i
o
n
s
 
Challenges:
Covid impact on services resulted in a longer set up time
RCT challenging to put something in place that was practical –
couldn’t focus on screening process
Hard to identify and measure outcomes for a control group
Ethics and informed consent difficult with a cohort at the point of
financial crisis
Adherence to process by case assessors was an issue – lower
than expected sample size
Long term outcomes difficult to capture
 
 
 
R
e
f
l
e
c
t
i
o
n
s
 
Benefits:
RCT over pre/post as there were constant changes throughout
the trial period – new staff, new funding streams, massive
increase in demand
Infrastructure now in place for future test and learn
Learnt what didn’t work and responded in a timely fashion
 
 
 
Slide Note
Embed
Share

Derbyshire County Council is evaluating a behavioural science approach to enhance the uptake of advice services for residents in economic hardship. The project focuses on increasing caller referrals to support agencies, using Randomised Controlled Trials (RCTs) to evaluate interventions' impact and prove the effectiveness of this approach. By improving signposting, referrals, and engagement strategies, the project aims to assist vulnerable residents in accessing crucial support services effectively.


Uploaded on Sep 14, 2024 | 2 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Evaluating a behavioural science approach to increasing the take up of advice services offered to residents in economic hardship Vicky Powers Derbyshire County Council

  2. Opportunity and challenge Opportunity Derbyshire s Local Welfare Assistance Fund (Derbyshire Discretionary Fund) is a key touchpoint - speaks with Derbyshire s most vulnerable residents, at a critical point. 27k calls, callers from 20% most deprived areas, ~40% of calls could be prevented with better financial management. Improve signposting and referrals to support Preventative opportunity - shift from head in the sand to engagement Build relationships with partners (big picture) - joined up approach between agencies. Challenge DDF was signposting to Citizens Advice / other agencies - but not tracking data or outcomes off into the ether Time pressured call, the callers were focused on the immediate decision (present biased), information wasn t landing (cognitive load, linked with debt). Welfare Rights internal referral data showed that contact was not always made, (38% dropped), despite repeat calls.

  3. Local Government Association (LGA) Behavioural Insights Project Apply behavioural insights to a range of local government challenges. & Introduce and embed a behavioural science approach. DDF Project 1. To apply behavioural insights to increase caller referrals to support agencies. Increasing support accessed by vulnerable residents. 2. To use an evidence based approach to evaluate the impact of any interventions - Randomised Controlled Trials (RCT). 3. To help prove the value and effectiveness of a behavioural science approach.

  4. Project Overview Scoping Phase - To establish feasibility and initial concepts COVID-19 - Paused and reviewed feasibility Delivery Phase Identify & build partnership relationships Project Management - GDPR / Ethics sign off Design and launch 3 trials Citizens Advice - Completed Welfare Rights - Completed Money Sorted - Low referral numbers Analysis & Write up Final Report - Draft 1 being reviewed

  5. Referral process - behaviour change REFERRAL TRIALS Increased focus and awareness on referrals. Through meetings, raffles Providing Feedback to DMs Case Studies on impact Reinforces behaviour Mosaic System. Call management system Provide prompts in the call process Increasing DM agency Only referring engaged callers, DM judgement is important. Screening and Decision Support Materials. Cheat sheets

  6. Randomised Control Trials Why RCT? Gold standard in terms of evaluation One of the aims of the funding stream Screening Trials - Impractical: randomising and control of the interventions. (Pre - Post) Referral handover trials Citizen s Advice Welfare Rights Randomising - Using the last digit of the National Insurance Number Monitoring / Tracking results Through Mosaic Data from partner organisation (contact and issues) (anonymised and aggregated)

  7. Citizens Advice Trial Project Set Up Defined the profile of caller CA wanted. No complex issues Broad range of issues Qualified / Engaged applicants Trial A commitment device to reduce the drop out of referrals to reduce staff time spent chasing referrals The referral was conditional on applicants responding to the SMS SMS - sent +24 hours after the call Sample of 610 was required to conduct an RCT.

  8. Citizens Advice trial results The CAB trial was run from the 11th May 2021 to the 9th November 2021. Stopped early as the data showed SMS intervention was acting as a barrier to referral take up. The SMS intervention reduces the likelihood of callers making contact with CAB at a 99.9% confidence level. The value of z is 4.6614. The value of p is < .00001. The result is significant at p < .01. Two tailed z- test Responding positively to the SMS text increases the likelihood of callers taking up the referral was marginally significant with 90% confidence. The value of z is 1.3617. The value of p is .08692. The result is significant at p < .10. One tailed z- test.

  9. Welfare Rights Trial Working in concert with Citizens Advice Trial Same set up as CA. The SMS was used as a reminder, recipients do not need to confirm to be referred. SMS Sent after the referral. Response was recorded. Wider access to data as an internal service

  10. Welfare Rights trial results The WR trial was run from the 22nd April 2021 to the 22nd Feb 2022. The trial was stopped early again because of the negative impact of the SMS on referral take up. The SMS intervention reduces the likelihood of callers making contact with WR at a 95% confidence level. The value of z is -2.223. The value of p is .02642. The result is significant at p < .05. Responding positively to the SMS text increases the likelihood of callers taking up the referral. This was marginally significant with 90% confidence. The value of z is 1.6341. The value of p is .05155. The result is significant at p < .10. One tailed z-test.

  11. SMS trial conclusions The SMS acts as a barrier to the take up of the referrals. Correct to stop the trials - Given DDF goals and the goals of the project - increasing support to vulnerable residents. We believe the reasons for these results could include: 1. Any extra step requiring effort results in drop-out. Complexity adds confusion. 2. Increase in SMS fraud during the period of the trial - reduces trust (increase 700%) 3. We changed the screening process / DM - Callers are only being referred when engaged. The need for a commitment device has been reduced. SMS has proved challenging for DCC in other contexts, for school holiday food vouchers, 15% of payments made by SMS were initially unclaimed, despite this being a direct payment to recipients. BUT for those who did respond to the CAB SMS it did act as a commitment device - increasing referral take up. Marginally significant, if the trial had gone on probably a significant finding

  12. Reflections Challenges: Covid impact on services resulted in a longer set up time RCT challenging to put something in place that was practical couldn t focus on screening process Hard to identify and measure outcomes for a control group Ethics and informed consent difficult with a cohort at the point of financial crisis Adherence to process by case assessors was an issue lower than expected sample size Long term outcomes difficult to capture

  13. Reflections Benefits: RCT over pre/post as there were constant changes throughout the trial period new staff, new funding streams, massive increase in demand Infrastructure now in place for future test and learn Learnt what didn t work and responded in a timely fashion

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#