Addressing Survey Fatigue in Program Evaluation
undefined
Overcoming
Death by
Survey
KIM STOKES, MHS, PA-C
AMI STEELE, MS, PA-C
Learning objectives
Learning Objectives:
Recognize the role of survey fatigue in insufficient effort responding (IER) and the subsequent
impact of IER on program evaluation data
Explore ways of transforming program evaluation practices to obtain meaningful learner
feedback
Evaluate a variety of formal and/or informal survey methods/platforms for potential use in
their programs
Discover resources for formal and informal feedback for program evaluation
Why do we
survey?
C
o
n
s
i
d
e
r
t
h
i
s
:
H
o
w
m
a
n
y
s
u
r
v
e
y
s
d
o
e
s
y
o
u
r
p
r
o
g
r
a
m
d
o
p
e
r
c
o
h
o
r
t
?
Insufficient Effort Responding
Socially desirable responses (“faking”)
◦
Content responsive-respond in a way that over endorses negatives and positives (complaints, praises)
◦
Content non-responsitivity– ignores the contents of the items, unwilling to comply with survey
procedures, linguistic incompetence
Acquiescence
Careless or inattentive/random responding
Furnham A, Hyde G, Trickey G. Personality and value correlates of careless and erratic questionnaire responses.
Personality and Individual Differences
.
2015;80:64-67.
Nguyen HLT.
Tired of Survey Fatigue? Insufficient Effort Responding Due to Survey Fatigue.
ProQuest Dissertations Publishing; 2017.
Survey fatigue
AUGMENTED BY
Long surveys
Difficult topics
Question complexity
Open-ended questions
POSSIBLE SOLUTIONS
Shorter surveys
Meaningful/Purposeful timing
Technology
Alternatives to “surveys”
O'Reilly-Shah VN. Factors influencing healthcare provider respondent fatigue answering a globally administered in-app survey.
PeerJ PrePrints
. 2017.
Alternatives to
Traditional Surveys
EXPLORATION AND DISCOVERY
A
c
t
i
v
i
t
y
Choose a table topic of interest (see labels on tables)
As a group, brainstorm alternatives to the traditional survey for this evaluation OR if you already
have a non-traditional method, share it with the group
Names a spokesperson for summary
Spokesperson will share with large group the topic of the table and alternatives discussed
E
x
a
m
p
l
e
s
o
f
A
l
t
e
r
n
a
t
i
v
e
s
QUANTITATIVE
Point/Click technology
◦
Poll anywhere
◦
Kazoo
◦
Turning point
Minute papers (1-2 take aways)
3 good/3 bad
Keep/Change/Eliminate
QUALITATIVE
Minute papers (1-2 impressions)
Focus groups
“Debriefings”
“Change” open-ended stickers
PROS
No screens, but VERY visual
Full participation (100%)
Fun/interactive WITHOUT a
SCREEN
Themes/concepts pre-determined
Quick and easy
Allows for quick follow up process
CONS
Loss of long qualitative responses
Carrying around large papers
Prep time to print labels
Time to count
K
e
e
p
/
C
h
a
n
g
e
/
E
l
i
m
i
n
a
t
e
PROS
No screens
Personal
Verbiage can be clarified
with script for consistency
Allows for immediate
probing
CONS
Possible long responses
Time: transcript, analysis,
theming
F
o
c
u
s
g
r
o
u
p
s
/
d
e
b
r
i
e
f
s
S
h
a
r
i
n
g
Y
O
U
R
e
x
a
m
p
l
e
s
!
(This slide will be completed based
on discussions we hear around the
room)
T
h
a
n
k
y
o
u
f
o
r
c
o
m
i
n
g
t
o
o
u
r
p
r
e
s
e
n
t
a
t
i
o
n
a
n
d
s
h
a
r
i
n
g
y
o
u
r
i
d
e
a
s
!
References
Furnham A, Hyde G, Trickey G. Personality and value correlates
of careless and erratic questionnaire responses.
Personality and
Individual Differences.
2015;80:64-67.
Nguyen HLT. Tired of Survey Fatigue? Insufficient Effort
Responding Due to Survey Fatigue.
ProQuest Dissertations
Publishing
; 2017.
O'Reilly-Shah VN. Factors influencing healthcare provider
respondent fatigue answering a globally administered in-app
survey.
PeerJ PrePrints
. 2017.
Survey fatigue can lead to insufficient effort responding in program evaluations, affecting the quality of data collected. This presentation explores the causes and impacts of survey fatigue, suggests solutions such as shorter surveys and open-ended questions, and discusses alternatives to traditional survey methods for more effective evaluation practices.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Overcoming Death by Survey KIM STOKES, MHS, PA-C AMI STEELE, MS, PA-C
Learning objectives Learning Objectives: Recognize the role of survey fatigue in insufficient effort responding (IER) and the subsequent impact of IER on program evaluation data Explore ways of transforming program evaluation practices to obtain meaningful learner feedback Evaluate a variety of formal and/or informal survey methods/platforms for potential use in their programs Discover resources for formal and informal feedback for program evaluation
Why do we survey? Consider this: Consider this: How many surveys does your program do per cohort?
Insufficient Effort Responding Socially desirable responses ( faking ) Content responsive-respond in a way that over endorses negatives and positives (complaints, praises) Content non-responsitivity ignores the contents of the items, unwilling to comply with survey procedures, linguistic incompetence Acquiescence Careless or inattentive/random responding Furnham A, Hyde G, Trickey G. Personality and value correlates of careless and erratic questionnaire responses. Personality and Individual Differences. 2015;80:64-67. Nguyen HLT. Tired of Survey Fatigue? Insufficient Effort Responding Due to Survey Fatigue. ProQuest Dissertations Publishing; 2017.
Survey fatigue AUGMENTED BY POSSIBLE SOLUTIONS Long surveys Shorter surveys Difficult topics Meaningful/Purposeful timing Question complexity Technology Open-ended questions Alternatives to surveys O'Reilly-Shah VN. Factors influencing healthcare provider respondent fatigue answering a globally administered in-app survey. PeerJ PrePrints. 2017.
Alternatives to Traditional Surveys EXPLORATION AND DISCOVERY
Activity Activity Choose a table topic of interest (see labels on tables) As a group, brainstorm alternatives to the traditional survey for this evaluation OR if you already have a non-traditional method, share it with the group Names a spokesperson for summary Spokesperson will share with large group the topic of the table and alternatives discussed
Examples of Alternatives Examples of Alternatives QUANTITATIVE QUALITATIVE Point/Click technology Poll anywhere Minute papers (1-2 impressions) Focus groups Kazoo Debriefings Turning point Change open-ended stickers Minute papers (1-2 take aways) 3 good/3 bad Keep/Change/Eliminate
PROS No screens, but VERY visual Keep/Change/Eliminate Keep/Change/Eliminate Full participation (100%) Fun/interactive WITHOUT a SCREEN At a glance Suggested setting Avoid use with Themes/concepts pre-determined Workshop/event with up to 12 components Course evaluation (clin med, anatomy, etc.) Pre-populated items with quick responses. Quick and easy Allows for quick follow up process Interactive Program/course with sell-defined sections Full program evaluation (too many pieces) CONS Loss of long qualitative responses Carrying around large papers Quantitative Analysis Prep time to print labels Time to count
Focus groups/debriefs Focus groups/debriefs PROS No screens Personal Verbiage can be clarified with script for consistency Allows for immediate probing At a glance Suggested setting Avoid use with Full program evaluation Course evaluation (clin med, anatomy, etc.) Divided cohort answering common scripted questions that can be reviewed. End of curriculum (end pre-clinical, end clinical) CONS Possible long responses Time: transcript, analysis, theming Whole cohorts at one time Qualitative Analysis
Survey Topic Suggested Alternate Sharing YOUR Sharing YOUR examples! examples! (This slide will be completed based on discussions we hear around the room)
References Furnham A, Hyde G, Trickey G. Personality and value correlates of careless and erratic questionnaire responses. Personality and Individual Differences. 2015;80:64-67. Thank you for Thank you for coming to our coming to our presentation presentation and sharing and sharing your ideas! your ideas! Nguyen HLT.Tired of Survey Fatigue? Insufficient Effort Responding Due to Survey Fatigue.ProQuest Dissertations Publishing; 2017. O'Reilly-Shah VN. Factors influencing healthcare provider respondent fatigue answering a globally administered in-app survey. PeerJ PrePrints. 2017.
