New Paradata from Text & Email Contact Attempts
Exploring New Paradata from Text & Email Contact
Attempts in Federal Survey Data Collection
Matt Virgile and Laura Hergert
Office of Survey and Census Analytics
U.S. Census Bureau
2024 FedCASIC Workshops
April 16, 2024
1
Disclaimer: This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those
of the authors and not those of the U.S. Census Bureau. The paper has been reviewed for disclosure avoidance and approved under CBDRB-
FY24-OSCA002-005.
Background – Contact History Instrument (CHI)
•
Used by Census interviewers (Field Representatives, or FRs) to record
each contact attempt in select Federal surveys
•
CHI captures
mode
of contact attempt on first screen
•
Mode options formerly limited to personal visit or telephone
•
Following CHI screens capture other data (e.g., whether contact made,
whether interview completed, contact strategies, respondent concerns)
•
New CHI mode options added in 2022 for
text and email
contacts
•
May not be used to conduct an interview, but may be used to make contact
for other reasons
•
New following CHI screens capture whether text/email was outgoing/incoming,
purpose(s) of contact attempt, who sent an incoming message
2
CHI Text/Email Screens (1)
3
CHI Text/Email Screens (2)
4
Research Questions
•
How common is text/email usage in Federal surveys?
•
How does text/email usage vary by Census Regional
Office (RO)?
•
What are the most common reasons for text/email
usage?
5
6
Data
•
All CHI records in 2023 from two housing unit (HU) surveys
•
American Community Survey (ACS)
•
300k HUs sampled monthly
•
Data collected through mail/web response, with Computer-Assisted
Personal Interview (CAPI) sampling for nonresponding HUs
•
Non-longitudinal, one interview per HU
•
Current Population Survey (CPS)
•
60k HUs sampled monthly
•
All data collected through CATI/CAPI
•
Longitudinal, eight interviews per HU
•
Monthly interviews for four months (1-4), then eight months break,
then monthly interviews for four months (5-8)
7
Methods*
•
For both surveys, compute percentage of:
•
Contact attempts made via text/email
•
FRs with any text/email usage (overall and by RO)
•
HUs with any text/email usage (overall and by RO)
•
Texts/emails with a specified purpose option
•
For CPS only, compute FR-level and HU-level percentages
by interview number (1 thru 8)
8
*Analysis excludes FRs/HUs that were assigned to CAPI but did not have any CHI records
Limitations
•
Completeness of CHI records
•
Some FRs may not record every contact attempt
•
Training on capturing text/email in CHI may vary by survey
or RO
•
Text/email differs from personal visit/telephone since
“threads” and gaps in time between messages may be
confusing for what to capture in CHI
•
Text attempts may also be confounded with telephone
contacts since both rely on phone numbers
9
Analysis 1 – Overall CHI Text/Email Contacts
10
D
i
s
t
r
i
b
u
t
i
o
n
o
f
A
l
l
C
H
I
C
o
n
t
a
c
t
A
t
t
e
m
p
t
s
,
b
y
S
u
r
v
e
y
a
n
d
M
o
d
e
11
P
e
r
c
e
n
t
o
f
A
l
l
C
H
I
C
o
n
t
a
c
t
A
t
t
e
m
p
t
s
,
b
y
S
u
r
v
e
y
a
n
d
M
o
d
e
(
T
e
x
t
/
E
m
a
i
l
O
n
l
y
)
12
Analysis 2 – CHI Text/Email Contacts by FR
13
P
e
r
c
e
n
t
o
f
F
R
s
w
i
t
h
C
H
I
T
e
x
t
/
E
m
a
i
l
C
o
n
t
a
c
t
A
t
t
e
m
p
t
s
,
b
y
S
u
r
v
e
y
a
n
d
M
o
d
e
14
P
e
r
c
e
n
t
o
f
F
R
s
w
i
t
h
C
H
I
T
e
x
t
/
E
m
a
i
l
C
o
n
t
a
c
t
A
t
t
e
m
p
t
s
,
b
y
S
u
r
v
e
y
a
n
d
R
O
15
P
e
r
c
e
n
t
o
f
C
P
S
F
R
s
w
i
t
h
C
H
I
T
e
x
t
/
E
m
a
i
l
C
o
n
t
a
c
t
A
t
t
e
m
p
t
s
,
b
y
I
n
t
e
r
v
i
e
w
N
u
m
b
e
r
a
n
d
M
o
d
e
(
n
=
3
,
3
0
0
)
16
Analysis 3 – CHI Text/Email Contacts by HU
17
P
e
r
c
e
n
t
o
f
H
U
s
w
i
t
h
C
H
I
T
e
x
t
/
E
m
a
i
l
C
o
n
t
a
c
t
A
t
t
e
m
p
t
s
,
b
y
S
u
r
v
e
y
a
n
d
M
o
d
e
18
19
P
e
r
c
e
n
t
o
f
H
U
s
w
i
t
h
C
H
I
T
e
x
t
/
E
m
a
i
l
C
o
n
t
a
c
t
A
t
t
e
m
p
t
s
,
b
y
S
u
r
v
e
y
a
n
d
R
O
20
P
e
r
c
e
n
t
o
f
C
P
S
H
U
s
w
i
t
h
C
H
I
T
e
x
t
/
E
m
a
i
l
C
o
n
t
a
c
t
s
A
t
t
e
m
p
t
s
,
b
y
I
n
t
e
r
v
i
e
w
N
u
m
b
e
r
a
n
d
M
o
d
e
(
n
=
2
4
5
,
0
0
0
)
Analysis 4 – CHI Text/Email Contacts’
Reported Purpose
21
22
D
i
s
t
r
i
b
u
t
i
o
n
o
f
H
U
-
L
e
v
e
l
O
u
t
g
o
i
n
g
T
e
x
t
/
E
m
a
i
l
P
u
r
p
o
s
e
,
b
y
S
u
r
v
e
y
23
D
i
s
t
r
i
b
u
t
i
o
n
o
f
H
U
-
L
e
v
e
l
I
n
c
o
m
i
n
g
T
e
x
t
/
E
m
a
i
l
P
u
r
p
o
s
e
,
b
y
S
u
r
v
e
y
Conclusions (1)
•
Texts and emails make up a low percentage of all contact
attempts in CHI
•
1% in ACS, 5% in CPS
•
Most are outgoing texts, least are emails
•
Usage varies between ROs
•
More common among FRs and HUs covered by Western ROs
(Los Angeles, Denver, Chicago) than Eastern ROs (New York,
Philadelphia, Atlanta)
•
Unclear how much is driven by differences in training,
culture, geographic types, or other factors
24
Conclusions (2)
•
Text/email usage more common in CPS than ACS
•
Likely driven by CPS longitudinal design, with higher usage
after the first interview
•
Text/email reported purpose varies by survey
•
CPS-recorded outgoing contacts mostly to request appts (88%),
but ACS-recorded outgoing contact purpose varies more
•
Both surveys have a majority of recorded incoming contacts as
responses to a prior notice (55% in CPS, 53% in ACS)
•
CPS-recorded incoming contacts also mainly to schedule appts
(56%), but more varied in ACS (37% “Other”)
25
Discussion
•
Recent qualitative research by Laura Hergert may
provide more insight on text/email usage
•
Whether procedures for text/email usage were
communicated clearly to ROs
•
Whether these were covered in training for FRs
•
Whether and when text/email contacts are recorded by
FRs in CHI
•
Findings from FR focus groups will be presented at
American Association for Public Opinion Research
(AAPOR) Conference in May 2024
26
Exploring New Paradata from Text & Email Contact
Attempts in Federal Survey Data Collection
Matthew.Virgile@census.gov
27
In this study by Matt Virgile and Laura Hergert from the Office of Survey and Census Analytics, the focus is on analyzing paradata collected from text and email contact attempts in federal survey data collection. The research delves into the effectiveness of these communication modes, shedding light on strategies for improving survey response rates and data quality. Insights gained from this analysis can inform future survey methodologies and enhance the success of data collection efforts.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Exploring New Paradata from Text & Email Contact Attempts in Federal Survey Data Collection Matt Virgile and Laura Hergert Office of Survey and Census Analytics U.S. Census Bureau 2024 FedCASIC Workshops April 16, 2024 Disclaimer: This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not those of the U.S. Census Bureau. The paper has been reviewed for disclosure avoidance and approved under CBDRB- FY24-OSCA002-005. 1
Background Contact History Instrument (CHI) Used by Census interviewers (Field Representatives, or FRs) to record each contact attempt in select Federal surveys CHI captures mode of contact attempt on first screen Mode options formerly limited to personal visit or telephone Following CHI screens capture other data (e.g., whether contact made, whether interview completed, contact strategies, respondent concerns) New CHI mode options added in 2022 for text and email contacts May not be used to conduct an interview, but may be used to make contact for other reasons New following CHI screens capture whether text/email was outgoing/incoming, purpose(s) of contact attempt, who sent an incoming message 2
Research Questions How common is text/email usage in Federal surveys? How does text/email usage vary by Census Regional Office (RO)? What are the most common reasons for text/email usage? 5
Data All CHI records in 2023 from two housing unit (HU) surveys American Community Survey (ACS) 300k HUs sampled monthly Data collected through mail/web response, with Computer-Assisted Personal Interview (CAPI) sampling for nonresponding HUs Non-longitudinal, one interview per HU Current Population Survey (CPS) 60k HUs sampled monthly All data collected through CATI/CAPI Longitudinal, eight interviews per HU Monthly interviews for four months (1-4), then eight months break, then monthly interviews for four months (5-8) 7
Methods* For both surveys, compute percentage of: Contact attempts made via text/email FRs with any text/email usage (overall and by RO) HUs with any text/email usage (overall and by RO) Texts/emails with a specified purpose option For CPS only, compute FR-level and HU-level percentages by interview number (1 thru 8) *Analysis excludes FRs/HUs that were assigned to CAPI but did not have any CHI records 8
Limitations Completeness of CHI records Some FRs may not record every contact attempt Training on capturing text/email in CHI may vary by survey or RO Text/email differs from personal visit/telephone since threads and gaps in time between messages may be confusing for what to capture in CHI Text attempts may also be confounded with telephone contacts since both rely on phone numbers 9
Distribution of All CHI Contact Attempts, by Survey and Mode Distribution of All CHI Contact Attempts, by Survey and Mode 11
Percent of All CHI Contact Attempts, by Survey and Mode (Text/Email Only) Percent of All CHI Contact Attempts, by Survey and Mode (Text/Email Only) 12
Percent of FRs with CHI Text/Email Contact Attempts, by Survey and Mode Percent of FRs with CHI Text/Email Contact Attempts, by Survey and Mode 14
Percent of FRs with CHI Text/Email Contact Attempts, by Survey and RO Percent of FRs with CHI Text/Email Contact Attempts, by Survey and RO 15
Percent of CPS FRs with CHI Text/Email Contact Attempts, Percent of CPS FRs with CHI Text/Email Contact Attempts, by Interview Number and Mode (n=3,300) by Interview Number and Mode (n=3,300) 16
Percent of HUs with CHI Text/Email Contact Attempts, by Survey and Mode Percent of HUs with CHI Text/Email Contact Attempts, by Survey and Mode 18
Percent of HUs with CHI Text/Email Contact Attempts, by Survey and RO Percent of HUs with CHI Text/Email Contact Attempts, by Survey and RO 19
Percent of CPS HUs with CHI Text/Email Contacts Attempts, Percent of CPS HUs with CHI Text/Email Contacts Attempts, by Interview Number and Mode (n=245,000) by Interview Number and Mode (n=245,000) 20
Analysis 4 CHI Text/Email Contacts Reported Purpose 21
Distribution of HU Distribution of HU- -Level Outgoing Text/Email Purpose, by Survey Level Outgoing Text/Email Purpose, by Survey 22
Distribution of HU Distribution of HU- -Level Incoming Text/Email Purpose, by Survey Level Incoming Text/Email Purpose, by Survey 23
Conclusions (1) Texts and emails make up a low percentage of all contact attempts in CHI 1% in ACS, 5% in CPS Most are outgoing texts, least are emails Usage varies between ROs More common among FRs and HUs covered by Western ROs (Los Angeles, Denver, Chicago) than Eastern ROs (New York, Philadelphia, Atlanta) Unclear how much is driven by differences in training, culture, geographic types, or other factors 24
Conclusions (2) Text/email usage more common in CPS than ACS Likely driven by CPS longitudinal design, with higher usage after the first interview Text/email reported purpose varies by survey CPS-recorded outgoing contacts mostly to request appts (88%), but ACS-recorded outgoing contact purpose varies more Both surveys have a majority of recorded incoming contacts as responses to a prior notice (55% in CPS, 53% in ACS) CPS-recorded incoming contacts also mainly to schedule appts (56%), but more varied in ACS (37% Other ) 25
Discussion Recent qualitative research by Laura Hergert may provide more insight on text/email usage Whether procedures for text/email usage were communicated clearly to ROs Whether these were covered in training for FRs Whether and when text/email contacts are recorded by FRs in CHI Findings from FR focus groups will be presented at American Association for Public Opinion Research (AAPOR) Conference in May 2024 26
Exploring New Paradata from Text & Email Contact Attempts in Federal Survey Data Collection Matthew.Virgile@census.gov 27
