Assessing the 2020 Census Quality Data Report
undefined
Assessing the 2020 Census:
Final Report
Consensus Study
Committee on National Statistics (CNSTAT)
Released October 3,
2023
(www.nap.edu)National Academies PressAssessing the 2020 Census: Final Report | The
ASSESSING THE 2020 CENSUS
Statement of Task
2
The National Academies of Sciences, Engineering, and Medicine will appoint an ad hoc panel to
review the quality of the data that were collected in the 2020 Census. As part of its work, the panel
will:
1.
Review information from the Census Bureau on the data collected as well as various process
measures and indicators of data quality obtained as part of the 2020 Census operations;
2.
Review other available information, such as results from demographic analysis, process
measures and preliminary results from the post-enumeration survey; and analyses of
administrative records; and
3.
Consider the results from evaluations of similar indicators from the 2010 and 2000 Censuses.
The panel will produce … a final report that includes conclusions about the quality of the data
collected in the 2020 Census and makes recommendations for further research by the Census
Bureau to evaluate the quality of the 2020 data and to begin planning the 2030 Census. …
Panel to Evaluate the Quality of the 2020 Census
T
e
r
e
s
a
A
.
S
u
l
l
i
v
a
n
(
C
h
a
i
r
)
,
D
e
p
a
r
t
m
e
n
t
o
f
S
o
c
i
o
l
o
g
y
,
U
n
i
v
e
r
s
i
t
y
o
f
V
i
r
g
i
n
i
a
M
a
r
g
o
A
n
d
e
r
s
o
n
,
D
e
p
a
r
t
m
e
n
t
o
f
H
i
s
t
o
r
y
,
U
n
i
v
e
r
s
i
t
y
o
f
W
i
s
c
o
n
s
i
n
–
M
i
l
w
a
u
k
e
e
(
e
m
e
r
i
t
a
)
R
o
b
e
r
t
M
.
B
e
l
l
,
*
G
o
o
g
l
e
a
n
d
A
T
&
T
L
a
b
s
(
r
e
t
i
r
e
d
)
K
a
t
h
r
y
n
E
d
i
n
(
N
A
S
)
,
D
e
p
a
r
t
m
e
n
t
o
f
S
o
c
i
o
l
o
g
y
,
P
r
i
n
c
e
t
o
n
U
n
i
v
e
r
s
i
t
y
M
a
r
c
H
a
m
e
l
,
S
t
a
t
i
s
t
i
c
s
C
a
n
a
d
a
(
r
e
t
i
r
e
d
)
G
e
o
r
g
e
T
.
L
i
g
l
e
r
(
N
A
E
)
,
D
e
p
a
r
t
m
e
n
t
o
f
M
u
l
t
i
d
i
s
c
i
p
l
i
n
a
r
y
E
n
g
i
n
e
e
r
i
n
g
,
T
e
x
a
s
A
&
M
U
n
i
v
e
r
s
i
t
y
T
h
o
m
a
s
A
.
L
o
u
i
s
,
*
D
e
p
a
r
t
m
e
n
t
o
f
B
i
o
s
t
a
t
i
s
t
i
c
s
,
J
o
h
n
s
H
o
p
k
i
n
s
U
n
i
v
e
r
s
i
t
y
(
e
m
e
r
i
t
u
s
)
L
l
o
y
d
B
.
P
o
t
t
e
r
,
*
T
e
x
a
s
D
e
m
o
g
r
a
p
h
i
c
C
e
n
t
e
r
,
U
n
i
v
e
r
s
i
t
y
o
f
T
e
x
a
s
a
t
S
a
n
A
n
t
o
n
i
o
J
o
s
e
p
h
J
.
S
a
l
v
o
,
*
U
n
i
v
e
r
s
i
t
y
o
f
V
i
r
g
i
n
i
a
B
i
o
c
o
m
p
l
e
x
i
t
y
I
n
s
t
i
t
u
t
e
R
e
g
i
n
a
S
h
i
h
,
D
e
p
a
r
t
m
e
n
t
o
f
E
p
i
d
e
m
i
o
l
o
g
y
,
R
o
l
l
i
n
s
S
c
h
o
o
l
o
f
P
u
b
l
i
c
H
e
a
l
t
h
,
E
m
o
r
y
U
n
i
v
e
r
s
i
t
y
3
C
.
M
a
t
t
h
e
w
S
n
i
p
p
,
S
c
h
o
o
l
o
f
H
u
m
a
n
i
t
i
e
s
a
n
d
S
c
i
e
n
c
e
s
,
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
E
d
w
a
r
d
T
e
l
l
e
s
,
D
e
p
a
r
t
m
e
n
t
o
f
S
o
c
i
o
l
o
g
y
,
U
n
i
v
e
r
s
i
t
y
o
f
C
a
l
i
f
o
r
n
i
a
,
I
r
v
i
n
e
W
e
n
d
y
U
n
d
e
r
h
i
l
l
,
N
a
t
i
o
n
a
l
C
o
n
f
e
r
e
n
c
e
o
f
S
t
a
t
e
L
e
g
i
s
l
a
t
u
r
e
s
,
D
e
n
v
e
r
D
a
v
i
d
V
a
n
R
i
p
e
r
,
*
M
i
n
n
e
s
o
t
a
P
o
p
u
l
a
t
i
o
n
C
e
n
t
e
r
,
U
n
i
v
e
r
s
i
t
y
o
f
M
i
n
n
e
s
o
t
a
* Denotes member of the panel’s designated data analytic subgroup.
D
a
n
i
e
l
L
.
C
o
r
k
,
S
t
u
d
y
D
i
r
e
c
t
o
r
C
o
n
s
t
a
n
c
e
F
.
C
i
t
r
o
,
S
e
n
i
o
r
S
c
h
o
l
a
r
M
i
c
h
a
e
l
L
.
C
o
h
e
n
,
S
e
n
i
o
r
P
r
o
g
r
a
m
O
f
f
i
c
e
r
A
n
t
h
o
n
y
M
a
n
n
,
S
e
n
i
o
r
P
r
o
g
r
a
m
A
s
s
o
c
i
a
t
e
K
a
t
r
i
n
a
B
a
u
m
S
t
o
n
e
,
S
e
n
i
o
r
P
r
o
g
r
a
m
O
f
f
i
c
e
r
How We Worked: Data Analysis Subgroup
10 plenary meetings, half involving Census Bureau presentations
Dedicated data analysis subgroup was defining characteristic of this panel
Sworn/cleared to work behind Census Bureau IT firewall
Met twice a week (and, for several months, a third, biweekly session with Census Bureau staff)
All panel members sworn/cleared at level to facilitate discussions/deliberations via Census VDI
Only
figures/tables cleared by Census Disclosure Review Board (and subject to
rounding or noise infusion) were ported from behind firewall
4
Disclosure Review Board
Approval Numbers
The Census Bureau’s Disclosure Review Board and
Disclosure Avoidance Officers have reviewed the
information products used to produce several figures and
tables in this report for unauthorized disclosure of
confidential information and have approved the disclosure
avoidance practices applied to these releases.
By mutual agreement, Disclosure Review Board clearance
was sought and obtained only for the generation of
specific figures and tables published in this report, and
not for the issuance of the underlying data files as
standalone products or as addenda to this report.
5
CBDRB-FY23-0171 (Nonresponse Followup [NRFU]
resolutions by phase, figures)
CBDRB-FY23-0179 (race/ethnicity analysis)
CBDRB-FY23-0180 (age heaping analysis)
CBDRB-FY23-0197 (Master Address File [MAF]
development, figures)
CBDRB-FY23-0206 (correlation coefficients)
CBDRB-FY23-0214 (NRFU resolutions by phase, tables)
CBDRB-FY23-0215 (MAF development, tables)
CBDRB-FY23-0221 (Group Quarters data quality)
CBDRB-FY23-0224 (NRFU and Self-Response return rate
analysis by ventiles of American Community Survey
variables)
Main Messages:
Key Conclusions
About 2020
6
SEPTEMBER 2023 BRIEFING
Signature achievement of the 2020 Census was its very
completion under exceptionally difficult circumstances
7
Census Bureau’s focus on key innovation areas created the capability to react as well
as could be expected to unprecedented difficulties
Four innovation areas driving 2020 Census planning worked very well
Implement Internet Self-Response (and position it as modal response channel)
Shift precensus Address Canvassing work from fieldwork to in-office review of imagery, other data
Reengineer field management and case handling systems
Permit use of administrative records to enumerate some nonresponding households, for which reliable
records were available and at least one field visit was unsuccessful
Other helpful design features for meeting 2020 challenges included:
Careful planning and necessary modification of mailing/contact strategy
Reinstating Update Leave rather than pure reliance on Update Enumerate (reduced contacts during COVID)
Attention to Master Address File coverage and updating in preceding decade
No census is, or can be, perfect—and 2020 had its share
of data quality issues—
Age Heaping
8
Age heaping in 2020 much more severe than in 2010, and analyses suggest cause was
degradation in quality of proxy response in Nonresponse Follow-up
No census is, or can be, perfect—and 2020 had its share
of data quality issues—
Group Quarters
9
Group Quarters Enumeration is a challenge in any census, but difficulty made nearly
insurmountable by 2020 Census circumstances
•
COVID-19-related effects, delays, and
access restrictions hit GQs particularly
hard
–
Upended college student count, both on-
campus (GQ) and off-campus (cancelled
Early NRFU)
–
Nursing, health care facilities more restricted
•
Good idea to convert GQs to eResponse,
but problematic for college/university
student count, re Family Educational
Rights and Privacy Act (FERPA) restrict-
ions on only providing “directory info”
No census is, or can be, perfect—and 2020 had its share
of data quality issues—
Differential Net Undercount
10
Net percent undercoverage
with respect to race and ethnicity in 2020 consistent
with 2010 patterns but exacerbated in degree, for reasons not readily known
(Note:
Demographic Analysis estimates for Black and all other people not yet available)
No census is, or can be, perfect—and 2020 had its share
of data quality issues—
Missing People
11
Omissions decreased for White and Asian people in 2020, stayed at the same high
level for American Indians/Alaska Natives on reservations, and increased for Black
and Hispanic people.
Net Percent Undercoverage of
Black people
by age,
sex, and owner/renter status, 2020 and 2010 PES
12
Net Percent Undercoverage of
Hispanic people
by age,
sex, and owner/renter status, 2020 and 2010 PES
13
Item Imputation Rates by Response Mode, 2020
&
2010
14
Proxy responses: imputation rates for age/date of birth high in 2020 and 2010, but 2-3 times higher
In 2020 for race and ethnicity. Note high-ish rates of imputation for race and ethnicity for administrative
records enumerations in NRFU in 2020. If you can get a household member to respond (self or in NRFU),
data quality will be good.
Some Other Findings
R
E
S
P
O
N
S
E
M
O
D
E
S
H
o
u
s
e
h
o
l
d
s
i
n
t
h
e
p
o
o
r
e
s
t
c
e
n
s
u
s
t
r
a
c
t
s
l
e
a
s
t
l
i
k
e
l
y
a
n
d
h
o
u
s
e
h
o
l
d
s
i
n
c
e
n
s
u
s
t
r
a
c
t
s
w
i
t
h
t
h
e
l
a
r
g
e
s
t
s
i
z
e
h
o
u
s
i
n
g
u
n
i
t
s
(
#
r
o
o
m
s
)
m
o
s
t
l
i
k
e
l
y
t
o
s
e
l
f
-
r
e
s
p
o
n
d
H
o
u
s
e
h
o
l
d
s
i
n
t
h
e
p
o
o
r
e
s
t
c
e
n
s
u
s
t
r
a
c
t
s
l
e
a
s
t
l
i
k
e
l
y
a
n
d
h
o
u
s
e
h
o
l
d
s
i
n
c
e
n
s
u
s
t
r
a
c
t
s
w
i
t
h
t
h
e
m
o
s
t
p
r
e
v
a
l
e
n
t
b
r
o
a
d
b
a
n
d
&
h
i
g
h
e
s
t
m
e
d
i
a
n
i
n
c
o
m
e
s
m
o
s
t
l
i
k
e
l
y
t
o
s
e
l
f
-
r
e
s
p
o
n
d
b
y
i
n
t
e
r
n
e
t
H
o
u
s
e
h
o
l
d
s
i
n
c
e
n
s
u
s
t
r
a
c
t
s
w
i
t
h
t
h
e
m
o
s
t
o
w
n
e
r
u
n
i
t
s
l
e
a
s
t
l
i
k
e
l
y
a
n
d
h
o
u
s
e
h
o
l
d
s
i
n
c
e
n
s
u
s
t
r
a
c
t
s
w
i
t
h
t
h
e
m
o
s
t
r
e
n
t
e
r
u
n
i
t
s
m
o
s
t
l
i
k
e
l
y
t
o
b
e
e
n
u
m
e
r
a
t
e
d
b
y
p
r
o
x
y
I
n
t
e
r
n
e
t
s
e
l
f
-
r
e
s
p
o
n
d
e
r
s
m
o
s
t
p
r
e
v
a
l
e
n
t
a
m
o
n
g
A
s
i
a
n
h
o
u
s
e
h
o
l
d
s
,
f
o
l
l
o
w
e
d
b
y
t
w
o
o
r
m
o
r
e
r
a
c
e
s
,
W
h
i
t
e
a
l
o
n
e
,
s
o
m
e
o
t
h
e
r
r
a
c
e
a
l
o
n
e
,
H
i
s
p
a
n
i
c
,
B
l
a
c
k
a
l
o
n
e
,
N
H
O
P
I
A
l
o
n
e
,
A
I
A
N
a
l
o
n
e
R
A
C
E
W
R
I
T
E
-
I
N
S
I
n
t
e
r
n
e
t
r
e
s
p
o
n
s
e
s
p
r
o
d
u
c
e
d
4
t
i
m
e
s
a
s
m
a
n
y
w
r
i
t
e
-
i
n
r
a
c
e
r
e
s
p
o
n
s
e
s
(
r
e
l
a
t
i
v
e
t
o
c
h
e
c
k
-
b
o
x
-
o
n
l
y
r
e
s
p
o
n
s
e
s
)
a
s
p
a
p
e
r
a
n
d
N
R
F
U
h
o
u
s
e
h
o
l
d
m
e
m
b
e
r
r
e
s
p
o
n
s
e
s
—
i
m
p
l
i
c
a
t
i
o
n
s
f
o
r
b
i
g
c
h
a
n
g
e
s
i
n
r
a
c
e
r
e
p
o
r
t
i
n
g
i
n
2
0
2
0
a
m
o
n
g
d
i
f
f
e
r
e
n
t
g
r
o
u
p
s
(
b
e
c
a
u
s
e
u
s
e
o
f
i
n
t
e
r
n
e
t
v
a
r
i
e
d
a
c
r
o
s
s
g
r
o
u
p
s
)
15
Conclusion 11.2: Disclosure Avoidance
The decision of the U.S. Census Bureau to respond to the threats to confidentiality
protection at a very late date in 2020 Census planning with a new, more complex Disclosure
Avoidance System (DAS) using differential privacy-based algorithms went counter to long-
standing principles of decennial census planning.
The approach had not been tested in a census environment nor had the ability of the
algorithms to handle critical user data needs been assessed; the Census Bureau had no
backup plan should implementation of the new DAS prove challenging.
T
h
e
d
e
c
i
s
i
o
n
t
o
c
o
n
t
i
n
u
e
t
o
d
e
p
l
o
y
t
h
e
n
e
w
D
A
S
i
n
t
h
e
f
a
c
e
o
f
s
e
r
i
o
u
s
i
m
p
l
e
m
e
n
t
a
t
i
o
n
p
r
o
b
l
e
m
s
r
e
s
u
l
t
e
d
i
n
m
a
r
k
e
d
d
e
l
a
y
s
i
n
d
e
l
i
v
e
r
y
o
f
d
a
t
a
p
r
o
d
u
c
t
s
,
w
i
t
h
s
o
m
e
v
a
r
i
a
b
l
e
s
a
n
d
t
y
p
e
s
o
f
g
e
o
g
r
a
p
h
i
c
u
n
i
t
s
o
f
q
u
e
s
t
i
o
n
a
b
l
e
u
t
i
l
i
t
y
,
a
n
d
o
t
h
e
r
s
n
o
t
p
r
o
v
i
d
e
d
a
t
a
l
l
.
[
T
o
o
l
s
t
o
h
e
l
p
u
s
e
r
s
u
n
d
e
r
s
t
a
n
d
a
n
d
a
n
a
l
y
z
e
t
h
e
d
a
t
a
a
r
e
a
l
s
o
p
r
e
t
t
y
m
u
c
h
l
a
c
k
i
n
g
.
]
In addition, it is not clear that the chosen privacy budgets for the various 2020 Census data
products, with high values of the 𝜀 parameter that trades off accuracy with confidentiality
protection, provide much actual protection.
16
Corollaries on Disclosure Avoidance System (DAS)
We do not assess differential privacy in general, agreeing that it has desirable
theoretical properties and is useful for some types of applications
We strongly agree that confidentiality protection is a vitally important responsibility for
the decennial census, and for statistical agencies in general
We find fault in the particular implementation for the 2020 Census, done without
prototyping and testing and without adequate engagement of the user/stakeholder
community — counter to the careful implementation of other major changes in census
methodology
The same criticisms attach to other major changes made in haste and without proper
development—which is why we say in Conclusion 8.2 that major motion toward an
administrative records-based census in 2030 is not feasible
17
Main Messages:
Recommendations
for 2030 and Beyond
18
SEPTEMBER 2023 BRIEFING
Priority Goals for 2030 Census Testing and Development
(Recommendation 12.1)
Like predecessor Panel to Review the 2010 Census, we recommend that the Census Bureau
focus primary attention on a small, manageable number of major innovation areas:
1.
Maximize self-response to the census, including better matching of contact and communication strategies to
the desired response mode
2.
Improve the quality of the data collection in NRFU, including reduction if not elimination of proxy reporting
when a good alternative is available
3.
R
e
d
u
c
e
g
a
p
s
i
n
c
o
v
e
r
a
g
e
a
n
d
d
a
t
a
q
u
a
l
i
t
y
a
s
s
o
c
i
a
t
e
d
w
i
t
h
r
a
c
e
,
e
t
h
n
i
c
i
t
y
,
a
n
d
s
o
c
i
o
e
c
o
n
o
m
i
c
s
t
a
t
u
s
4.
Improve the quality of address listings and contact strategies for conventional housing units and group
quarters alike
5.
R
e
a
l
i
g
n
b
a
l
a
n
c
e
b
e
t
w
e
e
n
u
t
i
l
i
t
y
,
t
i
m
e
l
i
n
e
s
s
,
a
n
d
c
o
n
f
i
d
e
n
t
i
a
l
i
t
y
p
r
o
t
e
c
t
i
o
n
i
n
2
0
3
0
d
a
t
a
p
r
o
d
u
c
t
s
Support improvements to census law
P
e
n
a
l
t
i
e
s
f
o
r
d
e
l
i
b
e
r
a
t
e
m
i
s
u
s
e
o
f
d
a
t
a
;
r
e
s
o
l
u
t
i
o
n
o
f
F
E
R
P
A
“
d
i
r
e
c
t
o
r
y
i
n
f
o
r
m
a
t
i
o
n
”
l
i
m
i
t
a
t
i
o
n
19
Implications/Suggestions for APDU
APDU members will have their own interests in particular aspects of the census, such as the
undercount for particular groups
APDU members collectively (I hope) will lobby for genuine interactive dialogue with Census
Bureau staff in charge of 2030 data products
My suggested strategy:
1.
M
a
k
e
u
s
e
r
r
e
q
u
i
r
e
m
e
n
t
s
c
l
e
a
r
—
t
i
m
e
l
i
n
e
s
s
,
u
s
a
b
i
l
i
t
y
,
“
j
o
i
n
”
v
a
r
i
a
b
l
e
s
(
e
.
g
.
,
p
e
r
s
o
n
s
p
e
r
o
c
c
u
p
i
e
d
u
n
i
t
)
2.
Meet Census Bureau partway on confidentiality concerns—say, agree to reduction of data for blocks (e.g.,
do we need race by every other race for people checking 4 and 5 races [~220K total]); work with Census
Bureau to reduce blocks with small populations (median block had 28 people in 2020)
3.
With these kinds of changes, whatever method Census uses for protection (DP, swapping, et al.) should be
much easier to implement and have much less of an adverse impact on quality
Bottom Line: 2030 Census data user working group needs to be established with Census
blessing ASAP, operational plan developed by 2027, and implementation tested in 2028, to
achieve timely, useful data in 2030!
20
Assessing the 2020 Census:
Final Report
Consensus Study
Committee on National Statistics (CNSTAT)
Released October 3,
2023
Assessing the 2020 Census: Final Report | The
National Academies Press
(www.nap.edu)
ASSESSING THE 2020 CENSUS
Thank You
ccitro@nas.edu
Please find training slides at the end of this template.
The 2020 Census quality data report was assessed by an ad hoc panel appointed by The National Academies of Sciences, Engineering, and Medicine. The panel reviewed data collected, process measures, demographic analysis, and more to evaluate the quality of the census data. Recommendations were made for further research by the Census Bureau for the 2030 Census.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Assessing the 2020 Census: Final Report Consensus Study Committee on National Statistics (CNSTAT) Released October 3, 2023 Assessing the 2020 Census: Final Report | The National Academies Press (www.nap.edu) ASSESSING THE 2020 CENSUS
Statement of Task The National Academies of Sciences, Engineering, and Medicine will appoint an ad hoc panel to review the quality of the data that were collected in the 2020 Census. As part of its work, the panel will: 1. Review information from the Census Bureau on the data collected as well as various process measures and indicators of data quality obtained as part of the 2020 Census operations; 2. Review other available information, such as results from demographic analysis, process measures and preliminary results from the post-enumeration survey; and analyses of administrative records; and 3. Consider the results from evaluations of similar indicators from the 2010 and 2000 Censuses. The panel will produce a final report that includes conclusions about the quality of the data collected in the 2020 Census and makes recommendations for further research by the Census Bureau to evaluate the quality of the 2020 data and to begin planning the 2030 Census. 2
Panel to Evaluate the Quality of the 2020 Census Teresa A. Sullivan (Chair), Department of Sociology, University of Virginia C. Matthew Snipp, School of Humanities and Sciences, Stanford University Margo Anderson, Department of History, University of Wisconsin Milwaukee (emerita) Edward Telles, Department of Sociology, University of California, Irvine Robert M. Bell,* Google and AT&T Labs (retired) Wendy Underhill, National Conference of State Legislatures, Denver Kathryn Edin (NAS), Department of Sociology, Princeton University David Van Riper,* Minnesota Population Center, University of Minnesota Marc Hamel, Statistics Canada (retired) * Denotes member of the panel s designated data analytic subgroup. George T. Ligler (NAE), Department of Multidisciplinary Engineering, Texas A&M University Thomas A. Louis,* Department of Biostatistics, Johns Hopkins University (emeritus) Daniel L. Cork, Study Director Lloyd B. Potter,* Texas Demographic Center, University of Texas at San Antonio Constance F. Citro, Senior Scholar Michael L. Cohen, Senior Program Officer Joseph J. Salvo,* University of Virginia Biocomplexity Institute Anthony Mann, Senior Program Associate Katrina Baum Stone, Senior Program Officer Regina Shih, Department of Epidemiology, Rollins School of Public Health, Emory University 3
How We Worked: Data Analysis Subgroup 10 plenary meetings, half involving Census Bureau presentations Dedicated data analysis subgroup was defining characteristic of this panel Sworn/cleared to work behind Census Bureau IT firewall Met twice a week (and, for several months, a third, biweekly session with Census Bureau staff) All panel members sworn/cleared at level to facilitate discussions/deliberations via Census VDI Only figures/tables cleared by Census Disclosure Review Board (and subject to rounding or noise infusion) were ported from behind firewall 4
Disclosure Review Board Approval Numbers CBDRB-FY23-0171 (Nonresponse Followup [NRFU] resolutions by phase, figures) CBDRB-FY23-0179 (race/ethnicity analysis) The Census Bureau s Disclosure Review Board and Disclosure Avoidance Officers have reviewed the information products used to produce several figures and tables in this report for unauthorized disclosure of confidential information and have approved the disclosure avoidance practices applied to these releases. CBDRB-FY23-0180 (age heaping analysis) CBDRB-FY23-0197 (Master Address File [MAF] development, figures) CBDRB-FY23-0206 (correlation coefficients) CBDRB-FY23-0214 (NRFU resolutions by phase, tables) By mutual agreement, Disclosure Review Board clearance was sought and obtained only for the generation of specific figures and tables published in this report, and not for the issuance of the underlying data files as standalone products or as addenda to this report. CBDRB-FY23-0215 (MAF development, tables) CBDRB-FY23-0221 (Group Quarters data quality) CBDRB-FY23-0224 (NRFU and Self-Response return rate analysis by ventiles of American Community Survey variables) 5
SEPTEMBER 2023 BRIEFING Main Messages: Key Conclusions About 2020 6
Signature achievement of the 2020 Census was its very completion under exceptionally difficult circumstances Census Bureau s focus on key innovation areas created the capability to react as well as could be expected to unprecedented difficulties Four innovation areas driving 2020 Census planning worked very well Implement Internet Self-Response (and position it as modal response channel) Shift precensus Address Canvassing work from fieldwork to in-office review of imagery, other data Reengineer field management and case handling systems Permit use of administrative records to enumerate some nonresponding households, for which reliable records were available and at least one field visit was unsuccessful Other helpful design features for meeting 2020 challenges included: Careful planning and necessary modification of mailing/contact strategy Reinstating Update Leave rather than pure reliance on Update Enumerate (reduced contacts during COVID) Attention to Master Address File coverage and updating in preceding decade 7
No census is, or can be, perfectand 2020 had its share of data quality issues Age Heaping Age heaping in 2020 much more severe than in 2010, and analyses suggest cause was degradation in quality of proxy response in Nonresponse Follow-up NRFU, Proxy Respondents 25.00 20.00 Estimated Heaping Metric Total Population 15.00 10.00 2.00 1.50 1.00 5.00 0.50 0.00 0.00 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 -0.50 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 -1.00 -5.00 Age in Census Edited File 2010 2020 -10.00 Age in Census Edited File 2010 2020 8
No census is, or can be, perfectand 2020 had its share of data quality issues Group Quarters Group Quarters Enumeration is a challenge in any census, but difficulty made nearly insurmountable by 2020 Census circumstances COVID-19-related effects, delays, and access restrictions hit GQs particularly hard 45% GQ Imputation Rates 50% Upended college student count, both on- campus (GQ) and off-campus (cancelled Early NRFU) 40% 35% 30% 25% Nursing, health care facilities more restricted 20% 15% Good idea to convert GQs to eResponse, but problematic for college/university student count, re Family Educational Rights and Privacy Act (FERPA) restrict- ions on only providing directory info 10% 5% 0% Sex Age/DOB Ethnicity Race 2020 Census 2010 Census 9
No census is, or can be, perfectand 2020 had its share of data quality issues Differential Net Undercount Net percent undercoverage with respect to race and ethnicity in 2020 consistent with 2010 patterns but exacerbated in degree, for reasons not readily known (Note: Demographic Analysis estimates for Black and all other people not yet available) 4 3 2 1 0 White Non- Hispanic Alone Black AOIC Asian AOIC AIAN AOIC on Reservations Hispanic -1 -2 -3 -4 -5 -6 -7 2020 PES 2010 PES 10
No census is, or can be, perfectand 2020 had its share of data quality issues Missing People Omissions decreased for White and Asian people in 2020, stayed at the same high level for American Indians/Alaska Natives on reservations, and increased for Black and Hispanic people. Percent P-Sample Omissions 16 14 12 10 8 6 4 2 0 White Non- Hispanic Alone Black AOIC Asian AOIC AIAN AOIC on Reservations Hispanic 2020 PES 2010 PES 11
Net Percent Undercoverage of Black people by age, sex, and owner/renter status, 2020 and 2010 PES Black Percent Net Undercount 6 4 2 0 -2 -4 -6 -8 -10 -12 -14 2020 PES 2010 PES 12
Net Percent Undercoverage of Hispanic people by age, sex, and owner/renter status, 2020 and 2010 PES Hispanic Percent Net Undercount 6 4 2 0 -2 -4 -6 -8 -10 -12 -14 2020 PES 2010 PES 13
Item Imputation Rates by Response Mode, 2020 & 2010 2020 2010 70% 60% 60% 50% 50% 40% 40% 30% 30% 20% 20% 10% 10% 0% 0% Self NRFU Household NRFU Proxy NRFU Ad Records Self NRFU Household NRFU Proxy Age/DOB Ethnicity Race Age/DOB Ethnicity Race Proxy responses: imputation rates for age/date of birth high in 2020 and 2010, but 2-3 times higher In 2020 for race and ethnicity. Note high-ish rates of imputation for race and ethnicity for administrative records enumerations in NRFU in 2020. If you can get a household member to respond (self or in NRFU), data quality will be good. 14
Some Other Findings RESPONSE MODES Households in the poorest census tracts least likely and households in census tracts with the largest size housing units (# rooms) most likely to self-respond Households in the poorest census tracts least likely and households in census tracts with the most prevalent broadband & highest median incomes most likely to self-respond by internet Households in census tracts with the most owner units least likely and households in census tracts with the most renter units most likely to be enumerated by proxy Internet self-responders most prevalent among Asian households, followed by two or more races, White alone, some other race alone, Hispanic, Black alone, NHOPI Alone, AIAN alone RACE WRITE-INS Internet responses produced 4 times as many write-in race responses (relative to check-box-only responses) as paper and NRFU household member responses implications for big changes in race reporting in 2020 among different groups (because use of internet varied across groups) 15
Conclusion 11.2: Disclosure Avoidance The decision of the U.S. Census Bureau to respond to the threats to confidentiality protection at a very late date in 2020 Census planning with a new, more complex Disclosure Avoidance System (DAS) using differential privacy-based algorithms went counter to long- standing principles of decennial census planning. The approach had not been tested in a census environment nor had the ability of the algorithms to handle critical user data needs been assessed; the Census Bureau had no backup plan should implementation of the new DAS prove challenging. The decision to continue to deploy the new DAS in the face of serious implementation problems resulted in marked delays in delivery of data products, with some variables and types of geographic units of questionable utility, and others not provided at all. [Tools to help users understand and analyze the data are also pretty much lacking.] In addition, it is not clear that the chosen privacy budgets for the various 2020 Census data products, with high values of the ? parameter that trades off accuracy with confidentiality protection, provide much actual protection. 16
Corollaries on Disclosure Avoidance System (DAS) We do not assess differential privacy in general, agreeing that it has desirable theoretical properties and is useful for some types of applications We strongly agree that confidentiality protection is a vitally important responsibility for the decennial census, and for statistical agencies in general We find fault in the particular implementation for the 2020 Census, done without prototyping and testing and without adequate engagement of the user/stakeholder community counter to the careful implementation of other major changes in census methodology The same criticisms attach to other major changes made in haste and without proper development which is why we say in Conclusion 8.2 that major motion toward an administrative records-based census in 2030 is not feasible 17
SEPTEMBER 2023 BRIEFING Main Messages: Recommendations for 2030 and Beyond 18
Priority Goals for 2030 Census Testing and Development (Recommendation 12.1) Like predecessor Panel to Review the 2010 Census, we recommend that the Census Bureau focus primary attention on a small, manageable number of major innovation areas: 1. Maximize self-response to the census, including better matching of contact and communication strategies to the desired response mode 2. Improve the quality of the data collection in NRFU, including reduction if not elimination of proxy reporting when a good alternative is available 3. Reduce gaps in coverage and data quality associated with race, ethnicity, and socioeconomic status 4. Improve the quality of address listings and contact strategies for conventional housing units and group quarters alike 5. Realign balance between utility, timeliness, and confidentiality protection in 2030 data products Support improvements to census law Penalties for deliberate misuse of data; resolution of FERPA directory information limitation 19
Implications/Suggestions for APDU APDU members will have their own interests in particular aspects of the census, such as the undercount for particular groups APDU members collectively (I hope) will lobby for genuine interactive dialogue with Census Bureau staff in charge of 2030 data products My suggested strategy: 1. Make user requirements clear timeliness, usability, join variables (e.g., persons per occupied unit) 2. Meet Census Bureau partway on confidentiality concerns say, agree to reduction of data for blocks (e.g., do we need race by every other race for people checking 4 and 5 races [~220K total]); work with Census Bureau to reduce blocks with small populations (median block had 28 people in 2020) 3. With these kinds of changes, whatever method Census uses for protection (DP, swapping, et al.) should be much easier to implement and have much less of an adverse impact on quality Bottom Line: 2030 Census data user working group needs to be established with Census blessing ASAP, operational plan developed by 2027, and implementation tested in 2028, to achieve timely, useful data in 2030! 20
Assessing the 2020 Census: Final Report Consensus Study Committee on National Statistics (CNSTAT) Released October 3, 2023 Assessing the 2020 Census: Final Report | The National Academies Press (www.nap.edu) Thank You ccitro@nas.edu ASSESSING THE 2020 CENSUS
