Generalizing Research on Older Adults in Seattle Integrated Health System
Generalizing research based in a
Seattle integrated health delivery
system to all older adults in the region
Laura Gibbons, PhD
University of Washington
gibbonsL@uw.edu
M
o
t
i
v
a
t
i
o
n
•
Invitation to participate in the Adult Changes in Thought (ACT)
study is sent to a randomly-selected group of Kaiser Permanente
Washington members aged 65+.
•
Assessed for dementia and join study if not demented.
•
More of a community-based sample than most research on older
adults.
•
How well do the ACT participants enrolled from 1994-2020
compare to the current Seattle area?
•
Will our findings generalize to all older adults currently in our
region?
•
“Transport” study population -> target population
S
t
r
a
t
e
g
y
•
Use the 1158 people age 65+ in the 2019
Behavioral Risk
Factor Surveillance System (BRFSS), which has survey
weights that allow the 1158 respondents in the Seattle
Metropolitan Region to represent over 400,000 people age
65+ in our entire region.
•
To compare the samples, find measures available in both the
BRFSS and ACT. Demographics, self-reported health, self-
reported ADLs.
S
t
r
a
t
e
g
y
•
Construct weights which make ACT look more like the Seattle
Metropolitan Region (BRFSS).
•
Demonstrate the use of these weights in transporting the
prevalence of common eye diseases and also their effect on
Alzheimer’s disease incidence.
Stata files
https://github.com/gibbonsL/ACT_transport
(overview in
Do_file_directory.do
in GitHub)
•
Code for data preparation and the alignment of the 2
datasets is in that folder.
H
a
r
m
o
n
i
z
e
d
m
e
a
s
u
r
e
s
•
ACT baseline (1994 to 2020) and 2019 BRFSS
•
Demographics:
age (grouped), sex, education, race, Hispanic
ethnicity, marital status, employment status.
•
Health (self-report):
smoking, current hypertension medication,
current cholesterol medication, history of heart disease, stroke,
asthma,
COPD or emphysema, diabetes, non-melanoma cancer,
or osteoarthritis, self-perceived general health
.
•
ADLs (self-report):
any difficulty dressing or bathing, any
difficulty walking or climbing stairs.
M
i
s
s
i
n
g
d
a
t
a
•
Cumulatively 12% of BRFSS and 15% of ACT participants
were missing at least one variable.
•
Multiple imputation analyses (40 imputations) so that
we could make weights for all participants.
impute_missing_data_by_group.do
mi register regular id act ageg5yr sex xmmsawt // no missingness
mi register imputed ed3 race3 hispanic marry work smoke bpcurr
cholcurr mi heart_dis stroke asthma copd diabetes cancer osteoa
ratehealth diffdres diffwalk // variables with missing data
mi impute chained
///
(ologit) ed3 smoke ratehealth ///
(mlogit) race3 marry work ///
(logit) hispanic bpcurr cholcurr mi heart_dis stroke
asthma copd diabetes cancer osteoa diffdres diffwalk ///
= ageg5yr sex ///
, add(40) rseed(2466755) force
by(act)
augment
// augment finesses perfect prediction of something in BRFSS
P
a
r
t
i
c
i
p
a
t
i
o
n
w
e
i
g
h
t
s
•
Weights: Winsorized, s
tabilized inverse odds of participation
for all ACT participants.
weights_with_imputed_data.do
mi svyset
[pweight=
xmmsawt
]
mi estimate
, saving(
temp3
, replace):
svy:
logit act i.ageg5yr sex i.ed3 i.race3 hispanic i.marry
i.work i.smoke bpcurr cholcurr mi heart_dis stroke
asthma copd diabetes osteoa ratehealth diffdres
diffwalk
*
default is linear prediction, not probability of a positive outcome
mi predict xbbase using
temp3
gen prbase=exp(xbbase)/(1+exp(xbbase)
weights_with_imputed_data.do
mi svyset
[pweight=
xmmsawt
]
mi estimate
, saving(
temp3
, replace):
svy:
logit act i.ageg5yr sex i.ed3 i.race3 hispanic i.marry
i.work i.smoke bpcurr cholcurr mi heart_dis stroke
asthma copd diabetes osteoa ratehealth diffdres
diffwalk
*
default is linear prediction, not probability of a positive outcome
mi predict xbbase using
temp3
gen prbase=exp(xbbase)/(1+exp(xbbase))
weights_with_imputed_data.do
, continued
mi unset
// no longer using mi - only wanted prbase on everyone
gen iowbase=(1-prbase)/prbase
// odds of being in ACT
gen pract=5762/431701.1
// numerical probability of being in ACT.
*stabilized inverse odds weight is IOW * odds of being in ACT:
gen siowbase=iowbase*pract/(1-pract)
*Winsorize….
wsiowbase (Not standardized)
A
C
T
b
a
s
e
l
i
n
e
f
a
i
r
l
y
s
i
m
i
l
a
r
t
o
t
h
e
2
0
1
9
B
R
F
S
S
Without ACT participation weights
•
More people with < HS education (cohort effect)
•
More Whites
•
More retired
•
More past smokers
•
Less likely to be on meds for hypertension and cholesterol
•
Better self-reported health
•
More likely to have difficulty walking or climbing stairs
Cholest.
meds
Heart
Dis.
HS ed
Standardized mean differences
:
Mean in ACT minus the mean in
BRFSS, divided by the SD in BRFSS.
A positive value means greater
frequency in ACT.
Ideally all values between +/- 0.10.
(For variables with more than 2 levels, each
level is shown compared to all others.)
Less than HS ed
Cholest.
Meds
HS ed
Heart Dis.
ADL Walking/stairs
P
a
r
t
i
c
i
p
a
t
i
o
n
w
e
i
g
h
t
s
Secular trend in education. Early ACT enrollees account for most of the < HS.
Recent ACT enrollees have a higher level of education than in 2019 BRFSS.
Note that we can get these further aligned with interaction terms in the model
for the weights, but multiple imputation with interaction terms in Stata is biased,
even/especially with –mi passive –. (Blimp)
Prevalence of common eye diseases
eye_disease_prev.do
•
Four common eye diseases from electronic medical records.
•
drtodate = history of diabetic retinopathy.
proportion drtodate // to get CI;
unweighted
svyset [pweight=
wsiowbase
]
svy: tab drtodate, col //
weighted
S
e
l
e
c
t
e
d
e
y
e
d
i
s
e
a
s
e
f
i
n
d
i
n
g
s
T
r
a
n
s
p
o
r
t
r
e
s
u
l
t
s
t
h
a
t
c
a
n
o
n
l
y
b
e
o
b
t
a
i
n
e
d
i
n
A
C
T
t
o
a
l
l
o
l
d
e
r
a
d
u
l
t
s
i
n
t
h
e
S
e
a
t
t
l
e
M
e
t
r
o
p
o
l
i
t
a
n
R
e
g
i
o
n
.
longitudinal_eye_example.do
// AD risk
To get weighted point estimates:
stset surv_age [
pweight=
wsiowbase
],
failure(anyad) origin(time 0)
enter(age_bl) id(subj)
Then run the regular Cox models.
Baseline eye disease history -> AD
Hazard ratios (HR) and 95% confidence intervals (CI) for Alzheimer’s disease in s
eparate Cox models
for time-dependent eye disease status within the ACT study (unweighted) and transported to the
SeattleMR (weighted, but these are
NOT
the bootstrapped CI).
All models adjusted for
baseline age, years of education, and smoking history.
To meet proportional
hazards assumptions, we allowed the baseline hazards to differ by gender and by the presence of
any
APOE
4 alleles. Age is the time axis.
bootstrap_CI.do
•
For the weighted analyses, CIs were bootstrapped
•
Error in estimating the weights
•
Rubin’s rules for combining variance across imputations
.
1)
Make 40 datasets, in the first:
a)
Keep complete data and _1_*
b)
Renaming _1_ed3 to ed3 , etc, for all variables
2) Merge in eye data
bootstrap_CI.do
, continued
3
) Bootstrap 500 times in each of the 40 datasets
1.
Estimate the weights
2.
Do the analyses
3.
Output the desired parameters
4.
Manipulate those parameters to the form we want
5.
Append the 40 x 500 results
6.
Take the 2.5
th
and 97.5
th
percentile.
4) The resulting CI are wider than un-bootstrapped, the price we pay.
S
e
n
s
i
t
i
v
i
t
y
a
n
a
l
y
s
e
s
I calculated new weights, excluding ACT participants born before 1920
and thus not eligible to be in BRFSS 2019.
•
The overall rate of cataract history was even higher.
•
Diabetic retinopathy
predicting
AD: weighted and unweighted more
similar than in full sample
.
I also computed new weights omitting the
ADL variables because ADLs
could possibly be on the causal pathway from eye disease to AD, but
the AD results were almost identical.
D
i
s
c
u
s
s
i
o
n
•
On variables measured in both groups, differences between those in
ACT and the general population of older adults in the region were
mostly small in magnitude. (Only education and cholesterol meds >
+/-0.25.)
•
Still might affect the generalizability of some research findings.
L
i
m
i
t
a
t
i
o
n
s
•
Can’t account for differences not measured in both studies.
•
Secular trends in education, medical care.
•
ACT had health insurance now and (mostly) in the past.
BRFSS eligible for Medicare, but may have lesser coverage, or
earlier health care may not have been as good.
W
h
e
n
d
o
w
e
h
a
v
e
t
o
d
o
t
h
i
s
?
•
Prevalence estimates for sure.
•
Causal questions:
If we suspect that the relationship may be
confounded or moderated by
measured
differences between
the study sample and the target population.
•
Future studies should gather data in a way that aligns with
variables collected in population-representative surveys so
that better participation weights can be constructed.
•
As more research is transported, we will have a better sense
of how often and for what kind of questions this matters.
C
o
n
c
l
u
s
i
o
n
s
•
Generalizing research on older adults to the target
population may require the construction of analytic weights.
(As well as careful consideration of secular trends)
Thank you
Coauthors:
•
UCLA
transport methods
experts Taylor Mobley, Elizabeth Rose Mayeda, and
Eleanor Hayes-Larson
•
UW Eye expert Cecilia S. Lee
•
ACT MPIs
Andrea LaCroix, Linda McEvoy, and Paul
K. Crane
Funding:
U19AG066567, U01AG006781,
R56AG069126
, P50AG05136,
R01AG060942,
5R13AG030995.
Selected questions from the talk
Q: Did you use the BRFSS survey weights in the multiple imputation?
•
I did not, based on the advice of my transport experts.
•
I later took a missing data short course and the instructor thought I should.
•
However, I don’t think this is an option in Stata and I have not been able to
find existing software that can handle my particular data. Let me know if you
have ideas!
Q: Why did you impute first and then bootstrap in that data, rather
than the other way around?
•
Only because it’s quicker to run. See Schomaker & Heumann
(
https://pubmed.ncbi.nlm.nih.gov/29682776/
) for more on this. They find
both orders to be valid.
gibbonsL@uw.edu
(gibbonsL not gibbons1).
https://github.com/gibbonsl/ACT_transport
This research project led by Laura Gibbons focuses on generalizing findings from the Adult Changes in Thought (ACT) study in a Seattle integrated health delivery system to all older adults in the region. By comparing ACT participants with the current Seattle area population and using survey weights from the Behavioral Risk Factor Surveillance System, the study aims to determine the generalizability of the research findings across the region. Strategies include constructing weights to align ACT with the broader region and addressing missing data through multiple imputation analyses.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Generalizing research based in a Seattle integrated health delivery system to all older adults in the region Laura Gibbons, PhD University of Washington gibbonsL@uw.edu
Motivation Motivation Invitation to participate in the Adult Changes in Thought (ACT) study is sent to a randomly-selected group of Kaiser Permanente Washington members aged 65+. Assessed for dementia and join study if not demented. More of a community-based sample than most research on older adults. How well do the ACT participants enrolled from 1994-2020 compare to the current Seattle area? Will our findings generalize to all older adults currently in our region? Transport study population -> target population
Strategy Strategy Use the 1158 people age 65+ in the 2019 Behavioral Risk Factor Surveillance System (BRFSS), which has survey weights that allow the 1158 respondents in the Seattle Metropolitan Region to represent over 400,000 people age 65+ in our entire region. To compare the samples, find measures available in both the BRFSS and ACT. Demographics, self-reported health, self- reported ADLs.
Strategy Strategy Construct weights which make ACT look more like the Seattle Metropolitan Region (BRFSS). Demonstrate the use of these weights in transporting the prevalence of common eye diseases and also their effect on Alzheimer s disease incidence.
Stata files https://github.com/gibbonsL/ACT_transport (overview in Do_file_directory.do in GitHub) Code for data preparation and the alignment of the 2 datasets is in that folder.
Harmonized measures Harmonized measures ACT baseline (1994 to 2020) and 2019 BRFSS Demographics: age (grouped), sex, education, race, Hispanic ethnicity, marital status, employment status. Health (self-report): smoking, current hypertension medication, current cholesterol medication, history of heart disease, stroke, asthma, COPD or emphysema, diabetes, non-melanoma cancer, or osteoarthritis, self-perceived general health. ADLs (self-report): any difficulty dressing or bathing, any difficulty walking or climbing stairs.
Missing data Missing data Cumulatively 12% of BRFSS and 15% of ACT participants were missing at least one variable. Multiple imputation analyses (40 imputations) so that we could make weights for all participants.
impute_missing_data_by_group.do mi register regular id act ageg5yr sex xmmsawt // no missingness mi register imputed ed3 race3 hispanic marry work smoke bpcurr cholcurr mi heart_dis stroke asthma copd diabetes cancer osteoa ratehealth diffdres diffwalk // variables with missing data mi impute chained (ologit) ed3 smoke ratehealth (mlogit) race3 marry work /// (logit) hispanic bpcurr cholcurr mi heart_dis stroke asthma copd diabetes cancer osteoa diffdres diffwalk = ageg5yr sex /// , add(40) rseed(2466755) force by(act) augment // augment finesses perfect prediction of something in BRFSS /// /// ///
Participation weights Participation weights Weights: Winsorized, stabilized inverse odds of participation for all ACT participants.
weights_with_imputed_data.do mi svyset [pweight=xmmsawt] mi estimate, saving(temp3, replace): svy: logit act i.ageg5yr sex i.ed3 i.race3 hispanic i.marry i.work i.smoke bpcurr cholcurr mi heart_dis stroke asthma copd diabetes osteoa ratehealth diffdres diffwalk * default is linear prediction, not probability of a positive outcome mi predict xbbase using temp3 gen prbase=exp(xbbase)/(1+exp(xbbase)
weights_with_imputed_data.do mi svyset [pweight=xmmsawt] mi estimate, saving(temp3, replace): svy: logit act i.ageg5yr sex i.ed3 i.race3 hispanic i.marry i.work i.smoke bpcurr cholcurr mi heart_dis stroke asthma copd diabetes osteoa ratehealth diffdres diffwalk * default is linear prediction, not probability of a positive outcome mi predict xbbase using temp3 gen prbase=exp(xbbase)/(1+exp(xbbase))
weights_with_imputed_data.do, continued mi unset // no longer using mi - only wanted prbase on everyone gen iowbase=(1-prbase)/prbase // odds of being in ACT gen pract=5762/431701.1 // numerical probability of being in ACT. *stabilized inverse odds weight is IOW * odds of being in ACT: gen siowbase=iowbase*pract/(1-pract) *Winsorize . wsiowbase (Not standardized)
ACT baseline fairly similar to the 2019 BRFSS ACT baseline fairly similar to the 2019 BRFSS Without ACT participation weights More people with < HS education (cohort effect) More Whites More retired More past smokers Less likely to be on meds for hypertension and cholesterol Better self-reported health More likely to have difficulty walking or climbing stairs
Less than HS ed HS ed HS ed Standardized mean differences: Mean in ACT minus the mean in BRFSS, divided by the SD in BRFSS. A positive value means greater frequency in ACT. Cholest. meds Heart Dis. Heart Dis. Cholest. Meds Ideally all values between +/- 0.10. (For variables with more than 2 levels, each level is shown compared to all others.) ADL Walking/stairs
Participation weights Participation weights Unweighted ACT 8% Weighted ACT 1.3% Education level Less than high school BRFSS 1.9% High school College degree or higher 42% 49% 51% 48% 55% 43% Secular trend in education. Early ACT enrollees account for most of the < HS. Recent ACT enrollees have a higher level of education than in 2019 BRFSS. Note that we can get these further aligned with interaction terms in the model for the weights, but multiple imputation with interaction terms in Stata is biased, even/especially with mi passive . (Blimp)
Prevalence of common eye diseases eye_disease_prev.do Four common eye diseases from electronic medical records. drtodate = history of diabetic retinopathy. proportion drtodate // to get CI; unweighted svyset [pweight=wsiowbase] svy: tab drtodate, col // weighted
Selected eye disease findings Selected eye disease findings Transport results that can only be obtained in ACT to all older adults in the Seattle Metropolitan Region. Eye disease (EMR at baseline) Total Diabetic retinopathy ACT study (unweighted) Transported to the SeattleMR (weighted) 4.1% 5.6% Cataract history ACT study (unweighted) Transported to the SeattleMR (weighted) 48.6% 51.8%
longitudinal_eye_example.do // AD risk To get weighted point estimates: stset surv_age [pweight=wsiowbase], failure(anyad) origin(time 0) enter(age_bl) id(subj) Then run the regular Cox models.
Baseline eye disease history -> AD Hazard ratios (HR) and 95% confidence intervals (CI) for Alzheimer s disease in separate Cox models for time-dependent eye disease status within the ACT study (unweighted) and transported to the SeattleMR (weighted, but these are NOT the bootstrapped CI). All models adjusted for baseline age, years of education, and smoking history. To meet proportional hazards assumptions, we allowed the baseline hazards to differ by gender and by the presence of any APOE 4 alleles. Age is the time axis. Years with diagnosis Diabetic retinopathy HR (95% CI) p-value ACT study (unweighted) >0 5 >5 1.32 (0.87, 2.00) 1.39 (1.02, 1.88) 0.187 0.037 Transported to the SeattleMR (weighted) (but these are not the bootstrapped CI) >0 5 >5 1.84 (1.08, 3.14) 1.29 (0.87, 1.91) 0.025 0.201
bootstrap_CI.do For the weighted analyses, CIs were bootstrapped Error in estimating the weights Rubin s rules for combining variance across imputations. 1) Make 40 datasets, in the first: a) Keep complete data and _1_* b) Renaming _1_ed3 to ed3 , etc, for all variables 2) Merge in eye data
bootstrap_CI.do, continued 3) Bootstrap 500 times in each of the 40 datasets 1. Estimate the weights 2. Do the analyses 3. Output the desired parameters 4. Manipulate those parameters to the form we want 5. Append the 40 x 500 results 6. Take the 2.5thand 97.5thpercentile. 4) The resulting CI are wider than un-bootstrapped, the price we pay.
Sensitivity analyses Sensitivity analyses I calculated new weights, excluding ACT participants born before 1920 and thus not eligible to be in BRFSS 2019. The overall rate of cataract history was even higher. Diabetic retinopathy predicting AD: weighted and unweighted more similar than in full sample. I also computed new weights omitting the ADL variables because ADLs could possibly be on the causal pathway from eye disease to AD, but the AD results were almost identical.
Discussion Discussion On variables measured in both groups, differences between those in ACT and the general population of older adults in the region were mostly small in magnitude. (Only education and cholesterol meds > +/-0.25.) Still might affect the generalizability of some research findings.
Limitations Limitations Can t account for differences not measured in both studies. Secular trends in education, medical care. ACT had health insurance now and (mostly) in the past. BRFSS eligible for Medicare, but may have lesser coverage, or earlier health care may not have been as good.
When do we have to do this? When do we have to do this? Prevalence estimates for sure. Causal questions: If we suspect that the relationship may be confounded or moderated by measured differences between the study sample and the target population. Future studies should gather data in a way that aligns with variables collected in population-representative surveys so that better participation weights can be constructed. As more research is transported, we will have a better sense of how often and for what kind of questions this matters.
Conclusions Conclusions Generalizing research on older adults to the target population may require the construction of analytic weights. (As well as careful consideration of secular trends)
Thank you Coauthors: UCLA transport methods experts Taylor Mobley, Elizabeth Rose Mayeda, and Eleanor Hayes-Larson UW Eye expert Cecilia S. Lee ACT MPIs Andrea LaCroix, Linda McEvoy, and Paul K. Crane Funding: U19AG066567, U01AG006781, R56AG069126, P50AG05136, R01AG060942, 5R13AG030995.
Selected questions from the talk Q: Did you use the BRFSS survey weights in the multiple imputation? I did not, based on the advice of my transport experts. I later took a missing data short course and the instructor thought I should. However, I don t think this is an option in Stata and I have not been able to find existing software that can handle my particular data. Let me know if you have ideas! Q: Why did you impute first and then bootstrap in that data, rather than the other way around? Only because it s quicker to run. See Schomaker & Heumann (https://pubmed.ncbi.nlm.nih.gov/29682776/ ) for more on this. They find both orders to be valid.
gibbonsL@uw.edu https://github.com/gibbonsl/ACT_transport (gibbonsL not gibbons1).
