Calibrated Bayesian Approach for Survey Inference

The Calibrated Bayes Approach to Sample Survey Inference Roderick Little Department of Biostatistics, University of Michigan Associate Director for Research & Methodology, Bureau of Census

Learning Objectives 1. Understand basic features of alternative modes of inference for sample survey data. 2. Understand the mechanics of Bayesian inference for finite population quantitities under simple random sampling. 3. Understand the role of the sampling mechanism in sample surveys and how it is incorporated in a Calibrated Bayesian analysis. 4. More specifically, understand how survey design features, such as weighting, stratification, post-stratification and clustering, enter into a Bayesian analysis of sample survey data. 5. Introduction to Bayesian tools for computing posterior distributions of finite population quantities. Models for complex surveys 1: introduction 2

Acknowledgement and Disclaimer These slides are based in part on a short course on Bayesian methods in surveys presented by Dr. Trivellore Raghunathan and I at the 2010 Joint Statistical Meetings. While taking responsibility for errors, I d like to acknowledge Dr. Raghunathan s major contributions to this material Opinions are my own and not the official position of the U.S. Census Bureau Models for complex surveys 1: introduction 3

Module 1: Introduction Distinguishing features of survey sample inference Alternative modes of survey inference Design-based, superpopulation models, Bayes Calibrated Bayes Models for complex surveys 1: introduction 4

Distinctive features of survey inference 1. Primary focus on descriptive finite population quantities, like overall or subgroup means or totals Bayes which naturally concerns predictive distributions -- is particularly suited to inference about such quantities, since they require predicting the values of variables for non-sampled items This finite population perspective is useful even for analytic model parameters: = model parameter (meaningful only in context of the model) ( ) = "estimate" of from fitting model to whole population (a finite population quantity, exists regardless of validity of model) A good estimate of should be a good estimate of (if not, then what's being estimated?) Models for complex surveys 1: introduction Y Y 5

Distinctive features of survey inference 2. Analysis needs to account for "complex" sampling design features such as stratification, differential probabilities of selection, multistage sampling. Samplers reject theoretical arguments suggesting such design features can be ignored if the model is correctly specified. Models are always misspecified, and model answers are suspect even when model misspecification is not easily detected by model checks (Kish & Frankel 1974, Holt, Smith & Winter 1980, Hansen, Madow & Tepping 1983, Pfeffermann & Holmes (1985). Design features like clustering and stratification can and should be explicitly incorporated in the model to avoid sensitivity of inference to model misspecification. Models for complex surveys 1: introduction 6

Distinctive features of survey inference 3. A production environment that precludes detailed modeling. Careful modeling is often perceived as "too much work" in a production environment (e.g. Efron 1986). Some attention to model fit is needed to do any good statistics Off-the-shelf" Bayesian models can be developed that incorporate survey sample design features, and for a given problem the computation of the posterior distribution is prescriptive, via Bayes Theorem. This aspect would be aided by a Bayesian software package focused on survey applications. Models for complex surveys 1: introduction 7

Distinctive features of survey inference 4. Antipathy towards methods/models that involve strong subjective elements or assumptions. Government agencies need to be viewed as objective and shielded from policy biases. Addressed by using models that make relatively weak assumptions, and noninformative priors that are dominated by the likelihood. The latter yields Bayesian inferences that are often similar to superpopulation modeling, with the usual differences of interpretation of probability statements. Bayes provides superior inference in small samples (e.g. small area estimation) Models for complex surveys 1: introduction 8

Distinctive features of survey inference 5. Concern about repeated sampling (frequentist) properties of the inference. Calibrated Bayes: models should be chosen to have good frequentist properties This requires incorporating design features in the model (Little 2004, 2006). Models for complex surveys 1: introduction 9

Approaches to Survey Inference Design-based (Randomization) inference Superpopulation Modeling Specifies model conditional on fixed parameters Frequentist inference based on repeated samples from superpopulation and finite population (hybrid approach) Bayesian modeling Specifies full probability model (prior distributions on fixed parameters) Bayesian inference based on posterior distribution of finite population quantities argue that this is most satisfying approach Models for complex surveys 1: introduction 10

Design-Based Survey Inference ( ,..., ) design variables, known for population N Z Z = = I IN = ( ,..., ) 1 = Sample Inclusion Indicators 1, unit included in sample 0, otherwise ( ,..., ) = population values, recorded only for sample Z I 1 = iI Y I Z = Y Y Y 1 1 1 0 0 0 0 0 1 N Y inc = = ( ) I part of included in the survey Y Y Y inc inc Note: here is random variable, ( , ) are fixed I [ ] Y Z Y exc = q ( , ) = target finite population quantity Q Y Z ( , , ) = sample estimate of q I Y Z = ( , , ) = sample estimate of V I Y Z ( Models for complex surveys 1: introduction Q Q inc V inc ) + = q 1.96 , 1.96 95% confidence interval for V q V Q 11

Random Sampling Random (probability) sampling characterized by: Every possible sample has known chance of being selected Every unit in the sample has a non-zero chance of being selected In particular, for simple random sampling with replacement: All possible samples of size n have same chance of being selected {1,..., } = set of units in the sample frame Z N = = N n N 1/ , , I n N n ! N i = Pr( | )= I Z ; = 1 i !( )! n N n 0, otherwise = = = ( | ) i E I Z Pr( 1| ) Z / I n N i Models for complex surveys 1: introduction 12

Example 1: Mean for Simple Random Sample 1 , population mean i i N = ( ) / , the sample mean i i i = Fixed quantity, not modeled Unbiased for : / I i i i = N = = Q Y y Random variable 1 N = = q I y I y n 1 = N N N = = ( ) / ( / n N y ) / Y E I y n E I y n n Y I i i i = = 1 1 1 i i 1 N = = = 2 2 2 Var ( ) I (1 / ) / , n S ( ) y V n N S y Y i 1 N = 1 i = (1 / ) finite populatio n correc tion n N 1 N = 2 2 2 (1 / ) / , n s = sample variance = ( Models for complex surveys 1: introduction ( ) V n N s I y y i i 1 n = 1 i ) = + 95% confidence interval for 1.96 , 1.96 Y y V y V 13

Example 2: Horvitz-Thompson estimator Q Y T Y YN ( ) ... = + + 1 ( | ) = inclusion probability i i E I Y = / , E ( ) i i i I i i = = Variance estimate, depends on sample design v = ( HT HT HT 1.96 , 1.96 t v t v 0 N N N = = = ( ) E I Y / = / t I Y t i i Y T HT HT i i i i = 1 1 1 i HT ) + = 95% CI for T HT Pro: unbiased under minimal assumptions Cons: variance estimator problematic for some designs (e.g. systematic sampling) can have poor confidence coverage and inefficiency Models for complex surveys 1: introduction 14

Role of Models in Classical Approach Inference not based on model, but models are often used to motivate the choice of estimator. E.g.: Regression model regression estimator Ratio model ratio estimator Generalized Regression estimation: model estimates adjusted to protect against misspecification, e.g. HT estimation applied to residuals from the regression estimator (Cassel, Sarndal and Wretman book). Estimates of standard error are then based on the randomization distribution This approach is design-based, model-assisted Models for complex surveys 1: introduction 15

Model-Based Approaches In our approach models are used as the basis for the entire inference: estimator, standard error, interval estimation This approach is more unified, but models need to be carefully tailored to features of the sample design such as stratification, clustering. One might call this model-based, design-assisted Two variants: Superpopulation Modeling Bayesian (full probability) modeling Common theme is Infer or predict about non-sampled portion of the population conditional on the sample and model Models for complex surveys 1: introduction 16

Superpopulation Modeling Model distribution M: ~ ( | , ), = design variables, Y f Y Z Z Predict non-sampled values : = fixed parameters Y exc I Z Y = E y z =RST over distribution of and , . v q v + 196 f = 95% CI for = = y ( | , ), model estimate of i i i 1 1 1 0 0 0 0 0 , if unit sampled; if unit not sampled y Y (~),~ Q Y i = inc q y i , y i = Y ( ), mse q . 196 v q a I M exc Q In the modeling approach, prediction of nonsampled values is central In the design-based approach, weighting is central: sample represents units in the population Models for complex surveys 1: introduction 17

Bayesian Modeling Bayesian model adds a prior distribution for the parameters: ( , ) ~ ( | ) ( | , ), Y Z f Y Z Inference about is based on posterior distribution from Bayes Theorem: ( | , ) ( | ) ( | , ), = likelihood p Z Y Z L Z Y L = ( | ) prior distribution Z I Z Y inc inc Inference about finite population quantitity ( ) based on ( ( )| ) posterior predictive distribution of given sample values Q Y = Q Y 1 1 1 0 0 0 0 0 Y inc = p Q Y Y inc inc Y , ) ( | , p ( ( )| , p Q Y ) ( ( )| , p Q Y ) Z Y Z Y Z Y d exc inc inc inc (Integrates out nuisance parameters ) In the super-population modeling approach, parameters are considered fixed and estimated In the Bayesian approach, parameters are random and integrated out of posterior distribution leads to better small-sample inference Models for complex surveys 1: introduction 18

Bayesian Point Estimates Point estimate is often used as a single summary best value for the unknown Q Some choices are the mean, mode or the median of the posterior distribution of Q For symmetrical distributions an intuitive choice is the center of symmetry For asymmetrical distributions the choice is not clear. It depends upon the loss function. Models for complex surveys: simple random sampling 19

Bayesian Interval Estimation Bayesian analog of confidence interval is posterior probability or credibility interval Large sample: posterior mean +/- z * posterior se Interval based on lower and upper percentiles of posterior distribution 2.5% to 97.5% for 95% interval Optimal: fix the coverage rate 1- in advance and determine the highest posterior density region C to include most likely values of Q totaling 1- posterior probability Models for complex surveys: simple random sampling 20

Bayes for population quantities Q Inferences about Q are conveniently obtained by first conditioning on and then averaging over posterior of . In particular, the posterior mean is: inc ( | ) ( | E Q Y E E Q Y = , )| Y inc inc and the posterior variance is: = , )| + , )| ( | ) ( | ( | Var Q Y E Var Q Y Y Var E Q Y Y inc inc inc inc inc Value of this technique will become clear in applications Finite population corrections are automatically obtained as differences in the posterior variances of Q and Inferences based on full posterior distribution useful in small samples (e.g. provides t corrections ) Models for complex surveys: simple random sampling 21

Simulating Draws from Posterior Distribution For many problems, particularly with high-dimensional it is often easier to draw values from the posterior distribution, and base inferences on these draws For example, if is a set of draws from the posterior distribution for a scalar parameter , then 1 1 1 approximates posterior mean d D = = ( ) 1 d ( : 1,..., ) d D 1 D = 1 ( ) d = D = 2 1 ( ) 1 2 d ( 1) ( ) approximates posterior variance s D 1 1 d ( approximates 95% posterior credibility interva Given a draw of , usually easy to draw non-sampled values of data, and hence finite population quantities 1.96 ) or 2.5th to 97.5th percentiles of draws s 1 l for ( ) d Models for complex surveys: simple random sampling 22

Calibrated Bayes Any approach (including Bayes) has properties in repeated sampling We can study the properties of Bayes credibility intervals in repeated sampling do 95% credibility intervals have 95% coverage? A Calibrated Bayes approach yields credibility intervals with close to nominal coverage Frequentist methods are useful for forming and assessing model, but the inference remains Bayesian See Little (2004) for more discussion Models for complex surveys 1: introduction 23

Summary of approaches Design-based: Avoids need for models for survey outcomes Robust approach for large probability samples Less suited to small samples inference basically assumes large samples Models needed for nonresponse, response errors, small areas this leads to inferential schizophrenia Models for complex surveys 1: introduction 24

Summary of approaches Superpopulation/Bayes models: Familiar: similar to modeling approaches to statistics in general Models needs to reflect the survey design Unified approach for large and small samples, nonresponse and response errors. Frequentist superpopulation modeling has the limitation that uncertainty in predicting parameters is not reflected in prediction inferences: Bayes propagates uncertainty about parameters, making it preferable for small samples but needs specification of a prior distribution Models for complex surveys 1: introduction 25

Module 2: Bayesian models for simple random samples 2.1 Continuous outcome: normal model 2.2 Difference of two means 2.3 Regression models 2.4 Binary outcome: beta-binomial model 2.5 Nonparametric Bayes Models for complex surveys 1: introduction 26

Models for simple random samples Consider Bayesian predictive inference for population quantities Focus here on the population mean, but other posterior distribution of more complex finite population quantities Q can be derived Key is to compute the posterior distribution of Q conditional on the data and model Summarize the posterior distribution using posterior mean, variance, HPD interval etc Modern Bayesian analysis uses simulation technique to study the posterior distribution Here consider simple random sampling: Module 3 considers complex design features Models for complex surveys: simple random sampling 27

Diffuse priors In much practical analysis the prior information is diffuse, and the likelihood dominates the prior information. Jeffreys (1961) developed noninformative priors based on the notion of very little prior information relative to the information provided by the data. Jeffreys derived the noninformative prior requiring invariance under parameter transformation. In general, 1/2 ( ) | ( )| where = J 2 log ( | ) f y ( ) J E t Models for complex surveys: simple random sampling 28

Examples of noninformative priors 2 2 Normal: ( , Binomial: ( ) ) 1/2 1/2 (1 ) 1/2 Poisson: ( ) 2 2 Normal regression with slopes : ( , ) In simple cases these noninformative priors result in numerically same answers as standard frequentist procedures Models for complex surveys: simple random sampling 29

2.1 Normal simple random sample = 2 ~ iid ( , simple random sample results in ( ny N Q Y N f y f = + ); 1,2,..., iY N i N 2 2 ( , ) = ( ,..., y ) Y y inc 1 n + ) n Y = = exc (1 ) Y exc Derive posterior distribution of Q Models for complex surveys: simple random sampling 30

2.1 Normal Example Posterior distribution of ( 2) 2 ( ) y 1 /2 1 ( , 2 2 n | ) (2 ) exp i p Y inc 2 2 2 inc i 1 2 /2 1 2 2 2 2 2 n ( ) exp ( ) / ( ) / y y n y i inc i The above expressions imply that 2 2 2 n (1) | ~ ( ) / Y y y inc 1 i inc ~ i 2 2 (2) | , ( , / ) n Y N y inc Models for complex surveys: simple random sampling 31

2.1 Posterior Distribution of Q 2 2 exc | , ~ , Y N N n 2 2 2 + = 2 | , ~ , Y Y N y exc inc (1 ) N n n f n = + (1 ) Q f y f Y exc 2 (1 ) f n 2 | , ~ , Q Y N y inc 2 s | ~ ,(1 Y Y t y exc inc 1 n ) f n 2 (1 ) f s n | ~ , Q Y t y inc 1 n Models for complex surveys: simple random sampling 32

2.1 HPD Interval for Q Note the posterior t distribution of Q is symmetric and unimodal -- values in the center of the distribution are more likely than those in the tails. Thus a (1- )100 HPD interval is: 2 (1 ) f s n y t 1,1 /2 n Like frequentist confidence interval, but recovers the t correction Models for complex surveys: simple random sampling 33

2.1 Some other Estimands Suppose Q=Median or some other percentile One is better off inferring about all non-sampled values As we will see later, simulating values of adds enormous flexibility for drawing inferences about any finite population quantity Modern Bayesian methods heavily rely on simulating values from the posterior distribution of the model parameters and predictive-posterior distribution of the nonsampled values Computationally, if the population size, N, is too large then choose any arbitrary value K large relative to n, the sample size National sample of size 2000 US population size 306 million For numerical approximation, we can choose K=2000/f, for some small f=0.01 or 0.001. Y exc Models for complex surveys: simple random sampling 34

2.1 Comments Even in this simple normal problem, Bayes is useful: t-inference is recovered for small samples by putting a prior distribution on the unknown variance Inference for other quantities, like Q=Median or some other percentile, is achieved very easily by simulating the nonsampled values (more on this below) Bayes is even more attractive for more complex problems, as discussed later. Models for complex surveys: simple random sampling 35

2.2 Comparison of Two Means Population 1 Population 2 = Population size Sample size ind N N = Population size Sample size ind N N 2 1 = n = n 2 1 2 2 ( , ) Y 2 1 ( , ) Y 2 2 i 1 1 i 2 2 2 ( , ) 2 1 2 ( , ) 2 2 1 1 2 1 2 2 : ( , ) : ( , ) Sample Statistics Posterior distributions y s Sample Statistics Posterior distributions y s 2 1 / : : 2 2 2 2 2 n 2 1 2 1 2 n ( 1) ~ n s ( 1) / ~ n s 2 1 1 1 2 1 2 2 2 2 2 1 ~ ( , / ) N y n ~ ( , / ) N y n 2 2 2 1 1 1 ~ ( , ), exc 2 1 Y N i ~ ( , ), exc Y N i 2 2 i 1 1 i Models for complex surveys: simple random sampling 36

2.2 Estimands Examples (Finite sample version of Behrens-Fisher Problem) Difference Difference in the population medians Ratio of the means or medians Ratio of Variances It is possible to analytically compute the posterior distribution of some these quantities It is a whole lot easier to simulate values of non- sampled in Population 1 and in Population 2 1 Y Y Y 1 2 ) Pr( Pr( ) Y c Y c 1 2 ' ' s s Y 2 Models for complex surveys: simple random sampling 37

2.3 Ratio and Regression Estimates x x y y . . . Population: (yi,xi; i=1,2, N) Sample: (yi, i inc, xi, i=1,2, ,N). 1 1 2 2 . . . x x x For now assume SRS yn n Objective: Infer about the population mean N Q y = + 1 n = i + 2 n 1 i . . . x Excluded Y s are missing values N Models for complex surveys: simple random sampling 38

2.3 Model Specification ) ~ ind ( , i N x x N , , 2 2 2 i g ( | Y x i g Prior distribution: ( , ) i i = 1,2,..., known 2 2 ) g=1/2: Classical Ratio estimator. Posterior variance equals randomization variance for large samples g=0: Regression through origin. The posterior variance is nearly the same as the randomization variance. g=1: HT model. Posterior variance equals randomization variance for large samples. Note that, no asymptotic arguments have been used in deriving Bayesian inferences. Makes small sample corrections and uses t- distributions. Models for complex surveys: simple random sampling 39

2.3 Posterior Draws for Normal Linear Regression g = 0 2 ( , ) ls estimates of slopes and resid variance ( 1) / A z = = s = ( )2 d 2 2 n p n p s 1 = + ( ) d ( ) d T 2 n p = chi-squared deviate with 1 df n p 1 z T ( ,..., ) , ~ (0,1) z z z N + 1 1 p i = 1 T upper triangular Cholesky factor of ( ( ) Nonsampled values | ) : A X X = 1 T T A A X X ( ) d ( ) d ( ) d ( )2 d , ~ ( , ) y N x i i Easily extends to weighted regression Models for complex surveys: simple random sampling 40

2.4 Binary outcome: consulting example In India, any person possessing a radio, transistor or television has to pay a license fee. In a densely populated area with mostly makeshift houses practically no one was paying these fees. Target enforcement in areas where the proportion of households possessing one or more of these devices exceeds 0.3, with high probability. Models for complex surveys: simple random sampling 41

2.4 Consulting example (continued) Population Size in particular area 1, if household has a device 0, otherwise = N i = Y i N = / Proportion of households with a device Q Y N i = 1 i Question of Interest: Pr( 0.3) Q Conduct a small scale survey to answer the question of interest Note that question only makes sense under Bayes paradigm Models for complex surveys: simple random sampling 42

2.4 Consulting example = = srs of size , | ~ i Y { ,..., }, Y { ,..., } n Y Y Y Y Y + inc 1 exc 1 n n N Bernoulli( ) iid n = x Y i = 1 i Model for observable n x = = n x ( | ) ( ) ( ) x (1 ) f x 1 (0,1) = + Prior distribution N N = / / Q Y N x Y N i i Estimand = i n = + 1 1 i Models for complex surveys: simple random sampling 43

2.4 Beta Binomial model The posterior distribution is ( | ) ( ) ( | ) ( | ) ( ) f x f x = ( | ) ( ) f x p x d n x n x x ( ) ( ) (1 (1 ) 1 ( | ) = p x n x n x x ) d + + | ~ ( 1, 1) x Beta x n x Models for complex surveys: simple random sampling 44

2.4 Infinite Population , x For N Y Y N Pr( Compute using cumulative distribution function of a beta distribution which is a standard function in most software such as SAS, R 0.3| ) Pr( 0.3| ) x N What is the maximum proportion of households in the population with devices that can be said with great certainty? Pr( ?| ) 0.9 Inverse CDF of Beta Distribution Models for complex surveys: simple random sampling = x 45

2.5 Bayesian Nonparametric Inference Population: All possible distinct values: Model: Prior: Mean and Variance: , , ,..., Y Y Y Y 1 2 3 N , ,..., d d 1 dK 2 = ,..., = Pr( ( , ) ) Y d i k k = 1 if 1 1 2 k k k k k = = ( | ) i E Y d k k k = = 2 2 k 2 ( | ) i Var Y d k k Models for complex surveys: simple random sampling 46

2.5 Bayesian Nonparametric Inference SRS of size n with nk equal to number of dk in the sample Objective is to draw inference about the population mean: As before we need the posterior distribution of and = + (1 ) Q f y f Y exc Models for complex surveys: simple random sampling 47

2.5 Nonparametric Inference Posterior distribution of is Dirichlet: = = 1 k n k ( | ) if 1 and Y n n inc k k k k k Posterior mean, variance and covariance of ( ) n n n n n n n = = ( | ) , ( | ) k k k E Y Var Y inc inc k k + 2 ( 1) n n ( , = | ) k l Cov Y inc k l + 2 ( 1) n n Models for complex surveys: simple random sampling 48

2.5 Inference for Q n n + + ( | = = ) k E Y d y inc k k 2 1 1 1 s n n n n n ( | = = 2 2 ) ; ( ) Var Y s y y inc i 1 n inc i 1 1 = 2 2 ( | ) E Y s inc Hence posterior mean and variance of Q are: = + = ( | E Q Y ) (1 ) ( | f E ) f y Y y inc inc + 2 1 1 s n n n = ( | ) (1 ) Var Q Y f inc Models for complex surveys: simple random sampling 49

Module 3: complex sample designs Considered Bayesian predictive inference for population quantities Focused here on the population mean, but other posterior distribution of more complex finite population quantities Q can be derived Key is to compute the posterior distribution of Q conditional on the data and model Summarize the posterior distribution using posterior mean, variance, HPD interval etc Modern Bayesian analysis uses simulation technique to study the posterior distribution Models need to incorporate complex design features like unequal selection, stratification and clustering Models for complex surveys: simple random sampling 50