Adaptive Mixed-Mode Survey Design for Social Surveys: Insights from MIMOD Project
The MIMOD project explores Adaptive Mixed-Mode Survey Design (ASD) to optimize quality-cost trade-offs in social surveys. The workshop in Rome discussed the implementation of ASD, the challenges faced, and steps for successful implementation. Findings indicated Stat Netherlands as the primary implementer of ASD. The key to successful ASD implementation lies in identifying priorities, managing risks, defining quality and cost indicators, and continuously monitoring outcomes. The checklist provided outlines crucial steps to implement ASD effectively.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
ROME April 11th | 12th 2019 MIMOD Mixed-Mode Designs for Social Surveys FINAL WORKSHOP Adaptive mixed-mode design WP1 Barry Schouten Statistics Netherlands (CBS)
Adaptive mixed-mode survey design (ASD) Adaptive survey design optimizes quality-cost trade-offs by differentiating effort to different (relevant) population strata. In MIMOD, differentiation of effort is focussed at the choice of mode strategy per population stratum. Objectives of WP1 mixed-mode ASD: Make an inventory of ASD implementations in ESS countries; Structure the decisions/steps towards an mixed-mode ASD; Illustrate using two case studies; 2 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019
WP1 MIMOD survey Findings from WP1 survey: Mixed-mode ASD implemented only at Stat Netherlands; ASD is a relatively unknown strategy to balance quality and costs. Eight countries indicated in survey they were unsure whether ASD is applied; Potential reasons: Implementation demands flexible case management system across modes; Relatively weak available auxiliary data to stratify the population; Mostly theoretical approach without many success stories; 2 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019
ASD steps to implementation Key ingredients of ASD: Explicit quality and costs metrics; Relevant auxiliary data; Design features/interventions In general: All possible elements of data collection strategy; In MIMOD: Modes; Optimization strategy, e.g. Case prioritization; Mathematical optimization; Stopping rules based on quota 2 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019
ASD steps to implementation checklist 1. 2. Identify priorities; Identify major risks: Consider risk of incomparability in time; Consider risk of incomparability between subgroups; Consider risk of budget overrun and heavy interviewer workloads in follow-up modes; Define quality and cost indicators; Consider nonresponse indicators; Consider measurement error indicators; Consider cost indicators; Define decision rules from: Trial-and-error; Case prioritization; Quota; Mathematical optimization; Modify the survey design and monitor the outcomes; Develop a dashboard for survey errors; Develop a dashboard for survey costs; Compute estimates; Document; 3. 4. 5. 6. 7. 2 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019
ASD case study Health Survey/EHIS priorities and risks Key design feature: yes/no F2F follow-up to web nonrespondents Main priorities: Acceptable and similar response rates among relevant population subgroups Sufficient precision on annual survey estimates Costs satisfying a specified budget Main risks: Incomparability in time Unpredictable CAPI workload due to varying monthly and annual web response rate Incomparability between different population subgroups of interest 2 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019
ASD case study Health Survey/EHIS quality and costs Objective: Maximize coefficient of variation (CV) of response propensities (combines R-indicator and response rate) Response propensities modelled by age, income, urbanization, type of household, ethnicity Constraints: A minimum annual total number of about 9500 respondents was requested An upper limit of 8000 was imposed to the number of nonrespondents that are sent to CAPI, as a proxy for a budget constraint An upper limit of 18000 persons was set to the sample size TO DO: Inclusion of constraint on mode-specific measurement bias 2 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019
ASD case study Health Survey/EHIS optimization Stratification based on classification tree of web response 2 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019
ASD case study Health Survey/EHIS optimization Optimal allocation probabilities of web nonrespondents to F2F follow-up were determined based on mathematical optimization. Per month allocation probabilities are rescaled to guarantee a fixed F2F workload. Stratum 1 2 3 4 5 6 7 8 9 total RR web 42.6 24.8 37.6 23.1 39.3 36.3 19.3 20.2 28.9 35.9 Alloc F2F 56,4 95.1 71.4 100.0 43.5 66.5 100.0 100.0 65.8 71.5 RR F2F 48.5 47.1 42.3 25.4 64.8 47.6 30.8 38.7 58.3 42.5 RR tot 58.1 56.4 57.1 42.3 57.2 57.4 44.4 52.3 57.0 55.7 CV uniform design = 0.158 CV adaptive design = 0.116 2 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019
Concluding remarks Adaptive mixed-mode survey design offers a flexible way to balance quality and budget Holds true especially in sequential designs with more expensive (interviewer) modes as optional. Further within mode differentiation (timing and number of calls/visits) is possible. However: Account of both representation and measurement is crucial; A flexible case management system and monitoring is required; Future: May be combined with re-interview designs (WP2) May be combined with sensor measurements/data (WP5) 2 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019
