Challenges in RDD Telephone Surveys for Young Adults: Item Nonresponse and Coverage Implications
This research study conducted in Quebec, Canada, in 2011 focused on the challenges faced in RDD telephone surveys, specifically targeting young adults. It discussed issues such as decreasing coverage due to wireless-only households, nonresponse rates, and the impact of emerging trends on survey methodologies. The study highlighted the importance of maintaining reliable survey estimates despite these challenges to ensure the validity of data collected.
Uploaded on Sep 29, 2024 | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Item Nonresponse in a Mail Survey of Young Adults Luciano Viera, Jr., Scott Turner, and Sean Marsh Fors Marsh Group LLC
2011 ITSEW Qubec, Canada Overview Background RDD Telephone Surveys: Coverage Implications Youth Poll Mail Study Purpose and Methodology Mail YP Frame Coverage Data Collection Efficiency Comparison Profile of Responders Survey Estimate Comparison Item Nonresponse Incentive Experiment Methodology Transition 2
2011 ITSEW Qubec, Canada Background The US Department of Defense (DoD) conducts the Youth Poll (YP) to track military propensity Tracked over the past 35 years Critical to maintain these trend lines Random-digit-dial (RDD) survey design Historically, estimates have been shown to be reliable and valid reliable and valid For decades, RDD telephone surveying has been a cost-efficient way to survey the general public RDD surveys are typically interviewer interviewer- -administered administered Emerging trends are impacting the future viability of this methodology Decreasing coverage Decreasing coverage of telephone surveys General decrease in survey research response rates Reduced efficiency Reduced efficiency due to coverage and nonresponse issues response rates 3
2011 ITSEW Qubec, Canada RDD Telephone Surveys: Coverage Implications CDC study found 46% of 18 Households with a landline are different Wireless-only households are more likely to be located in urban, metropolitan areas 46% of 18- - to 24 to 24- -year are different from those without a landline year- -olds olds live in wireless-only households Percentage of Adults and Children with Only Wireless or No Telephone Service Percentage of Adults and Children with Only Wireless or No Telephone Service 35% 31.8% 29.0% 30% 25.9% 27.8% Children with wireless service only 25% 21.3% 24.9% 18.7% 22.9% 20% 17.0% 21.1% 14.5% 18.4% Adults with wireless service only 15% 12.6% 16.1% 11.8% 9.6% 14.4% Children with no telephone service 7.7% 10% 11.9% 11.6% 6.7% Adults with no telephone service 8.6% 5.4% 4.9% 2.3% 4.4% 3.5% 7.6% 1.8% 5% 2.9% 2.5% 2.0% 5.8% 2.3% 1.7% 1.9% 2.1% 2.4% 1.9% 1.7% 1.7% 2.8% 1.8% 3.7% 2.0% 1.6% 1.5% Jan-Jun 2005 3.2% 1.8% 2.1% 1.7% 1.6% 1.9% 1.8% 1.6% 1.8% Jan-Jun 2006 1.7% Jul-Dec 2005 1.5% 1.8% 1.7% 1.5% 1.7% 1.7% 1.7% 0% Jan-Jun 2003 Jul-Dec 2003 Jan-Jun 2004 Jul-Dec 2004 Jul-Dec 2006 Jan-Jun 2007 Jul-Dec 2007 Jan-Jun 2008 Jul-Dec 2008 Jan-Jun 2009 Jul-Dec 2009 Jan-Jun 2010 Jul-Dec 2010 4 Source: Blumberg & Luke (2011)
2011 ITSEW Qubec, Canada Purpose and Methodology The Youth Poll Mail Study (YPMS) conducted a series of data collection mode comparisons Goal is to evaluate the feasibility of switching the current RDD telephone based Youth Poll to a mail-based survey methodology Run concurrently with RDD YP to compare coverage, nonresponse, and key metrics December 2008 June 2009 December 2009 June 2010 Mail YP RDD YP Age range 16-24 16-24 Survey Interview Method Pencil & Paper CATI Single-Stage Stratified Random List-Assisted Stratified Random Sampling 5
2011 ITSEW Qubec, Canada Mail YP Frame Coverage June 2010 frame was an updated version of December 2009 frame: Aged to remove any youth that were no longer 24 years old Augmented using additional lists to capture youth that had just turned 16 December 2009 frame coverage Comparison with Census estimates of the 16- to 24-year-old population indicated a 95% coverage rate, a 6 percentage point improvement from June 2009 (89%)* Coverage of both 16- (67%) and 24-year-old (95%) youth increased substantially since June 2009 (47% and 60%, respectively) Coverage of the West region also improved since June 2009 June 2009 December 2009 Frame size 33,902,404 36,483,999 Coverage rate 89% 95% 16-year-olds 47% 67% 24-year-olds 60% 95% West Region 75% 84% 7
2011 ITSEW Qubec, Canada Data Collection Efficiency The Mail YP required fewer contacts to reach the target population than the RDD YP and did so at less than one less than one- -third the cost per completed interview third the cost per completed interview with no additional fielding time required The Mail YP s cost per completed survey in June 2010 ($77) was lower than in December 2009 ($99) and roughly equal to that in June 2009 ($76) Increase in costs in December 2009 was largely a result of evaluating a push-to-Web solicitation strategy that was considerably less productive Mail YP RDD YP Length of Time to Field 2.5 months 2.5 months Contacts 30,000 223,173 Completed Surveys 5,119 4,023 Completion Rate 17% 2% Cost per Completed Survey $77 $273 8
2011 ITSEW Qubec, Canada Profile of Responders Demographics Demographics Compared with the RDD YP and census estimates, respondents in the Mail YP were more educated Telephone Status Telephone Status Compared with the RDD YP, respondents in Mail YP were: More likely to have a cell phone only More likely to be cell-mostly (i.e., both a landline and a cell phone who receive all or almost all of their calls on a cell phone) Less likely to have a landline only 9
2011 ITSEW Qubec, Canada Survey Estimate Comparisons Propensity estimates were statistically identical between the Mail and RDD YP True for both overall and Service-specific propensity comparisons Mail YP tends to provide slightly lower propensity estimates than RDD YP Finding is consistent with expectations June 2010 Key Metrics Mail YP RDD YP General Military Propensity 10% 10% Army Propensity 6% 6% Navy Propensity 6% 6% Marine Corps Propensity 6% 5% Air Force Propensity 7% 7% Coast Guard Propensity 5% 4% 10
2011 ITSEW Qubec, Canada June 2010 Mail YP Propensity Items 12
2011 ITSEW Qubec, Canada Item Nonresponse In the June 2010 Mail YP, Service-specific propensity grid items yielded overall missing data rate of 8% Non-random forms of nonresponse may bias point estimates Analyzed June 2010 Mail YP data to determine: Whether there was a pattern to the missing data How refusals should be handled pattern to the missing data; and handled (e.g., multiple imputation, etc.) Pattern of nonresponse was NOT RANDOM! Non Non- -propensed youth propensed youth were more likely to refuse answering all 12 items Conversely, propensed youth propensed youth more likely to refuse to answer 1 1- -11 items, which is probably a result of their preference for a specific Service(s) NOT RANDOM! all 12 Service-specific 11 Service-specific preference for a specific Service(s). 13
2011 ITSEW Qubec, Canada December 2010 Administration Modified the Service-specific propensity item in the December 2010 questionnaire: Proportion of missing data reduced by half! From June 2010 , 8% missing data rates down to 4% reduced by half! 14
AAPOR 2011 Phoenix, AZ Combining Prepaid and Promised Incentives Mail YP Mail YP Periodically conducts experiments experiments to enhance survey quality Combining prepaid and promised incentives Combining prepaid and promised incentives Ideally, this would maximize survey quality maximize survey quality at a reduced cost reduced cost Theoretical justification Theoretical justification Trust is the centerpiece of social influence theory (i.e., enhancing trust facilitates the persuasion process) Prepaid incentives may build trust such that they might magnify the positive effects of promised incentives 16
AAPOR 2011 Phoenix, AZ Methodology June 2010 Experiment June 2010 Experiment Sample of 30,000 young adults ages 16-24 living in US were randomly assigned to 1-of-6 conditions (5,000 each) Survey length (long vs. short) Promised incentive ($0 vs. $5 vs. $10) Everyone received $2 cash incentive in the 1st survey invitation package Promised incentives sent to respondents that returned questionnaires Where possible, all mailing materials sent to youth were identical 17
AAPOR 2011 Phoenix, AZ Results Nonresponse Nonresponse Completion rates for the short survey were higher than for long survey Promised incentives help offset completion rate drops in long survey Key Metric Measurement Key Metric Measurement In general, point estimates similar across all the experimental conditions Efficiency Efficiency Compared with offering just a prepaid incentive, promised incentive with long survey reduces costs by approximately 18-26% per completed survey Short; Short; $5 $5 19% 19% 67% 67% 55% 55% 9% 9% Short; Short; $10 $10 22% 22% 69% 69% 54% 54% 9% 9% Long; Long; $5 $5 15% 15% 66% 66% 57% 57% 10% 10% Long; Long; $10 $10 18% 18% 70% 70% 52% 52% 9% 9% Short; Short; $0 $0 17% 17% 72% 72% 56% 56% 8% 8% Long; Long; $0 $0 12% 12% 72% 72% 55% 55% 10% 10% Completed Surveys Completed Surveys Enrolled in School Enrolled in School Currently Employed Currently Employed General Military Propensity General Military Propensity 18 Note: All estimates in this table are unweighted; 5,000 cases sampled in each condition.
2011 ITSEW Qubec, Canada Summary Several indicators supporting the switch to a mail-based methodology Mail YP s existing frame coverage (95%) existing frame coverage (95%) of the target population represented a marked improvement over existing RDD methodologies that do not capture the steadily growing number of cell phone only young adults Metrics compared well across survey modes Both overall and Service-specific propensity estimates were statistically identical propensity estimates were statistically identical between the Mail and RDD YP Mail YP required fewer contacts than RDD YP to reach the target population Mail YP was less than one less than one- -third the cost third the cost per completed interview Mail YP required no additional fielding time Mail YP is proving to be a viable vehicle for motivating higher response rates Compared with offering no contingent incentive offering no contingent incentive, including a contingent incentive reduces data collection costs by approximately 18 approximately 18- -26% motivating higher response rates 26% per completed survey* 20 *Figures are based on a comparison of the cost per completed survey in the $5 contingent incentive condition ($63,745 survey cost 749 completed surveys = $85.11 per completed survey) and the $10 contingent incentive condition ($68,980 survey cost 898 completed surveys = $76.82 per completed survey) vs. the $0 contingent incentive condition ($60,000 survey cost 575 completed surveys = $104.35 per completed survey).
