Cautionary Tale: Internet Surveys for Identifying People with Disabilities

Internet surveys offer advantages over traditional methods like Random Digit Dial (RDD) surveys for identifying people with disabilities, including cost-effectiveness and flexibility. However, caution is needed due to potential pitfalls such as non-response rates and uncertain sample quality. Various sample design elements play a crucial role in the success and accuracy of data collection.

• Internet surveys
• Disabilities
• Cautionary tale
• Sample design
• Survey methods

brig_21 Follow

Uploaded on Oct 04, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Usefulness of Internet Surveys to Identify People with Disabilities: A Cautionary Tale Andrew J. Houtenville, Ph.D. October 6, 2022 University of New Hampshire

2. Random Digit Dial (RDD) surveys are increasingly ineffective Large non-response rates Expensive with lots of calls needed Internet-based samples are Inexpensive Readily available Flexible survey instrument Motivation and Purpose of the Study

3. Sample Design Elements Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web Mode Sample frame Additional costs per complete Days in the field Incentives Proxy allowed

4. Sample Design Elements Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web Telephone Interview Web browser Web browser Web browser Mode Sample frame Additional costs per complete Days in the field Incentives Proxy allowed

5. Sample Design Elements Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web Telephone Interview Web browser Web browser Web browser Mode Sample frame Landline/cell listings less nonresidential listings Unknown frame, prescreened for health condition(s) Listservs & social media postings Unknown Additional costs per complete Days in the field Incentives Proxy allowed

6. Sample Design Elements Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web Telephone Interview Web browser Web browser Web browser Mode Sample frame Landline/cell listings less nonresidential listings Unknown frame, prescreened for health condition(s) Listservs & social media postings Unknown Additional costs per complete $105 $6.50 $9.50 or $15.00 $0.00 Days in the field Incentives Proxy allowed

7. Sample Design Elements Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web Telephone Interview Web browser Web browser Web browser Mode Sample frame Landline/cell listings less nonresidential listings Unknown frame, prescreened for health condition(s) Listservs & social media postings Unknown Additional costs per complete $105 $6.50 $9.50 or $15.00 $0.00 189 22 13 159 Days in the field Incentives Proxy allowed

8. Sample Design Elements Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web Telephone Interview Web browser Web browser Web browser Mode Sample frame Landline/cell listings less nonresidential listings Unknown frame, prescreened for health condition(s) Listservs & social media postings Unknown Additional costs per complete $105 $6.50 $9.50 or $15.00 $0.00 189 22 13 159 Days in the field None Yes, unknown Yes, unknown None Incentives Proxy allowed

9. Sample Design Elements Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web Telephone Interview Web browser Web browser Web browser Mode Sample frame Landline/cell listings less nonresidential listings Unknown frame, prescreened for health condition(s) Listservs & social media postings Unknown Additional costs per complete $105 $6.50 $9.50 or $15.00 $0.00 189 22 13 159 Days in the field None Yes, unknown Yes, unknown None Incentives Yes No No Yes Proxy allowed

10. Sampling Results Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web Contacted Accessed Ineligible Screened out for inattentiveness Completed surveys Proxy responses AAPOR response rate 3

11. Sampling Results Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web 117,871 Unknown Unknown Unknown Contacted Accessed Ineligible Screened out for inattentiveness Completed surveys Proxy responses AAPOR response rate 3

12. Sampling Results Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web 117,871 Unknown Unknown Unknown Contacted Not applicable 11,045 6,350 682 Accessed Ineligible Screened out for inattentiveness Completed surveys Proxy responses AAPOR response rate 3

13. Sampling Results Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web 117,871 Unknown Unknown Unknown Contacted Not applicable 11,045 6,350 682 Accessed 7,440 2,665 Not ascertained Ineligible 95,347 Screened out for inattentiveness Completed surveys Proxy responses AAPOR response rate 3

14. Sampling Results Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web 117,871 Unknown Unknown Unknown Contacted Not applicable 11,045 6,350 682 Accessed 7,440 2,665 Not ascertained Ineligible 95,347 Screened out for inattentiveness Not applicable 583 664 Not applicable Completed surveys Proxy responses AAPOR response rate 3

15. Sampling Results Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web 117,871 Unknown Unknown Unknown Contacted Not applicable 11,045 6,350 682 Accessed 7,440 2,665 Not ascertained Ineligible 95,347 Screened out for inattentiveness Not applicable 583 664 Not applicable 3,013 3,022 3,021 579 Completed surveys Proxy responses AAPOR response rate 3

16. Sampling Results Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web 117,871 Unknown Unknown Unknown Contacted Not applicable 11,045 6,350 682 Accessed 7,440 2,665 Not ascertained Ineligible 95,347 Screened out for inattentiveness Not applicable 583 664 Not applicable 3,013 3,022 3,021 579 Completed surveys 16% Not applicable Not applicable 34% Proxy responses AAPOR response rate 3 13.4% Not applicable Not applicable Not applicable

17. Sampling Results Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web 117,871 Unknown Unknown Unknown Contacted Not applicable 11,045 6,350 682 Accessed 7,440 2,665 Not ascertained Ineligible 95,347 Screened out for inattentiveness Not applicable 583 664 Not applicable 3,013 3,022 3,021 579 Completed surveys 16% Not applicable Not applicable 34% Proxy responses AAPOR response rate 3

18. Sampling Results Random Digit Dial Health Cond. Panel General Pop. Panel Self-Recruited Web 117,871 Unknown Unknown Unknown Contacted Not applicable 11,045 6,350 682 Accessed 7,440 2,665 Not ascertained Ineligible 95,347 Screened out for inattentiveness Not applicable 583 664 Not applicable 3,013 3,022 3,021 579 Completed surveys 16% Not applicable Not applicable 34% Proxy responses AAPOR response rate 3 13.4% Not applicable Not applicable Not applicable

19. American Community Survey Self- Some Results (%) Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web Currently employed Four-year college or higher Annual household income < $30,000

20. American Community Survey Self- Some Results (%) Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 34.3 36.6* 41.9*** 51.6*** 66.4*** Currently employed Four-year college or higher Annual household income < $30,000

21. American Community Survey Self- Some Results (%) Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 34.3 36.6* 41.9*** 51.6*** 66.4*** Currently employed Four-year college or higher Annual household income < $30,000 13.2 29.1*** 41.1*** 25.3*** 47.2***

22. American Community Survey Self- Some Results (%) Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 34.3 36.6* 41.9*** 51.6*** 66.4*** Currently employed Four-year college or higher Annual household income < $30,000 13.2 29.1*** 41.1*** 25.3*** 47.2*** 39.4 44.3*** 34.2** 44.4*** 48.9*

23. American Community Survey Self- Disability Type Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web Hearing only Vision only Ambulatory only Cognitive only Multiple disabilities

24. American Community Survey Self- Disability Type Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 11.0 13.1 11.3 11.4 5.0 Hearing only Vision only Ambulatory only Cognitive only Multiple disabilities

25. American Community Survey Self- Disability Type Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 11.0 13.1 11.3 11.4 5.0 Hearing only 8.9 8.4 13.7 10.5 4.4 Vision only Ambulatory only Cognitive only Multiple disabilities

26. American Community Survey Self- Disability Type Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 11.0 13.1 11.3 11.4 5.0 Hearing only 8.9 8.4 13.7 10.5 4.4 Vision only 23.9 18.2 21.4 14.8 Ambulatory only 27.3 Cognitive only Multiple disabilities

27. American Community Survey Self- Disability Type Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 11.0 13.1 11.3 11.4 5.0 Hearing only 8.9 8.4 13.7 10.5 4.4 Vision only 23.9 18.2 21.4 14.8 Ambulatory only 27.3 28.8 15.7 16.2 18.4 Cognitive only 30.1 Multiple disabilities

28. American Community Survey Self- Disability Type Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 11.0 13.1 11.3 11.4 5.0 Hearing only 8.9 8.4 13.7 10.5 4.4 Vision only 23.9 18.2 21.4 14.8 Ambulatory only 27.3 28.8 15.7 16.2 18.4 Cognitive only 30.1 Multiple disabilities 27.5 44.6 37.3 44.9 33.2

29. American Community Survey Self- Disability Type Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 11.0 13.1 11.3 11.4 5.0 Hearing only 8.9 8.4 13.7 10.5 4.4 Vision only 23.9 18.2 21.4 14.8 Ambulatory only 27.3 28.8 15.7 16.2 18.4 Cognitive only 30.1 Multiple disabilities 27.5 44.6 37.3 44.9 33.2 *** *** *** ** Chi-Square Association Tests

30. American Community Survey Self- Some Results (%) Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web Currently employed Four-year college or higher Annual household income < $30,000

31. American Community Survey Self- Some Results (%) Random Digit Dial Health Cond. Panel General Pop. Panel Recruited Web 34.3 38.2** 41.0** 48.9*** 66.7*** Currently employed Four-year college or higher Annual household income < $30,000 13.2 29.9*** 41.3*** 25.3*** 48.7*** 39.4 43.4** 35.5* 44.7*** 48.3*

32. Reweighting on observables characteristics may not be enough Need a probability-based sampling frame Very expensive to do right Internet-based survey mode has barriers that need to be well- understood Recommend ations

33. Contact Information Andrew J. Houtenville, Ph.D. Andrew.Houtenville@unh.edu Institute on Disability University of New Hampshire 10 West Edge Drive, Suite 101 Durham, NH 03824 Website: www.ResearchOnDisability.org Technical Assistance: Disability.Statistics@unh.edu 866-538-9521

34. Thank you