Using System Paradata for Data Collection Operations

The provided content discusses the use of system paradata to target and evaluate data collection operations in the National Survey of College Graduates. It covers examples of paradata use, survey background, system paradata collected, and the standard data collection path for the survey. The types of paradata collected from major operations, such as outgoing mailings and incoming mailings, are also outlined. The survey, sponsored by NCSES at NSF, focuses on education, demographic, and employment information for college-educated science and engineering populations. It is a longitudinal survey with a total sample size of approximately 135,000 cases, targeting recent college graduates, women, minorities, and persons with disabilities.

• Paradata
• Data Collection
• Survey
• College Graduates
• NCSES

oluwas Follow

Uploaded on Feb 15, 2025 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

Presentation Transcript

1. Using System Paradata to Target and Evaluate Data Collection Operations Stephanie Coffey WSS Seminar 4/18/2016 U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

2. Overview National Survey of College Graduates (NSCG) Survey Background System Paradata Collected 2 Examples of Paradata Use Web instrument performance Intelligent Mail Barcoding Wrap-Up U.S. Department of Commerce Economics and Statistics Administration 2 U.S. CENSUS BUREAU

3. National Survey of College Graduates Sponsored by NCSES at NSF U.S. college-educated science & engineering population Education, demographic and employment info for S&E degrees and occupations Recent college graduates Women, minorities, persons with disabilities Longitudinal survey with four rotating panels New cohort is selected out of ACS for 1st interview Three follow-up interviews (every 2-3 years) Total sample size is approximately 135,000 cases U.S. Department of Commerce Economics and Statistics Administration 3 U.S. CENSUS BUREAU

4. National Survey of College Graduates Standard Data Collection Path (New Cohort) Weeks 0 6 Initial Invite Phase Weeks 7 - 11 1st Reminder Phase Weeks 12 - 17 Nonresp. FU Phase 1st Paper Q naire Web Invite #3 Prenotice Web Invites #1&2 CATI NRFU Begins Web Invite #4 Weeks 23 - end Final Reminder Phase Weeks 18 - 22 2nd Reminder Phase 2nd Paper Q naire Web Invite #5 Web Invite #6 U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

5. National Survey of College Graduates Types of Paradata Collected from Major Operations Outgoing Mailings: Mailed Web Invites, Reminders, Questionnaires Census National Processing Center (NPC) Mail-Out Date Undeliverable-as-Addressed (UAA) Status/Reason Upon Check-In at NPC USPS Integrated Postal Tracking Service (IPTS) New in 2015 First Scan into Postal System / Last Scan out of Postal System (out for delivery) UAA Status/Reason Upon USPS Determination Incoming Mailings: Return Paper Questionnaires Census NPC Return Date Upon Receipt and Check-In at NPC USPS IPTS First Scan Back into Postal System / Last Scan out of Postal System U.S. Department of Commerce Economics and Statistics Administration 5 U.S. CENSUS BUREAU

6. National Survey of College Graduates Types of Paradata Collected from Major Operations Web Access from Respondent Server-side Paradata Log-in Attempt Date/Time Log-in Success Status Time-in-Instrument Log-out Date/Time Submission Date/Time Outgoing Telephone Contact Attempt Centralized Call Center Case Management Call Attempt Date/Time Time in Instrument Outcome of Call Attempt Other Operations Not Included (Email Reminders, Incoming Telephone) U.S. Department of Commerce Economics and Statistics Administration 6 U.S. CENSUS BUREAU

7. Web Instrument Performance Event U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

8. 2013 Data Collection Web is initial mode offered in NSCG new cohort Web first used as production mode in 2013 Prior to 2013, web experiments, mode preference experiments NSCG achieves about 30% response rate in the first month NSCG has 6 month data collection First invites with usernames and passwords sent on 2/20 Arrival expected between 2/22-2/24 (Sat or Mon) U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

9. Web Interruption Afternoon of 2/25 Morning 2/26 External DHS simulated attack to check data security (unscheduled, unknown by Census) IT teams at Census: maintenance & applying updates Resulted in performance issues in the web instrument No advance warning of external simulations NSCG was informed by respondents calling TQA line: trouble logging in, delays in navigating through survey Question: Can we figure out who was affected? U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

10. Evidence in Server-Side Paradata service type id time (seconds) type2 NSCG paradata 10800084 1361842005 login 57 min NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login 60 min NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login 15 sec 5 min NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login 13 sec <1 min NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

11. Evidence in Server-Side Paradata service type id time (seconds) type2 NSCG paradata 10800084 1361842005 login 57 min NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout Patient respondents! NSCG paradata 10800159 1361843035 login 60 min NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login 15 sec 5 min NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login 13 sec <1 min NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

12. Evidence in Server-Side Paradata service type id time (seconds) type2 NSCG paradata 10800084 1361842005 login 57 min NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login 60 min NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout 2+ Logins in Short Time NSCG paradata 10800258 1361847610 login 15 sec 5 min NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login 13 sec <1 min NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

13. Evidence in Server-Side Paradata service type id time (seconds) type2 NSCG paradata 10800084 1361842005 login 57 min NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login 60 min NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login 15 sec 5 min Can t Successfully Log In NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login 13 sec <1 min NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

14. Operational Questions Major Concern: Burden / Respondent Frustration Time in instrument How long did individuals spend logged in? Server breakoffs Will sample persons come back to respond? Examine Login Events Login/Submit = 1 event Login/Submit/Logout = 1 event Login/Logout = 1 event If there was no Submission or Logout: Login alone = Server Breakoff Failed login = Failed Login U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

15. SubmitsMinutes in InstrumentBreakoffs Day 25-Feb 12AM Hour Mean 28.13 28.94 25.64 27.91 25.52 25.12 24.17 24.39 24.23 27.54 34.2 51.51 65.88 75.55 79.23 66.25 36.55 31.34 30.02 Maximum 45.72 13 14 66 74 93 5 1 1AM 10AM 11AM 12PM 1PM 2PM 3PM 4PM 5PM 6PM 7PM 8PM 9PM 10PM 11PM 56.3 81.48 69.67 63.43 105.45 77.1 93.57 75.2 143.48 110 117.4 157.85 159.18 237.38 242.18 228.8 162.13 103.03 17 20 27 42 59 79 72 148 186 206 294 399 349 337 340 392 419 529 243 97 56 Increase in the mean time-in-instrument 115 361 717 850 718 431 158 50 29 21 26-Feb 12AM 1AM 2AM U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

16. SubmitsMinutes in InstrumentBreakoffs Day 25-Feb 12AM Hour Mean 28.13 28.94 25.64 27.91 25.52 25.12 24.17 24.39 24.23 27.54 34.2 51.51 65.88 75.55 79.23 66.25 36.55 31.34 30.02 Maximum 45.72 13 14 66 74 93 5 1 1AM 10AM 11AM 12PM 1PM 2PM 3PM 4PM 5PM 6PM 7PM 8PM 9PM 10PM 11PM 56.3 81.48 69.67 63.43 105.45 77.1 93.57 75.2 143.48 110 117.4 157.85 159.18 237.38 242.18 228.8 162.13 103.03 17 20 27 42 59 79 72 148 186 206 294 399 349 337 340 392 419 529 243 97 56 Increase in the mean time-in-instrument More server breakoffs than submissions! 115 361 717 850 718 431 158 50 29 21 26-Feb 12AM 1AM 2AM U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

17. SubmitsMinutes in InstrumentBreakoffs Day 25-Feb 12AM Hour Mean 28.13 28.94 25.64 27.91 25.52 25.12 24.17 24.39 24.23 27.54 34.2 51.51 65.88 75.55 79.23 66.25 36.55 31.34 30.02 Maximum 45.72 13 14 66 74 93 5 1 1AM 10AM 11AM 12PM 1PM 2PM 3PM 4PM 5PM 6PM 7PM 8PM 9PM 10PM 11PM 56.3 81.48 69.67 63.43 105.45 77.1 93.57 75.2 143.48 110 117.4 157.85 159.18 237.38 242.18 228.8 162.13 103.03 17 20 27 42 59 79 72 148 186 206 294 399 349 337 340 392 419 529 243 97 56 Mean time in instrument nearly back to average 115 361 717 850 718 431 158 50 29 21 Server breakoffs still higher than expected 26-Feb 12AM 1AM 2AM U.S. Department of Commerce Economics and Statistics Administration 17 U.S. CENSUS BUREAU

18. Action & Results Determined time period of web issue: 6pm 2am Obtained approval to send special letter to affected cases Apology and reminder of sample person s importance to data quality Turnaround time: 3 business days from analysis to mailing Sent letter to all breakoff cases: Not an experiment Response rate of affected cases same as unaffected cases a month after mailing Would this be the same if this issue occurred late in data collection? (Early respondents = higher engagement?) Encouraged that we could be responsive to a new issue U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

19. Intelligent Mail Barcoding U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

20. Mailing Operation in NSCG NSCG heavily relies upon mailing operations Pre-notice Letter Web invites Paper Questionnaires Reminder Letters & Postcards Prepaid Incentives A respondent receiving all mailings in sequence: 11 mailings (plus any requested remails) Historically a low-information operation Was the address mailable? Did we receive something in return? Only have date of check-in at NPC U.S. Department of Commerce Economics and Statistics Administration 20 U.S. CENSUS BUREAU

21. Integrated Postal System Tracking USPS offers IPTS Data Tracks data through each step of postal delivery First scan at first post office Every transfer scan Last scan before delivery Scan to reroute due to UAAs Reasons for UAAs U.S. Department of Commerce Economics and Statistics Administration 21 U.S. CENSUS BUREAU

22. IPTS for NSCG Two points of interest in the NSCG How much earlier can we know about UAAs? Can send sample persons to locating earlier Can save cost through not mailing future mailings How much earlier can we know about return questionnaires? Can reduce burden by not contacting likely respondents Can reduce cost by not contacting likely respondents Discuss UAA timeliness today U.S. Department of Commerce Economics and Statistics Administration 22 U.S. CENSUS BUREAU

23. Implementation of IPTS Add barcodes to mailing packages Need outgoing barcodes to track progress and UAAs (letters and questionnaires) Need return barcodes to track progress of return questionnaires (questionnaires only) Redesign envelope to accommodate barcodes on printed letters and questionnaires Ensure barcode can be scanned by USPS Provide messaging on questionnaire return envelope to insert questionnaire correctly Ensure barcode is displayed U.S. Department of Commerce Economics and Statistics Administration 23 U.S. CENSUS BUREAU

24. Lag in UAA Identification UAAs can be identified at various points Delivery Point Verification before mailing (not mailed) During any scan through postal system When post office attempts to deliver mail After UAA is identified Mail package is rerouted through USPS back to NPC Upon arrival at NPC, UAAs are checked in as resources allow Report in Unified Tracking System (UTS) to track Each mail piece and its mail date When the UAA was reported by USPS When the UAA was checked in at NPC U.S. Department of Commerce Economics and Statistics Administration 24 U.S. CENSUS BUREAU

25. Lag in UAA Identification IPTS UAA Codes (from USPS) # with NPC Check-In % with NPC Check-In Mailing Type Prenotice Letter (PR) First Web Invite (1W) First Reminder (1R) Second Web Invite (2W) Third Web Invite (3W) Third Reminder (3R) Fourth Web Invite (4W) Fourth Reminder (4R) Mailing Week UAAs Week 0 Week 1 Week 2 Week 5 Week 6 Week 8 Week 12 Week 13 5697 5697 6125 2154 104 1463 992 545 5482 4553 96% 80% 0% 92% 88% 0% 93% 0% 0 1982 92 0 926 0 List of web invite letters and reminder mailings sent during Weeks 1 - 13 U.S. Department of Commerce Economics and Statistics Administration 25 U.S. CENSUS BUREAU

26. Lag in UAA Identification IPTS UAA Codes (from USPS) # with NPC Check-In % with NPC Check-In Mailing Type Prenotice Letter (PR) First Web Invite (1W) First Reminder (1R) Second Web Invite (2W) Third Web Invite (3W) Third Reminder (3R) Fourth Web Invite (4W) Fourth Reminder (4R) Mailing Week UAAs Week 0 Week 1 Week 2 Week 5 Week 6 Week 8 Week 12 Week 13 5697 5697 6125 2154 104 1463 992 545 5482 4553 96% 80% 0% 92% 88% 0% 93% 0% 0 1982 92 0 926 0 Number of UAAs recorded by the USPS For each data collection operation. U.S. Department of Commerce Economics and Statistics Administration 26 U.S. CENSUS BUREAU

27. Lag in UAA Identification IPTS UAA Codes (from USPS) # with NPC Check-In % with NPC Check-In Mailing Type Prenotice Letter (PR) First Web Invite (1W) First Reminder (1R) Second Web Invite (2W) Third Web Invite (3W) Third Reminder (3R) Fourth Web Invite (4W) Fourth Reminder (4R) Mailing Week UAAs Week 0 Week 1 Week 2 Week 5 Week 6 Week 8 Week 12 Week 13 5697 5697 6125 2154 104 1463 992 545 5482 4553 96% 80% 0% 92% 88% 0% 93% 0% 0 1982 92 0 926 0 Number & percentage of USPS records that also have a check-in record at NPC. U.S. Department of Commerce Economics and Statistics Administration 27 U.S. CENSUS BUREAU

28. Lag in UAA Identification IPTS UAA Codes (from USPS) # with NPC Check-In % with NPC Check-In Question: How long is the lag from when we could see this information in IPTS to when we see it in check- in records at NPC? Mailing Type Prenotice Letter (PR) First Web Invite (1W) First Reminder (1R) Second Web Invite (2W) Third Web Invite (3W) Third Reminder (3R) Fourth Web Invite (4W) Fourth Reminder (4R) Mailing Week UAAs Week 0 Week 1 Week 2 Week 5 Week 6 Week 8 Week 12 Week 13 5697 5697 6125 2154 104 1463 992 545 5482 4553 96% 80% 0% 92% 88% 0% 93% 0% 0 1982 92 0 926 0 Number & percentage of USPS records that also have a check-in record at NPC. U.S. Department of Commerce Economics and Statistics Administration 28 U.S. CENSUS BUREAU

29. Lag in UAA Identification IPTS UAA Codes (from USPS) # with NPC Check-In % with NPC Check-In Avg Lag Med Lag Max Lag Overlap? Mailing Type Prenotice Letter (PR) First Web Invite (1W) First Reminder (1R) Second Web Invite (2W) Third Web Invite (3W) Third Reminder (3R) Fourth Web Invite (4W) Fourth Reminder (4R) Mailing Week UAAs Week 0 Week 1 Week 2 Week 5 Week 6 Week 8 Week 12 Week 13 5697 5697 6125 2154 104 1463 992 545 5482 4553 96% 80% 0% 92% 88% 0% 93% 0% 19 18 --- 28 23 --- 29 --- 19 15 --- 27 23 --- 30 --- 189 165 --- 119 117 --- 81 --- Yes Yes --- Yes Yes --- Yes --- 0 1982 92 0 926 0 Average, median, and max lag between USPS and NPC U.S. Department of Commerce Economics and Statistics Administration 29 U.S. CENSUS BUREAU

30. Action & Results Clear that data from the USPS scanning More timely Cost neutral, if not cheaper Still some questions about UAA data Understand lag between mailing and USPS UAA Better understand NPC check-in process Records we cannot currently identify Monitor and use USPS UAA data in 2017 Implement return questionnaire scan data in 2017 Monitor and evaluate Place potential respondents on hold in other modes until we determine validity of complete U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU

31. Summary High volume of system-generated paradata Need to know what is available Need to know who provides it so you can ask Possible operational changes to collect or use (envelopes) May need new systems to manage or track paradata Can be monitored to improve data collection Efficiency and timeliness Might be able to help answer research questions May need to combine data across modes to answer research questions Important to keep in mind limitations of system paradata U.S. Department of Commerce Economics and Statistics Administration 31 U.S. CENSUS BUREAU

32. Contact Information stephanie.coffey@census.gov U.S. Department of Commerce Economics and Statistics Administration U.S. CENSUS BUREAU