Analysis of 2020 Census Audio Recordings Using Machine Learning

In this study, Joanna Fane Lineback, Elizabeth Nichols, and Brian F. Sadacca discuss the machine learning approach used to analyze 2020 Census audio recordings. The project aims to develop models that enhance the call experience for callers and customer service representatives (CSRs). Operational goals include flagging difficult calls, sentiment analysis, and identifying new FAQs through natural language processing (NLP) and machine learning techniques.

• Machine Learning
• Audio Analysis
• 2020 Census
• NLP
• Call Experience

mrocc Follow

Uploaded on Oct 08, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Analyzing 2020 Census Audio Recordings: Analyzing 2020 Census Audio Recordings: A machine learning approach A machine learning approach Joanna Fane Lineback and Elizabeth Nichols U.S. Census Bureau Brian F. Sadacca Accenture Federal Services Federal Computer Assisted Survey Information Collection (FedCASIC) Workshop April 5-6, 2022 The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau. CBDRB-FY22-CBSM002-020 1

2. Outline Outline Background Objectives Approach Findings Next steps 2

3. 2020 Census 2020 Census Short questionnaire Primarily online using internet self-response instrument (ISR) -> available in 13 languages Other modes of data collection: Mail In-person Phone Census Questionnaire Assistance (CQA): For questions Respond over the phone 3

4. 2020 CQA Operation 2020 CQA Operation Number of recordingsLanguage 5,156,000 412,000 54,500 11,500 8,600 8,000 Vietnamese 7,500 7,200 5,700 4,300 3,200 2,600 Portuguese 2,500 2,100 2,000 1,500 English Spanish Puerto Rican Spanish Mandarin Cantonese Interactive Voice Response (IVR) Customer Service Representatives (CSRs) 11 call centers, 7,300 CSRs With permission, calls were recorded -> about 5.7 million audio recordings Korean Russian Puerto Rican English Arabic Haitian Creole Tagalog Japanese Polish French 4

5. Objectives Objectives Project long-term objective: Using CQA audio, develop machine learning models that can be used to enhance overall call experience for callers and CSRs. Today s objective: Share progress to date 5

6. Operational goals Operational goals Flag difficult calls Identify features of negative calls Track sentiment dynamics Call sentiment Label call dispositions Search FAQs Identify new FAQs Call topic 6

7. Approach Pre-process audio Use natural language processing (NLP) and other machine learning models: Transcribe audio to text Emotion recognition Topic modeling 7

8. Pipeline Pipeline may I record this call for quality assurance purposes? 5 1 2 Raw audio Pre-processed audio Transcript Call similarity, clusters 6 4 3 Approaches currently used in pipeline: 1. Normalize volume 2. Transcribe audio 3. Identify emotion in audio 4. Identify emotion in text 5. Identify and group similar calls 6. Identify topics discussed in groups of similar calls Call topics, nearest FAQ Emotion scoring

9. Pre Pre- -process audio process audio 1 Raw audio Pre-processed audio Approaches currently used in pipeline: 1. Normalize volume

10. Preparing audio Preparing audio Improve the sound quality Adjust the volume Split left (CSR) and right (caller) channels 10

11. Transcribe audio Transcribe audio may I record this call for quality assurance purposes? 1 2 Raw audio Pre-processed audio Transcript Approaches currently used in pipeline: 1. Normalize volume 2. Transcribe audio

12. Sample transcripts* Sample transcripts* Manual Transcript: Hello you have reached the twenty-twenty Censusquestionnaire assistance line, my name is John do I have your permission to continue recording this call for quality assurance purposes? Thank you, how may I help you? Automated Example 1: Hello you've reached the twenty-twenty Censusquestion in your assistance line my name is John do I have your permission to continue recording this call for quality assurance purposes? Thank you and how may I help you? Automated Example 2: Hello you have reached the twenty-twenty sent thisquestion the assistants I, my name is John do I have your permission to continue recording this call for quality assurance purposes? Thank you, how may I help you? *Made up transcripts 12

13. Preliminary findings Preliminary findings - - transcription transcription Newer models are better Best model in one setting not best model in all settings 13

14. Summary of transcription rate to date by call type Summary of transcription rate to date by call type Total machine transcribed (%) Total machine transcribed Total recorded 1,570,000 General assistance 2,241,000 Enumerate 11,000 Complaint 34,500 Technical issue 5,700 Abusive caller 1,100 Threat 219,000 None selected 1,000 Media call 1,800 Robo call 258,000 Other Type 2 .5 74 15 91 91 31,000 10,500 8,100 5,200 5,200 1,000 0 0 0 0 0 0 0 0 14

15. Sentiment recognition Sentiment recognition may I record this call for quality assurance purposes? 1 2 Raw audio Pre-processed audio Transcript 4 3 Approaches currently used in pipeline: 1. Normalize volume 2. Transcribe audio 3. Identify emotion in audio 4. Identify emotion in text Emotion scoring

16. Sentiment examples For fictional text put into production model and model generated score Text: I think your survey was fantastic Score:0.987 Text:I d like to receive a paper form Score: 0.018 Text: My browser froze when entering my data Score: -0.907 16

17. Preliminary finding Preliminary finding emotion recognition emotion recognition Tested across 3,000 transcripts Combining output of multiple models yielded most accurate predictions Mislabeled calls based on score appear to often be human mis- labeling POSITIVE NEGATIVE 17

18. Call topic modeling Call topic modeling may I record this call for quality assurance purposes? 5 1 2 Raw audio Pre-processed audio Transcript Call similarity, clusters 6 4 3 Approaches currently used in pipeline: 1. Normalize volume 2. Transcribe audio 3. Identify emotion in audio 4. Identify emotion in text 5. Identify and group similar calls 6. Identify topics discussed in groups of similar calls Call topics, nearest FAQ Emotion scoring

19. Topic examples Fictional text- Text: What will happen if I refuse to complete the Census Text:I d like to receive a paper form Text: My browser froze when entering my data 19

20. Preliminary findings Preliminary findings topic modelling topic modelling Newer deep learning language models better group similar calls together for both large topics (e.g. technical assistance vs residency questions) and more specific sub-topics Example 1: Residence Example 2: Race and Hispanic origin 'white', 'box', 'race', 'hispanic', 'origin', 'filling' 'white', 'race', 'german', 'origin', 'american', 'irish' 'canadian', 'citizen', 'canada', 'months', 'citizens 'residence , 'live', 'state', 'address', 'winter', 'new 'florida', 'months', 'residence', 'home', 'winter', 'resident' 20

21. Next steps Next steps Improve existing functionality Improve existing functionality Improve call labels Implement speed improvements to pre-processing and transcription Evaluate pipeline for non-English languages Test transcription, emotion recognition on non-English languages Test methods for multi-language topic modeling 21