Using Machine Learning for Call Center Data Analysis

Applying machine learning Applying machine learning language models to answer a language models to answer a caller s question. caller s question. Monica Puerto*, Elizabeth May Nichols+, Shaun S Genter+, Brian Francis Sadacca* Accenture Federal Services* , Center for Behavioral Science Methods+, U.S. Census Bureau FEDCASIC APRIL 2023 Disclaimer: This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not those of the U.S. Census Bureau. Transcript text data is heavily paraphrased. The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012 1

Two processes on Call Center data using machine learning language models I have not received my paper questionnaire ? questionnaire ? questionnaire ? I have not received my paper received my paper I have not Process 1: Transcribe audio into text to use in NLP models Raw audio Call transcripts Pre-processed audio Call Topics Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding online is safe. The US Census Bureau is required, by law, to protect, your information, information, to protect, your information, to protect, your information, Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding online is safe. The US Census Bureau is required, by law, to protect, your required, by law, Census, Bureau, is required, by law, Transcript Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding online is safe. The US Census Bureau is is, safe, The, US, Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding, online Process 2: IVR I have not received my paper questionnaire? Automatically link calls to reference documents Call transcript agent The presentation has been reviewed for disclosure avoidance

Going from text to mathematic values to extract insight & document linkage The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012

Clustering transcripts mortgage (enumeration) The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012 4

Clustering transcript embeddings into topics Race / origin Snowbird / multiple residence Special living situation Census id / internet issues General questions moved No computer/ send paper Address issues Did not get questionnaire / send paper Census visit Verify computer response No confirmation number Completed census why am I receiving mail Enumeration Verify paper response The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012

Finding the best/ most relevant FAQ to answer a caller s concern with machine learning Caller transcript: I need help responding to the Census can I ask my daughter? FAQ Title FAQ Keywords FAQ Content Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding online is safe. The US Census Bureau is required, by law, to protect, your information, to protect, your information, to protect, your information, information, Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding online is safe. The US Census Bureau is required, by law, Census Bureau is required, by law, required, by law, to protect, your Can I ask someone else to help me respond? someone, else, behalf, assist, help, proxy, guardian, poa, power, attorney You can ask someone to help you complete the 2020 Census. Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding online is safe. The US is safe. The US Census Bureau is Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding online FAQ embeddings Is it safe to respond online? safe, online, internet, security, protect, data, information, hack, Responding online is safe. The U.S. Census Bureau is required by law to protect your information. Combined, filtered, FAQ Fields Nearest FAQ to search query (cosine similarity): Can I ask someone to help me respond? Cosine distance : .81 Raw FAQ data The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012

Creating a labeled dataset to evaluate accuracy Transcript Model Prediction Human Label Correct? I don t have a computer how do I respond? FAQ-1 FAQ-1 Yes FAQ-11 No Can I get a paper form? FAQ-10 60% accuracy How do I answer this race question? FAQ-7 FAQ-7 Yes FAQ-9 No I am America but I don t know my origin. FAQ-8 This is a vacant address. FAQ-45 FAQ-45 Yes The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012 7

How can we speed up labeling? Especially if we have to label hundreds of calls? I have two homes, I am a snowbird how do I respond? I don t have a computer how do I respond? I have moved. Can I get a paper form? I am a landlord do I count my tenants? How do I answer this race question? I am American but I don t know my origin. This is a vacant address. Examples of caller concerns The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012

Cluster label data and reference materials to help labeling I have two homes, I am a snowbird how do I respond? I don t have a computer how do I respond? Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding online is safe. The US Census Bureau is required, by law, to protect, your information, address of the place where you address of the place where you required, by law, to protect, your to protect, your Is, it, safe, to, respond online, safe, online, internet, security, protect, data, information, hack, Responding online is safe. The US Census Bureau is required, by law, to protect, your information, were staying on april first 2020 were staying on april first 2020 information, information, count yourself at your usual residence that is where you live and sleep most of the time if you spend equal time at both places use the equal time at both places use the is safe. The US Census Bureau is Census, Bureau, is required, by law, count yourself at your usual residence that is where you live and sleep most of the time if you spend information, hack, Responding online Responding, online is, safe, The, US, Is, it, safe, to, respond online, safe, online, internet, security, protect, data, protect, data, information, hack, Is, it, safe, to, respond online, safe, online, internet, security, I have moved. Can I get a paper form? I am a landlord do I count my tenants? How do I answer this race question? I am American but I don t know my origin. This is a vacant address. 300 different FAQs into 30 different categories 10 examples per topic from 100 topics = 1000 examples The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012

An average FAQ NLP Model identified FAQs What if I know the person s race but not their origin? how do i answer the race question? i am _____ but i don't see that on the race question what do i do? 2 i don't speak English well how do i respond? Other ~300 FAQS to choose from 4 Human identified FAQs 5 1 1 What if I know the person s race but not their origin? Same category (race / origin FAQ) and similar words used. Eg : hispanic origin, race, etc. The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012 10

Takeaways / Lessons Learned You can use clustering survey responses to identify common topics / themes that happened during operations. Cluster sampling calls/responses and clustering reference materials/FAQs greatly improved labeling efficiency and speed for document linkage modeling. We can figure out the best FAQs to answer a caller s question by semantically linking questions and reference materials through machine learning. We can provide the agent some prior context about the call and next best FAQ choices. We can understand strengths and shortcomings among different NLP models or the data itself because of the labeled dataset. Your model is as good as your data. We realized we will need to dedupe some FAQs whose responses are too similar and update language. The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-012 11