Machine Transcription for Call Center Efficiency
D
e
v
e
l
o
p
i
n
g
a
M
a
n
u
a
l
S
p
a
n
i
s
h
T
r
a
n
s
c
r
i
p
t
i
o
n
B
a
s
e
l
i
n
e
t
o
E
v
a
l
u
a
t
e
M
a
c
h
i
n
e
T
r
a
n
s
c
r
i
p
t
i
o
n
o
f
C
a
l
l
C
e
n
t
e
r
C
a
l
l
s
Marcus Berger, Betsarí Otero Class, Crystal Hernandez
Center for Behavioral Science Methods, U.S. Census Bureau
FedCASIC, Virtual Conference
April 11, 2023
1
Disclaimer: This presentation is released to inform interested
parties of research and to encourage discussion. The views
expressed are those of the authors and not those of the U.S.
Census Bureau.
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
Advantages of Machine Transcription
•
Machine transcriptions are useful for keeping records of otherwise
unwritten products
•
Podcasts, interviews, other recordings
•
Legal depositions
•
Call Centers
•
Transcriptions are needed for machine learning models to improve processes like:
•
Finding the correct scripted answer to a caller’s question
•
Identify new questions that callers might have
•
Monitor agent performance and script readability
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
2
Machine Transcription
•
As part of 2020 Census operations, we collected audio data from calls
made to our Census Questionnaire Assistance (CQA) Centers
•
These are call centers across the country with live agents providing assistance
in English and 12 non-English languages
•
The different available languages had separate phone lines
•
Due to the volume of calls, manual transcription is not feasible
•
Machine transcription will aid in analysis of these calls to automate processes
and improve customer experience
•
We have another presentation on this tomorrow
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
3
Developing a Machine Transcription Base
•
We tested several machine transcription models to determine which
model offered the best transcriptions in English and Spanish
•
We needed a high quality, human generated transcript to use as a
baseline to compare against
•
This provides a ground truth against which to categorize errors
•
We compare manual and machine transcription based on word error rate
•
As part of an earlier phase of this project, we found the best model
for calls in English
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
4
Developing a Machine Transcription Base
•
Spanish Transcription: We manually transcribed segments of Spanish
language calls made to CQA Centers during 2020 Census operations
•
Pulled from over 400,000 total Spanish language calls
•
Over 350,000 Stateside
•
Over 50,000 Puerto Rico
•
30 second call segments
•
1 hour of audio from stateside callers and Customer Service Representatives
(CSRs) – Total of 120 segments
•
15 minutes of audio from the Puerto Rican Spanish line – Total of 30 segments
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
5
Selecting Call Segments
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
6
Incoming Call (respondents)
Customer Service Rep
30 second segments
Developing a Machine Transcription Base
•
We had 3 Spanish speaking transcribers work to manually transcribe
each of these 30 second segments
•
Each segment was manually transcribed by one transcriber
•
Once complete, each segment was reviewed by a second transcriber
•
Any disagreement between transcribers was adjudicated by the third
transcriber
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
7
Guidelines for Transcription
•
We did need to follow certain conventions to match the machine
transcription
•
No use of punctuation
•
Proper use of accent marks
•
Spelling out the names of numbers and letters
•
e.g. “Twenty twenty” instead of “2020”
•
Filler words
•
We established a shared document with our own standardized spelling of filler words
•
E.g. “Umm”, “Hmm”, “Ahh”
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
8
Example of Manual vs. Machine Transcription
Manual Spanish Transcript
pero acá arriba dice seleccione
una o más casillas y anote los
orígenes para este Censo los
orígenes hispanos no son razas
entonces no puedo poner hispano
don-
Machine Transcription
pero
oh
acá
riba
dice
selecciona
una
masca silla la note
los
orígenes para este censo los
orígenes hispanos no son razas
entonces no
podrá
poner hispano
___
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
9
Language Misidentification
•
We did encounter some call segments on the Spanish line that were
not in Spanish
•
Some call segments were in English, others had no audio at all
•
In these cases, we searched for adjacent call segments and
transcribed the nearest segment with Spanish audio
•
In cases where this was not possible, we replaced the segment with
data from a new call
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
10
Difficulties encountered
•
Some calls included code-switching, both within and across sentences
•
Different dialects among speakers from different backgrounds
•
Capturing stutters and incomplete words
•
Different machine transcription models might handle stutters differently
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
11
Procedure Summary
•
Select a representative array of segments of audio
•
Ensure enough audio is selected to be able to train the model
•
Round 1 of Transcription: Initial transcription
•
Develop standardized spellings for filler words
•
Ensure all transcribers understand conventions and formatting to match the
machine transcription model
•
Round 2 of Transcription: Review of transcription
•
Ensure each transcription is reviewed by a second transcriber
•
Round 3 of Transcription: Adjudication (if necessary)
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
12
Future Research
•
We would like to continue this research in other non-English, non-
Spanish languages
•
Would any of these procedures change in character-based languages?
•
E.g. spelling out names of letters and numbers
•
For languages with multiple writing systems (e.g. Simplified and Traditional
Chinese), is there a preference for which system is used for base
transcription?
•
While we used this process for transcription, can the same process be
used to establish a base for machine translation?
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
13
Thank you!
Marcus Berger
Research Sociolinguist
Language and Cross-Cultural Research Group
marcus.p.berger@census.gov
The presentation has been reviewed for disclosure avoidance
and approved under CBDRB-FY23-CBSM002-011
14
Thank you and hello, my name is Marcus Berger and I am a sociolinguist in the Center for Behavioral Science Methods at the US Census Bureau
Today I will be presenting on Developing a Manual Spanish Transcription Baseline to Evaluate Machine Transcription of Call Center Calls
I’d like to start by acknowledging my coauthors on this project, Betsari Otero Class and Crystal Hernandez who both put in a lot of work towards to get this project done.
Explore the benefits of machine transcription in call centers for improving processes like scripted responses, identifying new questions, and monitoring agent performance. Learn how developing a transcription baseline helps evaluate machine transcription accuracy, enhancing customer experience. Discover the role of machine transcription in the 2020 Census operations and how it aids in automating processes to handle high call volumes effectively.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Developing a Manual Spanish Developing a Manual Spanish Transcription Baseline to Evaluate Transcription Baseline to Evaluate Machine Transcription of Call Center Machine Transcription of Call Center Calls Calls Marcus Berger, Betsar Otero Class, Crystal Hernandez Center for Behavioral Science Methods, U.S. Census Bureau FedCASIC, Virtual Conference April 11, 2023 Disclaimer: This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not those of the U.S. Census Bureau. The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 1
Advantages of Machine Transcription Machine transcriptions are useful for keeping records of otherwise unwritten products Podcasts, interviews, other recordings Legal depositions Call Centers Transcriptions are needed for machine learning models to improve processes like: Finding the correct scripted answer to a caller s question Identify new questions that callers might have Monitor agent performance and script readability The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 2
Machine Transcription As part of 2020 Census operations, we collected audio data from calls made to our Census Questionnaire Assistance (CQA) Centers These are call centers across the country with live agents providing assistance in English and 12 non-English languages The different available languages had separate phone lines Due to the volume of calls, manual transcription is not feasible Machine transcription will aid in analysis of these calls to automate processes and improve customer experience We have another presentation on this tomorrow The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 3
Developing a Machine Transcription Base We tested several machine transcription models to determine which model offered the best transcriptions in English and Spanish We needed a high quality, human generated transcript to use as a baseline to compare against This provides a ground truth against which to categorize errors We compare manual and machine transcription based on word error rate Manual Baseline Example Machine Transcription Example U.S. Census Bureau U.S. Sentence Bureau As part of an earlier phase of this project, we found the best model for calls in English The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 4
Developing a Machine Transcription Base Spanish Transcription: We manually transcribed segments of Spanish language calls made to CQA Centers during 2020 Census operations Pulled from over 400,000 total Spanish language calls Over 350,000 Stateside Over 50,000 Puerto Rico 30 second call segments 1 hour of audio from stateside callers and Customer Service Representatives (CSRs) Total of 120 segments 15 minutes of audio from the Puerto Rican Spanish line Total of 30 segments The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 5
Selecting Call Segments Incoming Call (respondents) Customer Service Rep 30 second segments The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 6
Developing a Machine Transcription Base We had 3 Spanish speaking transcribers work to manually transcribe each of these 30 second segments Each segment was manually transcribed by one transcriber Once complete, each segment was reviewed by a second transcriber Any disagreement between transcribers was adjudicated by the third transcriber The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 7
Guidelines for Transcription We did need to follow certain conventions to match the machine transcription No use of punctuation Proper use of accent marks Spelling out the names of numbers and letters e.g. Twenty twenty instead of 2020 Filler words We established a shared document with our own standardized spelling of filler words E.g. Umm , Hmm , Ahh The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 8
Example of Manual vs. Machine Transcription Manual Spanish Transcript pero ac arriba dice seleccione una o m s casillas y anote los or genes para este Censo los or genes hispanos no son razas entonces no puedo poner hispano don- Machine Transcription pero oh ac riba dice selecciona una masca silla la note los or genes para este censo los or genes hispanos no son razas entonces no podr poner hispano ___ The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 9
Language Misidentification We did encounter some call segments on the Spanish line that were not in Spanish Some call segments were in English, others had no audio at all In these cases, we searched for adjacent call segments and transcribed the nearest segment with Spanish audio In cases where this was not possible, we replaced the segment with data from a new call The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 10
Difficulties encountered Some calls included code-switching, both within and across sentences Different dialects among speakers from different backgrounds Capturing stutters and incomplete words Different machine transcription models might handle stutters differently The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 11
Procedure Summary Select a representative array of segments of audio Ensure enough audio is selected to be able to train the model Round 1 of Transcription: Initial transcription Develop standardized spellings for filler words Ensure all transcribers understand conventions and formatting to match the machine transcription model Round 2 of Transcription: Review of transcription Ensure each transcription is reviewed by a second transcriber Round 3 of Transcription: Adjudication (if necessary) The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 12
Future Research We would like to continue this research in other non-English, non- Spanish languages Would any of these procedures change in character-based languages? E.g. spelling out names of letters and numbers For languages with multiple writing systems (e.g. Simplified and Traditional Chinese), is there a preference for which system is used for base transcription? While we used this process for transcription, can the same process be used to establish a base for machine translation? The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 13
Thank you! Marcus Berger Research Sociolinguist Language and Cross-Cultural Research Group marcus.p.berger@census.gov The presentation has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM002-011 14
