Evaluating Collaborative Editing of AI-Generated Live Subtitles in German University Lectures
Explore the use of collaborative editing by students and non-professionals to correct AI-generated live subtitles in German university lectures. The study investigates the impact of subtitle insertion timing on accuracy and cognitive workload, providing valuable insights for improving accessibility in educational settings.
- Collaborative Editing
- AI-Generated Subtitles
- German University Lectures
- Accessibility
- Research Study
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Evaluating Collaborative Editin Collaborative Editing g of AI AI- -Generated Generated Live Subtitle by Non Non- -Professional Professionals s in German University Lectures Live Subtitles s
Awesome Group Picture from Lecco Research Project: SHUFFLE Patricia Piskorek, Nadine Sienel, Korbinian Kuhn Colleagues: Verena Kersken, Gottfried Zimmermann, Niklas Egger, Benedikt Reuter
Introduction Starting Point: Subtitles for lectures + Deaf and hard of hearing, but also other students benefit. - Professional subtitlers are expensive and not always available. Approach: Use of speech recognition + - Automated, inexpensive and always available. Insufficient accuracy (<98%) makes manual correction necessary. Tool for manual, collaborative corrections by students User test as a basis for development
Methodology Research Question: In the context of a lecture, can students collaboratively correct AI-generated subtitles? Usertest ( Wizard of Oz ): Students: Lecture: AI: Collaboration: Students and Non-Professionals Zoom Video Call Microsoft Azure Google Sheets
Methodology: Conditions Subtitles were inserted either simultaneously or realistically delayed Audio: Cond. 1: Hello, we are the SHUFFLE-team Hello, we are the SHUFFLE-team Cond. 2:
Results Yay statistics!
1. Research question Does the time of insertion of the subtitles impact the accuracy of manual corrections? Word Error Rate ????????????? + ????????? + ?????????? ????? ?? ?????????????? ??? = 100
Simultaneous insertion of subtitles has a positive influence on the resulting accuracy of the subtitles
2. Research question Does the time of insertion influence the perceived cognitive workload of the participants? NASA Raw Task Load NASA Raw Task Load Index (RTLX) Index (RTLX)
Delayed insertion of the subtitles has a negative influence on the perceived cognitive workload of the participants. RTLX Stress indications
3. Research question Can participants follow the content of the video while correctingsubtitles? Questionnaire
Simultaneous insertion of the subtitles has a positive influence on the ability to follow the content. Participants
Key Findings Word Error Rate improves through manual editing Delayed editing is stressful and less effective
Limitations / Open Questions Word Error Rate What accuracy is actually needed for text understanding? Is the WER suitableforGerman language? What results do other calculation methods indicate? User interface How can a specialized user interface support collaborative editing? What are necessary delays for simultaneous editing? Can we call it LIVE?
Whats next? Developing a fully open-source subtitle editor called MELVIN -> Further user studies based on this editor We are also here in Lecco: Visit us at our Tech-Demo!
Thank you! Questions? Comments? Feedback? Patricia Piskorek Nadine Sienel Korbinian Kuhn