Evaluating Collaborative Editing of AI-Generated Live Subtitles in German University Lectures

Slide Note

Explore the use of collaborative editing by students and non-professionals to correct AI-generated live subtitles in German university lectures. The study investigates the impact of subtitle insertion timing on accuracy and cognitive workload, providing valuable insights for improving accessibility in educational settings.

misha Follow

Uploaded on May 13, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

Evaluating Collaborative Editin Collaborative Editing g of AI AI- -Generated Generated Live Subtitle by Non Non- -Professional Professionals s in German University Lectures Live Subtitles s

Awesome Group Picture from Lecco Research Project: SHUFFLE Patricia Piskorek, Nadine Sienel, Korbinian Kuhn Colleagues: Verena Kersken, Gottfried Zimmermann, Niklas Egger, Benedikt Reuter

Introduction Starting Point: Subtitles for lectures + Deaf and hard of hearing, but also other students benefit. - Professional subtitlers are expensive and not always available. Approach: Use of speech recognition + - Automated, inexpensive and always available. Insufficient accuracy (<98%) makes manual correction necessary. Tool for manual, collaborative corrections by students User test as a basis for development

Methodology Research Question: In the context of a lecture, can students collaboratively correct AI-generated subtitles? Usertest ( Wizard of Oz ): Students: Lecture: AI: Collaboration: Students and Non-Professionals Zoom Video Call Microsoft Azure Google Sheets

Methodology: Setup

Methodology: Conditions Subtitles were inserted either simultaneously or realistically delayed Audio: Cond. 1: Hello, we are the SHUFFLE-team Hello, we are the SHUFFLE-team Cond. 2:

Results Yay statistics!

1. Research question Does the time of insertion of the subtitles impact the accuracy of manual corrections? Word Error Rate ????????????? + ????????? + ?????????? ????? ?? ?????????????? ??? = 100

Simultaneous insertion of subtitles has a positive influence on the resulting accuracy of the subtitles

2. Research question Does the time of insertion influence the perceived cognitive workload of the participants? NASA Raw Task Load NASA Raw Task Load Index (RTLX) Index (RTLX)

Delayed insertion of the subtitles has a negative influence on the perceived cognitive workload of the participants. RTLX Stress indications

3. Research question Can participants follow the content of the video while correctingsubtitles? Questionnaire

Simultaneous insertion of the subtitles has a positive influence on the ability to follow the content. Participants

Key Findings Word Error Rate improves through manual editing Delayed editing is stressful and less effective

Limitations / Open Questions Word Error Rate What accuracy is actually needed for text understanding? Is the WER suitableforGerman language? What results do other calculation methods indicate? User interface How can a specialized user interface support collaborative editing? What are necessary delays for simultaneous editing? Can we call it LIVE?

Whats next? Developing a fully open-source subtitle editor called MELVIN -> Further user studies based on this editor We are also here in Lecco: Visit us at our Tech-Demo!