Evaluating Collaborative Editing of AI-Generated Live Subtitles in German University Lectures

Slide Note
Embed
Share

Explore the use of collaborative editing by students and non-professionals to correct AI-generated live subtitles in German university lectures. The study investigates the impact of subtitle insertion timing on accuracy and cognitive workload, providing valuable insights for improving accessibility in educational settings.


Uploaded on May 13, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Evaluating Collaborative Editin Collaborative Editing g of AI AI- -Generated Generated Live Subtitle by Non Non- -Professional Professionals s in German University Lectures Live Subtitles s

  2. Awesome Group Picture from Lecco Research Project: SHUFFLE Patricia Piskorek, Nadine Sienel, Korbinian Kuhn Colleagues: Verena Kersken, Gottfried Zimmermann, Niklas Egger, Benedikt Reuter

  3. Introduction Starting Point: Subtitles for lectures + Deaf and hard of hearing, but also other students benefit. - Professional subtitlers are expensive and not always available. Approach: Use of speech recognition + - Automated, inexpensive and always available. Insufficient accuracy (<98%) makes manual correction necessary. Tool for manual, collaborative corrections by students User test as a basis for development

  4. Methodology Research Question: In the context of a lecture, can students collaboratively correct AI-generated subtitles? Usertest ( Wizard of Oz ): Students: Lecture: AI: Collaboration: Students and Non-Professionals Zoom Video Call Microsoft Azure Google Sheets

  5. Methodology: Setup

  6. Methodology: Conditions Subtitles were inserted either simultaneously or realistically delayed Audio: Cond. 1: Hello, we are the SHUFFLE-team Hello, we are the SHUFFLE-team Cond. 2:

  7. Results Yay statistics!

  8. 1. Research question Does the time of insertion of the subtitles impact the accuracy of manual corrections? Word Error Rate ????????????? + ????????? + ?????????? ????? ?? ?????????????? ??? = 100

  9. Simultaneous insertion of subtitles has a positive influence on the resulting accuracy of the subtitles

  10. 2. Research question Does the time of insertion influence the perceived cognitive workload of the participants? NASA Raw Task Load NASA Raw Task Load Index (RTLX) Index (RTLX)

  11. Delayed insertion of the subtitles has a negative influence on the perceived cognitive workload of the participants. RTLX Stress indications

  12. 3. Research question Can participants follow the content of the video while correctingsubtitles? Questionnaire

  13. Simultaneous insertion of the subtitles has a positive influence on the ability to follow the content. Participants

  14. Key Findings Word Error Rate improves through manual editing Delayed editing is stressful and less effective

  15. Limitations / Open Questions Word Error Rate What accuracy is actually needed for text understanding? Is the WER suitableforGerman language? What results do other calculation methods indicate? User interface How can a specialized user interface support collaborative editing? What are necessary delays for simultaneous editing? Can we call it LIVE?

  16. Whats next? Developing a fully open-source subtitle editor called MELVIN -> Further user studies based on this editor We are also here in Lecco: Visit us at our Tech-Demo!

  17. Thank you! Questions? Comments? Feedback? Patricia Piskorek Nadine Sienel Korbinian Kuhn

Related


More Related Content