Language Identification System for DARPA RATS Program Evaluation

The Patrol Team Language Identification System was developed for DARPA RATS Program to create algorithms and software for speech tasks in noisy environments. The system involved analyzing datasets, implementing subsystems for language identification, and evaluating performance across different languages and durations.

• Language Identification
• DARPA RATS
• Speech Recognition
• Data Analysis

vero Follow

Uploaded on Aug 25, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Patrol Team Language Identification System for DARPA RATS P1 Evaluation Pavel Matejka1, Oldrich Plchot1, Mehdi Soufifar1, Ondrej Glembek1, Luis Fernando D Haro1, Karel Vesely1, Frantisek Grezl1, Jeff Ma2, Spyros Matsoukas2, and Najim Dehak3 1Brno University of Technology, Speech@FIT and IT4I Center of Excellence, Czech 2Raytheon BBN Technologies, Cambridge, MA, USA 3MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA matejkap@fit.vutbr.cz, smatsouk@bbn.com, najim@csail.mit.edu 1 Pavel Matejka Patrol LID System for DARPA RATS P1 Evaluation

2. Outline About DARPA RATS program Datasets and task description Subsystems with analysis Fusion and Results Conclusion 2 Pavel Matejka Patrol LID System for DARPA RATS P1 Evaluation

3. DAPRA RATS Program RATS = Robust Automatic Trascription of Speech Goal : create algorithms and software for performing the following tasks on speech-containing signals received over communication channels that are extremely noisy and/or highly distorted. Tasks : Speech Activity Detection Keyword Spotting Language Identification Speaker Identification Data collector : LDC Evaluation by SAIC Performer: PATROL Team led by BBN 3 Pavel Matejka Patrol LID System for DARPA RATS P1 Evaluation

4. Data Specification Languages: Dari, Levantine Arabic, Urdu, Pashtu, Farsi >10 out of set languages Durations: 120s, 30s, 10s, 3s Telephone conversations retransmitted over 8 noisy radio communication channels [marked as A-H] Available: collections of 2-min audio samples LDC2011E95 split to train and dev by SAIC LDC2011E111 split to train and dev by Patrol team LDC2012E03 supplemental training for non-target languages The amount of audio data for different languages heavily unbalanced Added shorter duration samples Derived from 2-min samples, based on our SAD output 4 Pavel Matejka Patrol LID System for DARPA RATS P1 Evaluation

5. Datasets Train Main Only files where VAD detects >60s of speech 30774 files together Unbalanced = 668 files for Dari, 12778 for Leventine Arabic Balanced Balanced over files for each language and channel 7150 files for each duration 673 files for Dari, otherwise ~1300 Extended Main + all 30sec cuts from Main set + entire LDC2012e03 (only nontarget languages) ~170k segments Development Set Corpus was driven by Dari - only 679 source files, other languages limited to 1000 files, 2432 files for non target languages ~7120 files for each duration Evaluation Data 2527 files for each duration 5 Pavel Matejka Patrol LID System for DARPA RATS P1 Evaluation

6. LID Patrol System Architecture JFA LID iVector LID BUT SAD Calibration & Fusion Combined Score Phonotactic iVector LID CZ Phoneme Recognition Audio BBN SAD iVector LID 6 Pavel Matejka Patrol LID System for DARPA RATS P1 Evaluation

7. Speech Activity Detection One of the most important blocks since the data really difficult See separate paper about SAD development on Wednesday 16:00 in Pavilon West Used both GMM-based (BBN) and MLP-based (BUT) detectors. 7 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

8. Speech Activity Detection Comparison of different SAD systems Robust SAD tuned for noisy telephone speech Robust SAD tuned for RATS Results (Cavg) are on DEV set (but scored with SRC channel) iVector system (600dim) used for this experiment SAD type/ Cavg[%] 120s Telephone 2,2 RATS 1,6 25% relative gain 8 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

9. iVector LID System (BUT) Acoustic Features Dithering, bandwidth 300-3200Hz for 25 Mel-filters, 6 MFCC+C0 CMN/CVN (based on SAD), RASTA normalization Shifted Delta Cepstra (SDC) 7-1-3-1 UBM Language independent, diagonal-covariance, 2048 Gaussians Trained on balanced train set iVector 600 dimensions Trained on main set Neural network classifier iVector input, 6 outputs (1 nontarget + 5 target languages) Hidden layer with 200 nodes Stochastic Gradient Descent training with L2 regularization Trained on extended set (all data + all 30 sec splits) 9 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

10. Comparison of Logistic Regression and Neural Network as final classifier BUT iVector system (600dim) Results on Development set Logistic Regression trained by: Trusted Region Conjugate-GD Results on Development set Neural Net: one hidden layer 200 trained by: Stochastic-GD with L2 regularization also experiments with Conjugate-GD, but no improvement 10 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

11. JFA LID System (BUT) Acoustic Features Same as for iVector system + Wiener filtering Universal Background Model (UBM) Language independent, Diagonal-covariance, 2048 Gaussians Trained on balanced train set JFA Trained on main train set = m + Dz + Ux Models of languages D are MAP adapted from UBM with tau =10 Channel matrix U with 200 dimensions Linear scoring 11 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

12. Importance of Wiener Filter 400dim i-vector + logistic regression experimental system Results on Development set 12 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

13. iVector LID System (BBN) Acoustic Features RASTA-PLP Block of 11-frame PLPs, projected to 60 dimensions via HLDA UBM Language dependent (5 target, 1 non-target ), 1024 Gaussians iVector 400 dimensions Group adjacent speech segments into 20s chunks, estimate one iVector per chunk improves performance on short duration conditions by 28% Estimate 6 iVectors (one per UBM) Apply neural network (NN) to each iVector - 6 outputs (1 nontarget + 5 target languages) Combine NN outputs to form 6-dimensional score vector 26% relative improvement compared to using language independent i-vectors 13 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

14. Analysis of iVector LID System (BBN) Analysis of the BBN iVector extractor training and UBM: 1. Whole audio segments, single UBM 2. Audio split to 20s segments, single UBM 3. Audio split to 20s segments, language dependent background models (LDBM) 14 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

15. Phonotactic iVector LID System (BUT) Phoneme recognizer Czech CTS recognizer trained on artificially noised data Added noise with varying SNR (lowest 10dB) to 30% of the corpus 38 phonemes 3-gram counts: sum of posterior probabilities of 3-grams from phone lattices iVector Multinomial subspace modeling 600 dimensions, trained on main train set Training a low-dimensional subspace in the framework of total variability model using multinomial distribution Using point-estimate of the model s latent variable for each utterance as our new features Logistic regression as final classifier Trained on main train set 15 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

16. Fusion and Calibration Regularized logistic regression Objective: minimize cross-entropy on development set Duration-independent trained on files from 10s, 30s, and 120s conditions Procedure Calibrate (tune) each system individually Combine calibrated system outputs into a single output vector Fusion parameters estimated on the same development set Performance evaluation Primarily Cavg score Also computed PMISS and PFA at Phase 1 target operating points 16 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

17. Overall Results 120s 1.61 1.60 2.58 2.60 0.83 7.05 7.83 9.03 9.06 6.56 30s 6.14 4.94 5.92 8.95 2.92 12.92 9.97 11.96 15.56 8.33 10s 12.52 10.36 12.06 16.84 6.85 17.36 14.52 17.83 21.90 11.40 3s 23.53 21.73 28.21 30.53 18.08 22.68 21.46 27.10 29.12 17.45 System\Duration\Cavg[%] JFA iVector NN BUT iVector NN BBN CZ3-iVec600 Fusion JFA iVector NN BUT iVector NN BBN CZ3-iVec600 Fusion DEV EVL 17 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

18. Robustness There is channel B completely removed from the training of the contrastive system (noB) (channel B is unseen channel) Results on Development set with BUT iVector system (600dim) Over all results System/Cavg[%] iVector NN iVector NN noB 120s 1.9 2.5 30s 6.6 7.2 10s 13.0 13.4 3s 23.6 23.8 Results only for channel B System/Cavg[%] iVector NN iVector NN noB 120s 3.4 8.1 30s 9.7 14.7 10s 16.8 19.8 3s 25.8 29.0 18 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka

19. Conclusion SAD is crucial De-noising helps Benefit from using Language dependent UBM Benefit from using NN as final classifier for LID 19 Patrol LID System for DARPA RATS P1 Evaluation Pavel Matejka