Regularized Minimum Bayes Risk Reranking Framework
"Explore a regularized MBR reranking framework for machine translation, addressing issues of overconfident candidate selection and model uncertainty. Design includes semantic-based similarity, partial candidate lists, and two types of regularization scores for translation quality and model uncertainty."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
RMBR: A Regularized Minimum Bayes Risk Reranking Framework for Machine Translation Yidan Zhang1, Yu Wan2, Dayiheng Liu1, Baosong Yang2, Zhenan He1
Introduction NMT model can output high-quality candidates, but fails at picking the best one. Top-1 candidates is overconfident compare the token probability between top-1 candidates and references Oracle ranking of samples generated by multiple decoding strategies.
Introduction Minimum Bayesian Risk (MBR) decoding find the translation that is closest to other candidate translations. minimize the expected risk. avoid the overconfident candidates. MBR suffers from the following problems: Utility function only considers the lexical-level similarity. Considers the entire n-best list time-consuming inadequate candidates may hurt the performance. Ignore the model uncertainty and the translation quality. The distribution of oracle translations
Introduction We design a regularized MBR reranking framework (RMBR) Use semantic-based similarity utility function Use partial candidate list Consider two type of regularization scores translation quality model uncertainty
Regularized MBR Reranking Framework Given ?-best candidates: {?1,?2,...,??} ??: j-th regularizer score ?? {0.001,0.01,0.1,1,10} selected by validation set
MBR Score Given a utility function ? (e.g., BLEU) ?( ?) is tuned on validation set. We further explore two semantic-based evaluation methods BLEURT: BERT based Metric for Natural Language Generation ??????(????? ????,?????????) ?(??,??) COMET: use source and reference sentence to measures Translation quality. ?????(??????,????? ????,?????????) ?(?,??,??)
Quality Regularizer Language Model (LM) score Back-Translation (BT) score Quality Estimation (QE) score log-probability scores (Translation model score)
Uncertainty Regularizer Monte Carlo (MC) dropout At test time, perform ? forward passes through the NMT model with randomly deactivates part of neurons. ??: t-th pass model parameter Entropy Measures ? = {?1,?2,...,??}: vocabulary of NMT ???: j-th token in Hypothesis i
Experiment Dataset: IWSLT 14 task, WMT 14 task, Newstest 14 German-English (De En) English-German (En De) English-French (En Fr) NMT model: Transformer base Baselines: top-1 results from beam search with beam size 5
Results We observe the results of MBRCOMET+LP+QE+LM that achieves the highest BLEU score . Combining quality and uncertainty regularizers with MBRCOMET can not lead to further performance gains.
Utility Functions comparison Results show that semantic-based MBR leads to better translation.
Inference Time ? = 30,?1= 21,?2= 3
