ELECTRA: Pre-Training Text Encoders as Discriminators

Efficiently learning an encoder that classifies token replacements accurately using ELECTRA method, which involves replacing some input tokens with samples from a generator instead of masking. The key idea is to train a text encoder to distinguish input tokens from negative samples, resulting in bet

0 views • 12 slides