electra-replaced-token-detection-gan-inspired

IN premise — entries/2026/06/21/wiki-BERT_language_model.md

Created 2026-06-21T09:50:09+00:00

ELECTRA (2020) replaces MLM with replaced-token detection: a small generator creates plausible substitutions while a larger discriminator detects them, inspired by GANs

Summary

ELECTRA trains language models by pitting a small forger against a big detector: the forger swaps in plausible-looking words, and the detector must flag which positions were tampered with. This adversarial setup gives the model a training signal at every single position in a sentence rather than only at a handful of masked slots, making pretraining far more data-efficient.

Dependents

These beliefs depend on this one:

IN bert-variants-systematically-ablated-pretraining-design — BERT's immediate successors (RoBERTa, ALBERT, ELECTRA) each modified different pretraining design choices — NSP removal with larger batches, cross-layer parameter sharing with SOP, and replaced-token detection — suggesting that BERT's specific design decisions could be improved upon while its core architecture remained a viable foundation for alternative pretraining objectives.