weight-tying-shares-embedding-unembedding
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-2.md
Created 2026-06-21T09:50:11+00:00
Weight tying uses the transpose of the embedding matrix as the un-embedding matrix, halving embedding parameters and preventing training divergence.
Summary
Instead of learning two separate tables—one to turn words into vectors and another to turn vectors back into word probabilities—the model reuses the same table in reverse, which cuts the parameter count in half for those layers. This keeps the model more compact and prevents the two ends of the network from drifting apart during training, making it easier to stabilize.