weight-tying-embedding-unembedding

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-3.md

Created 2026-06-21T09:55:55+00:00

Weight tying uses the transpose of the embedding matrix as the un-embedding matrix, halving embedding parameters and preventing training divergence.

Dependents

These beliefs depend on this one:

• IN weight-sharing-is-universal-efficiency-principle — Weight sharing is a universal efficiency principle appearing independently across architectures — CNN parameter sharing reduces weights by orders of magnitude (25 vs 10,000 for a 5×5 kernel), transformer weight tying halves embedding parameters, and the principle generalizes to any architecture where symmetry in the data structure (spatial, positional) can be exploited to reduce free parameters without reducing expressiveness.