weight-sharing-is-universal-efficiency-principle

IN derived (depth 1)

Created 2026-06-21T11:27:22+00:00 · Reviewed 2026-06-21T15:37:01+00:00

Weight sharing is a universal efficiency principle appearing independently across architectures — CNN parameter sharing reduces weights by orders of magnitude (25 vs 10,000 for a 5×5 kernel), transformer weight tying halves embedding parameters, and the principle generalizes to any architecture where symmetry in the data structure (spatial, positional) can be exploited to reduce free parameters without reducing expressiveness.

Justifications

SL — Weight sharing invented independently for CNNs (spatial) and transformers (vocabulary) follows the same principle — exploit data symmetry to trade parameters for inductive bias

Antecedents (all must be IN):

IN cnn-weight-sharing-parameter-reduction — CNN weight sharing reduces parameters dramatically: a 5x5 kernel uses 25 weights vs 10,000 for a fully connected layer on 100x100 input
IN weight-tying-embedding-unembedding — Weight tying uses the transpose of the embedding matrix as the un-embedding matrix, halving embedding parameters and preventing training divergence.
IN cnn-parameter-sharing-translation-invariance — Parameter sharing in convolutional layers (all neurons in a depth slice share weights and bias) reduces free parameters and provides translation invariance

Dependents

These beliefs depend on this one:

IN convergent-discovery-reveals-mathematical-necessity — Three of deep learning's foundational mechanisms — gradient computation (backprop independently discovered across fields), gradient flow solutions (residual connections and LSTM gating converging independently), and weight sharing (appearing independently across architectures) — were all independently discovered or converged upon, suggesting these are mathematical necessities of the problem structure rather than contingent design choices.
IN weight-sharing-implements-geometry-matched-compression — Weight sharing can be understood as a primary mechanism through which neural architectures implement geometry-matched compression — CNN shared kernels exploit spatial structure via parameter reuse, transformer weight tying exploits distributional symmetry across embedding spaces, and both achieve efficiency by matching parameter reuse patterns to the data's intrinsic geometry.