residual-connection-formula

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-4.md

Created 2026-06-21T09:55:55+00:00

Residual connections in transformers use the formula x -> F(x) + x, originating from ResNet, preventing vanishing gradients in deep stacking.

Dependents

These beliefs depend on this one:

• IN residual-connections-enable-arbitrary-depth — Residual connections (F(x) + x) are the key architectural innovation enabling arbitrarily deep networks — ResNet solved the degradation problem for CNNs, and the same mechanism was adopted by transformers to enable deep stacking without vanishing gradients.