relu-dominance-despite-theoretical-imperfection

IN derived (depth 1)

Created 2026-06-21T10:13:05+00:00 · Reviewed 2026-06-21T15:37:01+00:00

ReLU became the dominant hidden-layer activation function despite violating backpropagation's differentiability requirement at exactly zero, demonstrating that empirical effectiveness (faster training, no vanishing gradient) trumps theoretical correctness in neural network design.

Justifications

SL — depth-1 thematic — ReLU's dominance despite a known theoretical flaw exemplifies ML's empirics-over-theory character

Antecedents (all must be IN):

IN relu-replaced-sigmoid-default-hidden-activation — ReLU has largely replaced sigmoid as the default hidden-layer activation function in neural networks.
IN relu-nondifferentiable-at-zero-works-in-practice — ReLU is non-differentiable at exactly 0 (violating backpropagation's theoretical requirement for differentiable activations), but works in practice by convention of setting f'(0) = 0 (subgradient approach), and has been dominant since AlexNet.
IN relu-activation-definition — ReLU activation is defined as f(x) = max(0, x), a non-saturating function that trains faster than tanh or sigmoid without significant accuracy loss

Dependents

These beliefs depend on this one:

IN ml-theory-consistently-violated-without-penalty — ML's relationship with its own theory is consistently paradoxical — ReLU violates differentiability requirements yet outperforms smooth activations, while double descent and benign overfitting violate the bias-variance tradeoff — suggesting ML's theoretical frameworks describe idealized conditions that practice routinely transcends without penalty.