relu-nondifferentiable-at-zero-works-in-practice

IN premise — entries/2026/06/21/wiki-Backpropagation-chunk-4.md

Created 2026-06-21T09:55:49+00:00

ReLU is non-differentiable at exactly 0 (violating backpropagation's theoretical requirement for differentiable activations), but works in practice by convention of setting f'(0) = 0 (subgradient approach), and has been dominant since AlexNet.

Dependents

These beliefs depend on this one:

• IN relu-dominance-despite-theoretical-imperfection — ReLU became the dominant hidden-layer activation function despite violating backpropagation's differentiability requirement at exactly zero, demonstrating that empirical effectiveness (faster training, no vanishing gradient) trumps theoretical correctness in neural network design.