relu-replaced-sigmoid-default-hidden-activation

IN premise — entries/2026/06/21/wiki-Backpropagation-chunk-1.md

Created 2026-06-21T09:55:49+00:00

ReLU has largely replaced sigmoid as the default hidden-layer activation function in neural networks.

Dependents

These beliefs depend on this one:

• IN relu-dominance-despite-theoretical-imperfection — ReLU became the dominant hidden-layer activation function despite violating backpropagation's differentiability requirement at exactly zero, demonstrating that empirical effectiveness (faster training, no vanishing gradient) trumps theoretical correctness in neural network design.