relu-activation-definition

IN premise — entries/2026/06/21/wiki-Convolutional_neural_network-chunk-3.md

Created 2026-06-21T09:55:49+00:00

ReLU activation is defined as f(x) = max(0, x), a non-saturating function that trains faster than tanh or sigmoid without significant accuracy loss

Dependents

These beliefs depend on this one:

• IN relu-dominance-despite-theoretical-imperfection — ReLU became the dominant hidden-layer activation function despite violating backpropagation's differentiability requirement at exactly zero, demonstrating that empirical effectiveness (faster training, no vanishing gradient) trumps theoretical correctness in neural network design.