gradient-masking-sparse-relu-networks

IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-8.md

Created 2026-08-25T02:58:01+00:00

Standard gradient-based adversarial attacks fail on sparse ReLU networks because ~99% of ReLU neurons are in the zero regime, effectively zeroing out gradients (gradient masking).

Summary

In networks built with ReLU activations where most neurons sit in their inactive (zero-output) state, the gradient signal that standard adversarial attacks rely on to find a harmful input perturbation is almost entirely wiped out. This means the network's sparsity acts as a built-in defense: an attacker following the usual gradient-descent recipe simply cannot get useful direction information, so the most common attack class breaks down without any extra hardening.