Convolutional Networks

48 beliefs (48 IN, 0 OUT)

IN activation-function-provides-nonlinearity
The activation function is what gives multi-layer networks their representational power; without non-linearity, stacking layers adds no capacity and the network collapses to a linear transformation.
IN activation-jacobian-is-diagonal
Element-wise activation functions have a diagonal Jacobian matrix, which can be equivalently computed as a Hadamard (element-wise) product with the vector of per-node derivatives.
IN amari-1967-first-sgd-trained-mlp
Shun'ichi Amari (1967) published the first multilayer perceptron trained by stochastic gradient descent, with 5 layers and 2 learnable layers.
IN backprop-computes-gradient-not-learning
Backpropagation only computes the gradient; a separate optimization algorithm (e.g., SGD, Adam) performs the actual parameter updates.
IN backprop-delta-recursive-computation
The key intermediate quantity delta^l (gradient of the weighted input at layer l) is computed recursively from delta^(l+1) via δ^(l-1) = (f^(l-1))' ∘ (W^l)^T · δ^l, and is sufficient to compute all weight gradients at that layer.
IN backprop-efficient-recursive-gradient-factorization
Backpropagation's computational efficiency stems from a recursive factored structure: the error signal delta propagates backward layer-by-layer via matrix-vector products, each weight gradient factors into a compact o_i * delta_j form, and the full weight gradient is a simple outer product of local quantities — avoiding redundant computation entirely.
IN backprop-gradient-factors-to-oi-delta-j
The gradient ∂E/∂w_ij factors into the compact form o_i · δ_j, where o_i is the activation of the sending neuron and δ_j is the error signal at the receiving neuron.
IN backprop-is-gradient-computation-not-optimizer
Backpropagation is strictly a gradient computation algorithm (reverse-mode chain rule); it does not specify how weights are updated — that is the role of an optimizer such as SGD or Adam.
IN backprop-is-reverse-mode-autodiff
Backpropagation is a special case of reverse mode automatic differentiation.
IN backprop-weight-gradient-outer-product
The weight gradient at layer l is computed as an outer product: ∇_{W^l} C = δ^l · (a^(l-1))^T, which is why activations from the forward pass must be cached.
IN backpropagation-computes-loss-gradient-wrt-weights
Backpropagation calculates the gradient of the loss function with respect to weights, propagating error from output nodes back through the network to input nodes.
IN cross-entropy-loss-classification-squared-error-regression
Cross-entropy (log loss) is the standard loss function for classification tasks; squared error loss is standard for regression tasks in neural network training.
IN cybenko-universal-approximation-theorem-1989
Cybenko's Universal Approximation Theorem (1989) proves that a single hidden layer with sigmoidal activation functions can approximate any continuous function on compact subsets of R^n.
IN deep-nn-at-least-two-hidden-layers
A neural network is classified as 'deep' if it has at least two hidden layers
IN dropout-probability-conventions
Dropout keeps each node with probability p (typically 0.5 for hidden layers, higher for input layers) during training; at test time outputs are scaled by p to approximate an ensemble of 2^n sub-networks
IN echo-state-network-only-trains-output-weights
Echo state networks have a sparsely connected random hidden layer where only output weights are trainable; the hidden layer is fixed.
IN elman-network-context-from-hidden-layer
Elman networks feed hidden layer output to context units via fixed weight-one connections; Jordan networks feed output layer to context units (state layer) with self-recurrent connections.
IN encoder-2-sublayers-decoder-3-sublayers
Each encoder layer has 2 sublayers (self-attention, FFN); each decoder layer has 3 sublayers (masked self-attention, cross-attention, FFN).
IN feedforward-network-dag-no-cycles
In a feedforward neural network, information flows in one direction only (input to output) with no cycles or feedback loops — the dependency graph is a directed acyclic graph (DAG).
IN forward-pass-must-cache-activations-and-derivatives
During the forward pass, both activations a^l and activation function derivatives (f^l)' must be cached at each layer for use in the backward pass.
IN fully-connected-layer-parameter-count
The number of learnable parameters in a single fully-connected layer is (p × q) weights + q biases, where p is the number of inputs and q is the number of outputs.
IN helmholtz-machines-asymmetric-weights-wake-sleep
Helmholtz machines use asymmetric weights with separate recognition (forward) and generative (backward) networks, trained with the Wake-Sleep algorithm; they were early inspiration for VAEs
IN hidden-neuron-delta-recursive-from-downstream
For hidden neurons, δ_j = (Σ_ℓ w_jℓ · δ_ℓ) · φ'(net_j) — the error signal is computed recursively as a weighted sum of downstream δ values, which is the mechanism that gives backpropagation its name.
IN indrnn-each-neuron-own-past-state-only
IndRNN neurons receive only their own past state (not other neurons' states), eliminating inter-neuron dependencies within a layer and preventing vanishing/exploding gradients.
IN infinite-width-networks-behave-linearly
As neural network width approaches infinity, behavior is well-described by first-order Taylor expansion, inheriting convergence properties of affine (linear) models.
IN loss-function-requirements-for-backprop
A loss function usable with backpropagation must satisfy two requirements: (1) decomposable as an average over individual training examples E = (1/n) Σ E_x, and (2) expressible as a function of the network's outputs.
IN matrix-backprop-only-feedforward-no-skip-connections
The clean matrix multiplication formulation of backpropagation applies to feedforward networks with no skip connections and a scalar loss function; more general architectures require the adjoint graph / reverse-mode AD framework.
IN minsky-papert-1969-xor-limitation
Minsky and Papert (1969) demonstrated that single-layer perceptrons cannot compute XOR (non-linearly-separable functions), contributing to the first AI winter for neural networks.
IN neuron-computation-formula
An artificial neuron computes: output = activation_function(sum(weight_i * input_i) + bias), applying a non-linear activation function to the weighted sum of inputs plus a bias term
IN nn-layers-hierarchical-abstraction
Neural network layers provide hierarchical abstraction: bottom layers handle raw data, intermediate layers progressively increase abstraction (e.g., pixels → edges → objects), and top layers produce final results
IN probabilistic-interpretation-led-to-dropout
The probabilistic interpretation of neural networks (treating activation nonlinearities as cumulative distribution functions) led to the introduction of dropout as a regularization technique
IN relu-activation-definition
ReLU activation is defined as f(x) = max(0, x), a non-saturating function that trains faster than tanh or sigmoid without significant accuracy loss
IN relu-dominance-despite-theoretical-imperfection
ReLU became the dominant hidden-layer activation function despite violating backpropagation's differentiability requirement at exactly zero, demonstrating that empirical effectiveness (faster training, no vanishing gradient) trumps theoretical correctness in neural network design.
IN relu-introduced-fukushima-1969
The ReLU activation function was first introduced by Fukushima in 1969, though it was not used in the neocognitron itself
IN relu-nondifferentiable-at-zero-works-in-practice
ReLU is non-differentiable at exactly 0 (violating backpropagation's theoretical requirement for differentiable activations), but works in practice by convention of setting f'(0) = 0 (subgradient approach), and has been dominant since AlexNet.
IN relu-replaced-sigmoid-default-hidden-activation
ReLU has largely replaced sigmoid as the default hidden-layer activation function in neural networks.
IN relu-width-universal-approximation-constraint
Lu et al. proved that for ReLU networks, the width must be strictly larger than the input dimension to be a universal approximator; width ≤ input dimension fails
IN rosenblatt-perceptron-first-trainable-nn
Frank Rosenblatt's Perceptron (1957-1958) was the first trainable single-layer neural network.
IN single-layer-feedforward-equation
A single-layer feedforward network computes output as y_q = K(Σ_i(x_i · w_iq) − b_q), where K is the activation function, w_iq are weights, and b_q is the bias.
IN single-layer-only-linearly-separable
A single-layer network (no hidden layer) can only learn linearly separable functions; hidden layers are required for non-linear decision boundaries.
IN softmax-multiclass-logistic-binary
Softmax activation is used at the output layer for multi-class classification; the logistic (sigmoid) function is used for binary classification.
IN sparse-polynomials-exponentially-easier-deep
Sparse multivariate polynomials are exponentially easier to approximate with deep networks than shallow ones, providing a key theoretical justification for network depth
IN spectral-bias-low-to-high-frequencies
Neural networks with small parameter counts exhibit spectral bias (frequency principle): they fit target functions from low to high frequencies, and deeper networks are more biased toward low frequencies.
IN stacked-rbms-greedy-layerwise-no-backprop
Stacked RBMs (Deep Belief Networks) are trained greedily one layer at a time without backpropagation; optimal depth was found to be 3-4 layers as of 2009
IN td-learning-models-dopamine-neuroscience
TD learning models dopamine-based learning in neuroscience; dopaminergic projections from substantia nigra to basal ganglia encode prediction error signals
IN tdnn-first-weight-sharing-with-backprop
The Time Delay Neural Network (TDNN, Waibel 1987) was the first CNN to combine weight sharing with gradient descent/backpropagation training
IN universal-approximation-theorem-no-architecture-spec
The universal approximation theorem proves multilayer perceptrons can approximate any function, but does not specify how many neurons, what topology, or what weights are needed.
IN weight-matrix-rows-cols-convention
Weight matrix W^l has rows equal to the number of neurons in layer l and columns equal to the number of neurons in layer l-1.