ML Engineering & Practice

48 beliefs (48 IN, 0 OUT)

• IN alexnet-ilsvrc-2012-googlenet-2014
  AlexNet won ILSVRC 2012 as the breakthrough CNN for image recognition; GoogLeNet (30+ layers) won ILSVRC 2014 with 0.439 mAP for detection and near-human classification accuracy
• IN channel-max-pooling-reduces-channels
  Channel max pooling (CMP) pools along the channel dimension rather than spatially, reducing channel count while preserving spatial dimensions
• IN cnn-biological-origin-hubel-wiesel-1959
  CNNs trace their biological inspiration to Hubel and Wiesel's 1959/1968 discovery of receptive fields in cat and monkey visual cortex, with simple and complex cell hierarchies
• IN cnn-biologically-inspired-but-technically-imprecise
  CNNs are biologically inspired (receptive fields from Hubel-Wiesel 1959) but technically imprecise in two ways: the "convolution" operation is actually cross-correlation, and the features are translation-equivariant rather than translation-invariant as commonly claimed.
• IN cnn-convolution-is-cross-correlation
  The convolution operation in CNNs is technically cross-correlation, not mathematical convolution
• IN cnn-efficiency-from-inductive-biases
  CNNs achieve dramatic parameter efficiency through two complementary inductive biases — local connectivity restricts each neuron's receptive field, and weight sharing forces all spatial positions to use identical filters — reducing parameters by orders of magnitude.
• IN cnn-four-distinguishing-features
  CNNs have four distinguishing architectural features: 3D neuron volumes, local connectivity, shared weights, and pooling
• IN cnn-input-tensor-shape
  CNN input tensor shape is (batch) x (height) x (width) x (channels)
• IN cnn-lineage-neocognitron-to-alexnet
  CNN lineage runs from Fukushima's Neocognitron (1979–1980) to Zhang's shift-invariant networks (1988) to LeCun's LeNet (1989) to LeNet-5 (1998) to AlexNet (2012).
• IN cnn-local-connectivity
  Convolutional layers differ from fully connected layers by connecting each neuron to only a local subset of neurons in the previous layer rather than all neurons
• IN cnn-not-limited-to-images
  CNNs are not limited to image data — they apply wherever local spatial or temporal structure exists, including audio (time domain), spectrograms (frequency domain), time series, and text
• IN cnn-output-size-formula
  The output spatial size of a convolutional layer is computed as (W - K + 2P) / S + 1, where W is input size, K is kernel size, P is padding, and S is stride
• IN cnn-parameter-sharing-translation-invariance
  Parameter sharing in convolutional layers (all neurons in a depth slice share weights and bias) reduces free parameters and provides translation invariance
• IN cnn-pragmatic-not-faithful-biological-inspiration
  CNNs demonstrate that effective ML architectures don't require biological fidelity — their inductive biases (local connectivity, weight sharing) capture the right structural constraints for dramatic parameter efficiency, even though the "convolution" is technically cross-correlation and the equivariance doesn't produce true invariance.
• IN cnn-translation-equivariant-not-invariant
  CNNs produce translation-equivariant feature maps but are typically NOT translation-invariant due to downsampling/pooling — a common misconception
• IN cnn-weight-sharing-parameter-reduction
  CNN weight sharing reduces parameters dramatically: a 5x5 kernel uses 25 weights vs 10,000 for a fully connected layer on 100x100 input
• IN cnns-approximate-not-exact-translation-invariance
  CNNs achieve only approximate translation invariance, not exact — stride and pooling provide partial invariance, and global average pooling at the last layer is one solution for complete translation invariance
• IN cnns-near-human-imagenet-but-fail-small-objects
  CNNs achieve near human-level performance on ImageNet but still struggle with small/thin objects and filtered images
• IN deconvolution-checkerboard-artifact-fix
  Deconvolutional (transposed convolution) layers can create checkerboard artifacts; the fix is to use upscale-then-convolve instead
• IN depthwise-separable-convolution-structure
  Depthwise separable convolution splits into depthwise convolution (spatial, per-channel) followed by pointwise convolution (1x1 kernels) to reduce computation
• IN depthwise-separable-convolutions-xception
  Depthwise separable convolutions (Chollet 2017, Xception) factor standard convolutions into depthwise and pointwise operations for computational efficiency, key to mobile architectures
• IN dilated-convolution-expands-receptive-field
  Dilated (atrous) convolution expands the receptive field without increasing parameters by interleaving visible and blind regions in the kernel
• IN dilated-convolutions-solve-vanishing-gradients-timeseries
  Dilated (atrous) convolutions enable 1D CNNs to learn long-range time series dependencies without the vanishing gradient problem that affects RNNs, making CNNs competitive for forecasting
• IN dilation-effective-receptive-field
  Dilation of d on a kernel of size k expands the effective receptive field to k + (k-1)(d-1) pixels while processing only k*k values; e.g., dilation 2 on 3x3 covers a 5x5 area with 9 parameters
• IN dropconnect-drops-weights-not-neurons
  DropConnect generalizes dropout by dropping individual connection weights rather than entire neuron outputs, creating dynamic sparsity on weights
• IN fc-layers-break-translation-invariance
  Fully connected layers break translation invariance even when all preceding convolutional layers preserve equivariance
• IN lecun-1989-backprop-cnn
  LeCun et al. (1989) applied backpropagation to train CNNs for handwritten zip code recognition, establishing the modern CNN training paradigm
• IN lecun-1989-first-backprop-learned-conv-kernels
  LeCun et al. (1989) were the first to use backpropagation to learn convolution kernels directly from images rather than hand-designing them
• IN lenet5-1998-bank-check-recognition
  LeNet-5 (1998) by LeCun was used by banks for check digit recognition; CNNs processed 10-20% of US checks by the early 2000s
• IN lenet5-architecture-1995
  LeNet-5 (LeCun et al., 1995) was a 7-layer CNN for handwritten digit recognition on 32x32 images, deployed commercially in bank check reading systems
• IN locally-connected-layer-no-weight-sharing
  Locally connected layers use independent weights per spatial location instead of parameter sharing, used when different spatial locations should learn different features (e.g., centered face images)
• IN max-pooling-2x2-discards-75-percent
  Max pooling with 2x2 filters and stride 2 discards 75% of activations, with downsampling caused by the stride not the pooling operation itself
• IN max-pooling-introduced-1990
  Max pooling was introduced by Yamaguchi et al. in 1990 for speech recognition; Weng et al. (1993, cresceptron) brought it to computer vision
• IN modern-cnns-compute-correlation-not-convolution
  Modern CNNs actually compute correlation and call it convolution — reversing the kernel weights doesn't affect the learned function
• IN neocognitron-1980-cnn-precursor
  Fukushima's Neocognitron (1980) was a self-organizing neural network for shift-invariant pattern recognition and the direct precursor to modern CNNs
• IN neocognitron-1980-s-layers-c-layers
  Fukushima's Neocognitron (1980) introduced S-layers (shared-weight convolutional layers) and C-layers (downsampling/pooling layers), the architectural precursors to modern CNN conv and pooling layers
• IN residual-connection-formula
  Residual connections in transformers use the formula x -> F(x) + x, originating from ResNet, preventing vanishing gradients in deep stacking.
• IN residual-connections-enable-arbitrary-depth
  Residual connections (F(x) + x) are the key architectural innovation enabling arbitrarily deep networks — ResNet solved the degradation problem for CNNs, and the same mechanism was adopted by transformers to enable deep stacking without vanishing gradients.
• IN resnet-2015-skip-connections-degradation
  ResNet (2015) by He et al. solved the degradation problem of very deep networks (20-30+ layers) using skip/residual connections
• IN resnets-highway-nets-2015-degradation
  Highway networks (May 2015) and ResNets (December 2015) solved the degradation problem in very deep networks of 20-30+ layers
• IN stride-gt1-breaks-translation-equivariance
  Stride greater than 1 in convolutional or pooling layers violates the Nyquist-Shannon sampling theorem, causing aliasing and breaking translation equivariance
• IN transposed-convolution-checkerboard-artifacts
  Transposed convolutions (deconvolution) produce checkerboard artifacts as a known failure mode; resize-convolution is the standard fix
• IN vae-encodes-gaussian-parameters-reparameterization
  VAE middle layer encodes parameters (mean, variance) of Gaussian distributions rather than fixed values, enabling stochastic generation via the reparameterization trick
• IN video-cnn-three-approaches
  Three main approaches for video CNNs: (1) 3D convolutions over space+time, (2) two-stream fusion of spatial and temporal CNNs, (3) CNN feature extraction followed by LSTM for inter-frame dependencies
• IN vision-transformers-competing-with-cnns
  Vision Transformers (ViT) are increasingly competing with or replacing CNNs in computer vision tasks
• IN vit-16x16-patches-as-tokens
  Vision Transformer (ViT) by Dosovitskiy et al. (2021) divides images into 16x16 patches treated as tokens for image recognition
• IN weight-sharing-implements-geometry-matched-compression
  Weight sharing can be understood as a primary mechanism through which neural architectures implement geometry-matched compression — CNN shared kernels exploit spatial structure via parameter reuse, transformer weight tying exploits distributional symmetry across embedding spaces, and both achieve efficiency by matching parameter reuse patterns to the data's intrinsic geometry.
• IN weight-sharing-is-universal-efficiency-principle
  Weight sharing is a universal efficiency principle appearing independently across architectures — CNN parameter sharing reduces weights by orders of magnitude (25 vs 10,000 for a 5×5 kernel), transformer weight tying halves embedding parameters, and the principle generalizes to any architecture where symmetry in the data structure (spatial, positional) can be exploited to reduce free parameters without reducing expressiveness.