Recurrent Architectures

49 beliefs (48 IN, 0 OUT)

• IN bidirectional-rnn-processes-both-directions
  Bidirectional RNNs (BRNNs) process input sequences in both forward and backward directions simultaneously, enabling access to both past and future context at each time step.
• IN birnn-concatenates-forward-backward-outputs
  Bidirectional RNNs concatenate forward and backward outputs at each position (they do not average or add them).
• IN bptt-extends-backprop-for-recurrent-architectures
  Backpropagation through time (BPTT) is an extension of standard backpropagation for training recurrent neural network architectures
• IN ctc-loss-for-variable-alignment
  Connectionist Temporal Classification (CTC) is a loss function for sequence modeling where input-output alignment timing is variable, such as speech recognition.
• IN ctrnn-discrete-equivalent-via-shannon-sampling
  Discrete-time RNNs are equivalent to continuous-time RNNs (CTRNNs) via the Shannon sampling theorem after low-pass filtering.
• IN echo-state-network-reservoir-computing
  Echo State Networks are a reservoir computing approach to RNNs where only the output weights are trained
• IN elmo-is-stacked-bidirectional-lstm
  ELMo (2018) is a stacked bidirectional LSTM that takes character-level inputs and produces word-level embeddings.
• ? exam-fail-q20
  
• IN feedforward-nn-is-dag
  Feedforward neural networks form a directed acyclic graph (DAG) with no connections to same-layer or previous-layer neurons, while recurrent networks allow cycles
• IN finite-impulse-rnn-dag-infinite-impulse-cyclic
  Finite-impulse RNNs are DAGs (unrollable to feedforward nets); infinite-impulse RNNs are directed cyclic graphs (cannot be unrolled).
• IN fully-recurrent-nn-most-general-topology
  A fully recurrent neural network (every neuron's output connects to every neuron's input) is the most general RNN topology — all other architectures are special cases with some weights set to zero.
• IN gnmt-whole-sentence-english-pivot
  Google Neural Machine Translation (GNMT) uses end-to-end LSTM to translate whole sentences (not phrase-by-phrase) and uses English as an intermediate pivot language between most language pairs
• IN gradient-flow-triply-convergent-across-architectures
  The gradient flow bottleneck has been addressed through at least three architecturally distinct mechanisms — LSTM gating (1997), GRU simplified gating without output gate (2014, achieving comparable performance with fewer parameters), and residual connections (2015) — suggesting that gradient flow is among the most convergently addressed problems in deep learning and that the bottleneck admits multiple valid solutions of varying complexity.
• IN gru-fewer-params-than-lstm-no-output-gate
  GRU (introduced 2014) has fewer parameters than LSTM because it lacks an output gate; empirical performance is comparable with no clear winner.
• IN gru-lightweight-lstm-alternative
  GRU (Gated Recurrent Unit) is a more computationally efficient alternative to LSTM with fewer parameters while also addressing the vanishing gradient problem.
• IN gru-proposed-2014-cho-et-al
  GRU was proposed by Cho et al. in 2014 (arXiv:1406.1078); Gruber & Jockisch (2020) found GRU cells more specific and LSTM cells more sensitive.
• IN gru-simplification-validates-sparse-mathematical-necessity
  GRU's comparable performance to LSTM with fewer parameters (notably lacking the output gate) within the space of convergent gradient flow solutions suggests that convergently discovered solutions to mathematical bottlenecks may admit simpler formulations — the core requirement (unimpeded gradient flow) appears sparser than initial implementations suggest, and the output gate may represent implementation complexity beyond the minimal mathematical requirement rather than a necessity.
• IN hopfield-networks-connect-rnns-to-ising-model
  Hopfield networks (1982) connect RNNs to statistical mechanics, drawing from the Ising model (Lenz 1920, Ising 1925) and spin-glass models (Sherrington & Kirkpatrick 1975).
• IN lstm-designed-to-solve-vanishing-gradient
  LSTM was designed specifically to solve the vanishing gradient problem using forget gates as the key mechanism.
• IN lstm-forget-gate-1999
  LSTM was introduced by Hochreiter and Schmidhuber in 1997 to solve the vanishing gradient problem (identified 1991), with the forget gate added in 1999 completing the standard architecture
• IN lstm-gru-are-gated-rnn-variants
  LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) are gated variants of RNNs designed to address the vanishing gradient problem
• IN lstm-introduced-1997-gating
  LSTM was introduced in 1997 with multiplicative gating units to mitigate the vanishing gradient problem.
• IN lstm-learns-over-thousands-millions-timesteps
  LSTM can learn dependencies across thousands or millions of time steps by allowing error to flow backward through unlimited virtual layers.
• IN lstm-proposed-1997-hochreiter-schmidhuber
  LSTM was proposed by Hochreiter & Schmidhuber in 1997 (doi:10.1162/neco.1997.9.8.1735), addressing the vanishing gradient problem identified in Hochreiter's 1991 diploma thesis.
• IN lstm-recognizes-context-sensitive-languages
  LSTM can recognize context-sensitive languages, surpassing HMM-based models which are limited to regular languages.
• IN lstm-solves-vanishing-gradient
  LSTM (Long Short-Term Memory), invented by Hochreiter and Schmidhuber (1995/1997), uses gating mechanisms to preserve long-term dependencies and was specifically designed to address the vanishing gradient problem in vanilla RNNs.
• IN neural-arch-taxonomy-mlp-cnn-rnn-transformer-mamba
  Neural network architecture taxonomy progresses: MLP (feedforward) → CNN (spatial) → RNN/LSTM/GRU (sequential) → Transformer (attention-based, now dominant) → Mamba (state-space, emerging alternative).
• IN neural-history-compressor-1991-pretraining
  Schmidhuber's neural history compressor (1991) used predictive coding and self-supervised pre-training with a hierarchy of RNNs; in 1993 it solved a task requiring >1000 unfolded layers
• IN neural-history-compressor-1993-deep-learning
  The neural history compressor solved a task requiring 1000+ unfolded RNN layers in 1993, using an unsupervised stack of RNNs where each level only receives inputs unpredictable to the level below.
• IN ntm-differentiable-end-to-end
  Neural Turing Machines couple an RNN to external memory analogous to a Turing machine but are differentiable end-to-end and trainable with gradient descent.
• IN pragmatism-wins-even-within-discovered-necessities
  GRU's successful simplification of LSTM (fewer parameters, no output gate, comparable performance) combined with the broader principle that mathematical completeness is counterproductive for survival demonstrates that pragmatic minimalism outperforms theoretical completeness even within convergently-discovered mathematical necessities — the pragmatism principle operates recursively, governing not only which mechanisms are adopted but which implementations of those mechanisms survive.
• IN rbm-removes-lateral-connections-tractable
  Restricted Boltzmann Machines (RBMs) remove lateral (within-layer) connections from Boltzmann machines, making training tractable via Contrastive Divergence with Gibbs Sampling
• IN recurrent-networks-turing-completeness
  A recurrent neural network with rational-valued weights has universal Turing machine power; irrational weights theoretically yield super-Turing power (though the latter claim has failed verification).
• IN residual-connections-and-lstm-gating-are-convergent-solutions
  Residual connections (ResNet) and LSTM gating are convergent solutions to the same underlying problem — both enable gradient flow through deep computation graphs, with LSTM's forget gate solving vanishing gradients in recurrent (temporal) depth and skip connections solving the degradation problem in feedforward (spatial) depth — suggesting that identity-preserving shortcuts are a universal architectural principle for deep networks regardless of topology.
• IN rnn-config-vs-architecture-orthogonal
  RNN configuration (data flow wiring: standard, stacked, bidirectional, encoder-decoder) and architecture (cell design: LSTM, GRU, vanilla) are orthogonal — any configuration can use any architecture.
• IN rnn-design-space-two-dimensional
  RNN design involves two orthogonal dimensions: cell architecture (vanilla RNN, LSTM, GRU) determines how hidden state is updated, while configuration (standard, stacked, bidirectional, encoder-decoder) determines data flow topology — the canonical function signature f_theta: (x_t, h_t) -> (y_t, h_{t+1}) is invariant across both, and the time-unfolded diagram represents temporal steps not spatial depth.
• IN rnn-hidden-state-sequential-processing
  Recurrent neural networks process sequential data by maintaining a hidden state updated at each time step based on current input and the previous hidden state, enabling temporal dependency capture.
• IN rnn-historical-lineage-cajal-to-transformers
  RNN historical lineage: neuroscience (Cajal 1909) → computational neuroscience (McCulloch-Pitts 1943) → perceptrons (Rosenblatt 1961) → associative memory (Hopfield 1982) → trainable RNNs (Rumelhart 1986) → LSTM (1997) → GRU (2014) → Transformers (2017).
• IN rnn-lineage-jordan-to-lstm-forget-gates
  RNN lineage runs from Jordan networks (1986) to Elman networks (1990) to LSTM (Hochreiter & Schmidhuber, 1995/1997) to forget gates (Gers et al., 1999).
• IN rnn-standard-function-signature
  A standard RNN is defined as f_θ: (x_t, h_t) → (y_t, h_{t+1}), mapping an input and hidden state to an output and updated hidden state.
• IN rnn-theoretical-power-exceeded-practical-utility
  RNNs possess strong theoretical computational power — Turing-completeness with rational weights (Siegelmann & Sontag 1994) and context-sensitive language recognition via LSTM that surpasses what HMM-based models achieve — while their sequential hidden-state processing, which updates state at each time step, represents an inherent constraint on parallelism.
• IN rnn-turing-completeness-purest-case-of-theoretical-irrelevance
  RNNs' proven Turing-completeness (Siegelmann & Sontag 1994) — the strongest computational-theoretic result for any neural architecture family — is the purest demonstration that theoretical computational power is irrelevant to paradigm survival, as Transformers displaced RNNs solely through superior hardware utilization despite possessing strictly less formal computational power.
• IN rnn-turing-completeness-siegelmann-sontag
  Siegelmann & Sontag (1994) proved that recurrent neural networks with rational weights are Turing-complete, and with real-valued weights they are super-Turing.
• IN rnn-unfolded-diagram-shows-timesteps-not-layers
  Unfolding an RNN across time steps produces a diagram that resembles layers, but these represent time steps, not depth layers.
• IN rnns-are-turing-complete
  RNNs are Turing complete — they can run arbitrary programs on arbitrary input sequences.
• IN second-order-rnns-use-weight-tensors
  Second-order RNNs use weight tensors w_ijk (3-index) instead of matrices, enabling direct mapping to finite-state machines in both training and representation.
• IN seq2seq-sutskever-2014-variable-length-mapping
  Seq2Seq (Sutskever, Vinyals & Le, 2014, arXiv:1409.3215) uses an encoder RNN and decoder RNN to enable variable-length input-to-output mapping for machine translation.
• IN vanishing-gradient-exponential-decay
  The vanishing gradient problem causes error gradients to decay exponentially with the time lag between relevant events, preventing standard RNNs from learning long-range dependencies.
• IN werbos-bptt-1990
  Werbos (1990) described Backpropagation Through Time (BPTT) for applying backpropagation to recurrent networks.