Classical ML Methods

66 beliefs (66 IN, 0 OUT)

• IN alibi-linear-bias-attention
  ALiBi adds a linear bias matrix B (where B_{i,j} = j-i) scaled by scalar s directly to attention logits, enabling short-context pretraining followed by long-context fine-tuning.
• IN alpac-report-1966-cut-mt-funding
  The ALPAC report (1966) dramatically cut machine translation funding after a decade of unmet expectations, effectively defunding MT research for years.
• IN alphafold-2020-protein-structure-prediction
  AlphaFold (2020) achieved unprecedented accuracy in protein structure prediction from amino acid sequences, far beyond prior computational methods
• IN alphafold-transformer-protein-folding
  AlphaFold applies transformers to protein structure prediction, demonstrating the architecture generalizes beyond language to biological sequence analysis.
• IN attention-architecturally-entrenches-diagnostic-futility
  The attention mechanism's hardware synergy may reinforce NLP's position as a limiting case of ML's diagnostic constraints — attention-friendly hardware specialization makes alternatives economically unviable, potentially entrenching the architecture that both enabled NLP's transformative capabilities and contributes to the domain where diagnostic capacity appears least able to resolve the underlying reliability crisis, suggesting that NLP's crisis position is resistant to change through architectural alternatives.
• IN attention-asymmetric-stabilized-mathematical-structure
  The attention mechanism has a carefully engineered mathematical structure: scaled dot-product computation with sqrt(d_k) normalization prevents gradient explosion in high dimensions, while the asymmetry (Q/K projections differ) means token-to-token relevance is inherently directional — attention from A to B does not imply attention from B to A.
• IN attention-evolution-extends-convergent-discovery-pattern
  The attention mechanism's independent evolution through multiple paradigms (connectionist models 1982 → fast weights 1992 → additive attention 2014 → scaled dot-product 2017) extends the convergent discovery pattern established for gradient computation, gradient flow, and weight sharing — attention's mathematical form was converged upon across disconnected research traditions rather than invented, suggesting it is another mathematical necessity of sequence-aware computation.
• IN attention-evolution-timeline
  Attention mechanism evolution: connectionist models (1982) → fast weights (1992) → additive attention (Bahdanau 2014) → multiplicative attention (Luong 2015) → self-attention (Vaswani 2017)
• IN attention-formula-scaled-dot-product
  The scaled dot-product attention formula is Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) * V.
• IN attention-hardware-synergy-locks-nlp-at-crisis-apex
  The attention mechanism's hardware synergy both enabled NLP's transformative capabilities and locked it at the crisis apex — attention is load-bearing for NLP's position as the domain where pragmatism's dual innovation-crisis dynamic is most extreme, while hardware specialization for attention-friendly architectures (GPU-optimized parallel matrix multiplication) entrenches this position by making alternatives to attention-based architectures economically unviable.
• IN attention-load-bearing-for-nlp-crisis-apex
  The attention mechanism illustrates a notable intersection within NLP's role as the purest exemplar of ML's pragmatism-crisis dynamic — attention's pragmatically discovered mathematical precision (scaled dot-product stabilization, engineered asymmetry) enabled the RNN-to-Transformer paradigm shift, representing a rare case where getting the mathematics right was prerequisite for pragmatic success within the subfield that most dramatically demonstrates both the innovation power and consequences of hardware-driven pragmatic selection.
• IN attention-non-symmetric
  Attention is non-symmetric: high attention from token i to token j does not imply high attention from j to i, because W^Q and W^K are different matrices.
• IN attention-scaling-factor-sqrt-dk
  Attention scores are divided by sqrt(d_k) (not d_k) to stabilize gradients during training.
• IN bengio-2003-mlp-beat-ngram-language-model
  In 2003, Bengio et al. demonstrated a multi-layer perceptron that outperformed the best statistical n-gram model for language modeling, trained on up to 14 million words.
• IN bert-encoder-only-gpt-decoder-only-t5-enc-dec
  BERT is encoder-only (masked LM task), GPT is decoder-only (autoregressive task), and T5 is encoder-decoder (prefixLM task).
• IN bert-gpt-divergence-from-common-ancestor
  BERT and GPT represent a speciation event from a common ancestor: both derived from the 2017 Transformer in the same year (2018), but chose opposite pretraining strategies (masked vs. causal LM) and opposite architectural halves (encoder vs. decoder), yielding fundamentally different capability profiles.
• IN bert-masked-lm-gpt-causal-lm-same-year-2018
  BERT (2018) uses masked language modeling (bidirectional) while GPT (2018) uses causal language modeling (autoregressive left-to-right); both published the same year with opposite pre-training strategies
• IN bptt-vs-rtrl-locality-tradeoff
  BPTT is O(weights) per step but requires storing all activations (local in space, not time); RTRL is O(hidden × weights) per step but is online (local in time, not space).
• IN california-deepfake-laws-2019
  California signed AB-602 (banning non-consensual synthetic pornography) and AB-730 (banning manipulated political candidate videos within 60 days of elections) in 2019, effective 2020
• IN chatgpt-launched-nov-30-2022-gpt35-rlhf
  ChatGPT launched November 30, 2022, built on GPT-3.5 with RLHF
• IN chomskyan-linguistics-antagonistic-to-statistical-nlp
  Chomskyan linguistics was historically antagonistic to statistical/ML approaches to language, due to the 'poverty of the stimulus' argument and focus on constructed corner cases rather than corpus data.
• IN conll-multilinguality-1-to-100-languages-1999-2018
  CoNLL shared tasks expanded from 1 language (English, 1999) to over 100 languages by 2018, illustrating NLP's growing multilinguality.
• IN conll-three-trend-lines-through-2020
  Three major NLP trend lines observed from CoNLL Shared Tasks through 2020: increasing cognitive abstraction, growing multilinguality/multimodality, and elimination of symbolic representations.
• IN coreference-broader-than-anaphora-resolution
  Coreference resolution is broader than anaphora resolution — it includes bridging relationships and not just pronoun matching.
• IN cross-attention-queries-decoder-kv-encoder
  In cross-attention, queries come from the decoder and keys/values come from the encoder output.
• IN decoder-only-removes-cross-attention
  In decoder-only models (e.g., GPT), the cross-attention sublayer is removed entirely, leaving only 2 sublayers: masked self-attention and FFN.
• IN deep-learning-overtook-statistical-nlp-around-2015
  Deep learning overtook statistical NLP as the dominant framework around 2015, despite initial skepticism when Richard Socher presented a deep learning tutorial at ACL 2012.
• IN ffn-intermediate-4x-embedding
  The feedforward network intermediate size is conventionally 4x the embedding size (d_ffn = 4 * d_emb) in GPT-2 and BERT families.
• IN flashattention-exact-communication-avoiding
  FlashAttention is a communication-avoiding algorithm that computes exact attention (not an approximation) using block matrix multiplications sized to fit GPU cache.
• IN flashattention2-230tflops-a100
  FlashAttention-2 achieves up to 230 TFLOPS/s on A100, 2x faster than v1 and 9x over standard PyTorch attention.
• IN georgetown-experiment-1954-first-auto-translation
  The Georgetown experiment (1954) was the first automatic translation demonstration, translating Russian to English.
• IN google-model-chronology-bert-to-gemma
  Google's key model chronology: BERT (2018) → T5 (2019) → LaMDA (2021) → Chinchilla (2022) → PaLM (2022) → Gemini (2023) → Gemma (2024)
• IN hallucination-recognized-nlp-taxonomy
  Hallucination is formally recognized as a related problem in the NLP field taxonomy
• IN hmm-pos-tagging-marked-transition-from-rules
  The application of Hidden Markov Models to part-of-speech tagging marked the real transition away from rule-based NLP toward statistical methods.
• IN instructgpt-first-major-rlhf-application
  InstructGPT was the first major application of RLHF to language models for instruction-following; ChatGPT built on this approach for response quality and safety
• IN koskenniemi-1983-two-level-morphology
  Koskenniemi (1983) introduced two-level morphology, an early computational model for word-form recognition and production, and a classic computational linguistics contribution.
• IN ml-compression-learning-equivalence
  Prediction and compression are formally equivalent; language models can exceed PNG/FLAC in lossless compression (Delétang et al., 2023: 'language modeling is compression')
• IN ml-netflix-prize-ensemble
  The Netflix Prize (2006-2009) was won by an ensemble model achieving 10%+ improvement over Netflix's Cinematch algorithm; Netflix later shifted from ratings to behavioral signals for recommendations
• IN mqa-gqa-mla-kv-cache-variants
  MQA shares K/V projections across all heads; GQA partitions heads into groups sharing K/V; MLA uses low-rank projection to compress KV cache to low-dimensional latent vectors.
• IN ner-also-called-token-classification
  Named Entity Recognition (NER) identifies proper names and their types (person, location, organization) and is also called token classification.
• IN nlp-ai-completeness-guarantees-permanent-crisis-epicenter
  NLP's classification as AI-complete provides a structural reason to expect it will remain near the epicenter of ML's reliability crisis for the foreseeable future — since full NLP requires solving the general AI problem, NLP is likely to continue occupying the frontier where capability advances outpace reliability, making it a domain where the gap between what models can do and what can be done reliably tends to grow rather than shrink, even as methodological progress occurs.
• IN nlp-classified-ai-complete
  Natural Language Processing is classified as AI-complete, meaning full NLP requires solving the general AI problem
• IN nlp-distributional-semantics-models
  Key distributional semantics models in NLP include BERT, Word2Vec, GloVe, fastText, Transformers, LSTM, and Seq2Seq
• IN nlp-doubly-contingent-and-paradigm-validating
  NLP simultaneously validates ML's hardware-driven paradigm selection law and demonstrates its most extreme consequence — NLP's trajectory independently confirms that scalability trumps theory while its own pretraining dominance is doubly hardware-contingent, making NLP both the strongest evidence for economic evolution and the paradigm most vulnerable to hardware shifts.
• IN nlp-machine-translation-approaches
  Machine translation approaches include rule-based, statistical, example-based, transfer-based, neural, and computer-assisted methods
• IN nlp-standard-language-resources
  Standard NLP language resources include WordNet (lexical database), FrameNet (semantic frames), Universal Dependencies (syntactic annotation), and PropBank (predicate-argument structures)
• IN nlp-text-segmentation-pipeline
  The NLP text segmentation pipeline proceeds: lexical analysis → stemming/lemmatization → sentence segmentation → word segmentation → chunking
• IN nlp-three-paradigms-symbolic-statistical-neural
  NLP has been driven by three major paradigms historically: symbolic/rule-based (1950s-early 1990s), statistical/ML (1990s-present), and neural network approaches (2010s-present).
• IN prefixlm-bidirectional-prefix-causal-rest
  PrefixLM is a decoder-only variant with a modified mask where prefix tokens attend bidirectionally and subsequent tokens attend causally.
• IN reformer-n-log-n-bigbird-linear
  Reformer achieves O(N log N) attention via locality-sensitive hashing; BigBird achieves O(N) via random small-world networks.
• IN rl-paradigm-dissolution-validates-crisis-universality-in-temporal-domain
  Two independent RL developments — Decision Transformer dissolving the RL/sequence-modeling boundary by absorbing RL into the Transformer's native modality, and DeepSeek-R1 eliminating the supervised fine-tuning step from the LLM pipeline — jointly validate that paradigm taxonomy dissolution extends into the temporal/decision-making domain, not just the perceptual (CV) and linguistic (NLP) domains, establishing that the crisis dynamic is truly universal across all data modalities.
• IN rope-relative-position-rotation
  RoPE (Rotary Positional Embedding) encodes position by rotating pairs of embedding dimensions by angle mθ; dot products depend only on relative position, not absolute.
• IN speculative-decoding-preserves-distribution
  Speculative decoding uses a small model to generate candidate tokens verified in parallel by the large model, preserving the large model's output distribution via stochastic accept/reject.
• IN speech-recognition-and-mt-are-ai-complete
  Speech recognition and machine translation are classified as AI-complete problems, requiring grammar, semantics, and world knowledge.
• IN stemming-rule-based-vs-lemmatization-dictionary-based
  Stemming applies rules to strip affixes (approximate), while lemmatization uses a dictionary to return the true base form (lemma) — lemmatization is more accurate but requires a dictionary.
• IN subword-tokenization-bpe-ulm
  Common subword tokenization algorithms are Byte Pair Encoding (BPE) and Unigram Language Model (ULM).
• IN swiGLU-llama-palm-gelu-gpt-bert
  SwiGLU activation is used by Llama and PaLM; GELU is used by GPT-1 and BERT.
• IN t5-text-to-text-unified-nlp-2020
  T5 (Raffel et al., 2020) frames every NLP task as text-to-text, enabling unified fine-tuning across tasks
• IN teacher-forcing-feeds-ground-truth-not-predictions
  Teacher forcing feeds ground-truth output tokens (not model predictions) to the decoder during training to prevent error accumulation across the sequence.
• IN textual-entailment-three-outcomes
  Textual entailment classification has three possible outcomes: entails, contradicts, or neutral.
• IN three-lm-task-types
  The three language modeling task types are: masked (BERT), autoregressive (GPT), and prefixLM (T5).
• IN tokenization-trivial-english-hard-cjk-thai
  Tokenization is trivial for space-delimited languages like English but a significant challenge for Chinese, Japanese, and Thai which lack space-delimited word boundaries.
• IN transformer-multi-head-attention-subspaces
  Multi-head attention runs scaled dot-product attention h times in parallel with different learned projections, allowing the model to attend to information from different representation subspaces at different positions simultaneously.
• IN weight-tying-embedding-unembedding
  Weight tying uses the transpose of the embedding matrix as the un-embedding matrix, halving embedding parameters and preventing training divergence.
• IN weight-tying-input-output-embeddings
  Press & Wolf (2017) showed that sharing (tying) input and output embedding weights improves language model performance
• IN word2vec-mikolov-grew-from-rnn-language-modeling
  Word2vec (Mikolov, ~2013) grew from applying simple RNNs to language modeling (2010), catalyzing the deep learning wave in NLP.