Transformer Architecture

109 beliefs (109 IN, 0 OUT)

• IN all-training-objectives-are-compression-variants-v2
  The three main Transformer training task types — masked language modeling (BERT-style bidirectional), autoregressive/causal (GPT-style left-to-right), and PrefixLM (hybrid bidirectional prefix with causal generation) — are all forms of self-supervised prediction, and Chinchilla's establishment that model quality directly measures data compression capability provides an information-theoretic lens through which they can be viewed as differing in predictive approach rather than in fundamental kind.
• IN alphafold-transformer-protein-folding
  AlphaFold uses the Transformer architecture for protein structure prediction, demonstrating Transformers solving scientific problems beyond language tasks.
• IN annotated-transformer-rush-2018-pytorch
  The Annotated Transformer (Rush, 2018) provides a step-by-step PyTorch implementation of the Transformer architecture for educational purposes.
• IN anthropic-sae-512-neuron-mlp-architecture
  The Bricken et al. (2023) paper uses a one-layer transformer with a 512-neuron MLP using ReLU activation, trained on 8 billion tokens
• IN attention-mechanism-bahdanau-2014
  The attention mechanism for neural machine translation was introduced by Bahdanau et al. in 2014, and was a direct precursor to the transformer architecture
• IN attention-universality-extends-through-modality-agnostic-tokenization
  Attention's validated universality — grounded in its structural computational richness (asymmetric, position-dependent, learned scaling) — extends to arbitrary modalities through a common tokenization abstraction: ViT converting image patches to token sequences, AlphaFold processing protein residues, and Decision Transformer treating RL trajectories as sequences all demonstrate that attention's universality is not domain-specific but architecture-inherent, requiring only a tokenization bridge to apply.
• IN bert-encoder-only-cannot-generate-text
  BERT's encoder-only architecture cannot naturally generate text; using multiple [MASK] tokens as a generation mechanism constitutes dataset shift and degrades performance.
• IN bert-established-pretrain-finetune-paradigm
  BERT (Devlin et al., 2018) is an encoder-only Transformer pre-trained on BookCorpus and English Wikipedia using two simultaneous self-supervised objectives — masked language modeling (MLM) and next sentence prediction (NSP) — to achieve bidirectional pre-training. Its [CLS] token output serves as a sequence-level representation for classification tasks, while individual token outputs support token-level tasks, enabling a single pre-trained model to be adapted to diverse downstream tasks.
• IN bert-masked-language-modeling-2018
  BERT (Devlin et al., 2018) is an encoder-only Transformer that uses masked language modeling (MLM) and next sentence prediction for bidirectional pre-training.
• IN bert-paradigm-survived-its-own-architectural-obsolescence
  BERT's pretrain-then-fine-tune paradigm persisted even as decoder-only models superseded encoder-only architectures in the scaling race — the methodology that BERT proved was inherited by the very architecture class that replaced it, demonstrating that methodological contributions can outlast the architectures that introduce them.
• IN cad-models-evaluated
  CAD is evaluated on OPT (13B/30B), GPT-Neo (2.7B/20B), LLaMA (13B/30B), and FLAN-T5 (XL 3B / XXL 11B), spanning decoder-only and encoder-decoder architectures.
• IN chinchilla-information-theory-predicts-parameter-redundancy-v2
  Chinchilla's information-theoretic grounding — establishing that language model quality directly measures data compression capability — offers a conceptual lens under which the significant parameter redundancy observed in transformer models (95% performance at 60% of parameters, cross-layer sharing, weight tying) becomes more interpretable: if language modeling is fundamentally tied to compression and models carry surplus capacity relative to that compression target, redundancy is a plausible theoretical expectation, lending coherence to what would otherwise be a purely empirical observation.
• IN conformer-convolution-augmented-transformer-speech
  Conformer (Gulati et al., 2020) is a convolution-augmented Transformer architecture for speech recognition that injects local convolution into the Transformer block and was deployed in Google Speech.
• IN covariance-geometry-as-canonical-tool
  Independent lines of work (ROME's key-space projection and Park's unembedding whitening) converge on using empirical second-moment matrices to define the "correct" inner product for reasoning about transformer representations.
• IN dalle1-not-diffusion-uses-autoregressive-vae
  DALL-E 1 is not a diffusion model; it uses a decoder-only autoregressive Transformer for image token generation plus a VAE decoder for pixel reconstruction, distinguishing it from later diffusion-based models (Stable Diffusion, DALL-E 2/3)
• IN decision-transformer-rl-as-sequence-modeling
  Decision Transformer (Chen et al., 2021) recasts reinforcement learning as sequence modeling, applying the Transformer architecture to RL problems.
• IN decoder-only-dominance-driven-by-data-scaling-advantage
  Decoder-only architectures became dominant in the scaling era, coinciding with evidence that data volume matters more than parameter count for model performance. Autoregressive next-token prediction's ability to leverage massive unsupervised text corpora may have been a contributing factor in this dominance.
• IN decoder-only-won-the-scaling-race
  Decoder-only architecture became the dominant paradigm for large language models, with GPT, Llama, and most frontier models choosing autoregressive causal modeling over encoder-only or encoder-decoder variants.
• IN dola-contrastive-decoding-formula
  DoLa (Chuang et al., 2023) computes next-token probability as p(next) ∝ exp(log_p_mature − log_p_premature), contrasting log-probabilities between mature (late) and premature (early) transformer layers.
• IN eh2022-circuit-defined-sufficiency-necessity
  A transformer circuit is defined as a minimal set of model components that is both sufficient for the task and minimal (no proper subset reproduces the behavior), not merely correlated with it
• IN fast-weight-networks-1992-equivalent-linear-transformer
  Fast-weight/dynamic-link networks (1992) are mathematically equivalent to the unnormalized linear transformer, providing a theoretical predecessor to the 2017 architecture.
• IN feature-manifolds-higher-dimensional
  Features in transformer networks may not be constrained to one-dimensional directions but may occupy higher-dimensional subspaces called feature manifolds, where a single semantic concept varies across a patch of activation space
• IN ffn-original-relu-modern-geglu-swiglu
  The original 2017 Transformer paper used ReLU as the FFN activation function; modern variants (GPT-3, LLaMA) have replaced it with GeGLU/SwiGLU.
• IN ffn-size-typically-4x-embedding-size
  The feedforward network intermediate size (d_ffn) is typically 4 times the embedding dimension (d_emb), as used in GPT-2 and BERT.
• IN flan-ul2-position-spread-2048
  Flan-UL2 (2048-token encoder / 512-token decoder) shows only 1.9% absolute difference between best- and worst-case positions within its trained 2048-token context window, but exhibits U-shaped degradation when evaluated beyond 2048 tokens.
• IN foundational-innovations-resist-proprietary-containment
  Two major foundational innovations in the LLM field face intellectual property challenges that may limit proprietary containment: Google's Transformer patent (US 10,452,978, issued 2019) has seen widespread unlicensed adoption, while OpenAI's attempt to trademark 'GPT' encountered a USPTO office action indicating the term may be too descriptive or generic for US trademark protection (though it was secured in the EU and Switzerland). These cases suggest a pattern where widely adopted AI contributions may outpace IP protections, though the outcomes remain partially unresolved.
• IN frozen-transformer-executes-arithmetic-circuits
  Lu et al. (2022) demonstrated that a fine-tuned frozen transformer can execute arithmetic and logic circuits without retraining, suggesting general-purpose programmability of the architecture.
• IN gelu-approximates-multiplication-via-third-order-expansion
  The identity xy ≈ √(π/2)·(GeLU(x+y) − GeLU(x) − GeLU(y)) holds for small x, y with error O(x³+y³), allowing a transformer MLP to perform element-wise multiplication using three GeLU evaluations.
• IN google-ai-timeline-transformer-between-mobilenet-efficientnet
  In Google's neural-network research taxonomy, the Transformer (2017) is positioned between MobileNet (2017) and EfficientNet (2019) in the chronological lineage.
• IN google-contributions-became-universal-foundation-beyond-google
  Google's research contributions (Transformer architecture, pretrain-finetune paradigm via BERT) became the universal foundation for all frontier models — with the pretrain-finetune paradigm notably outlasting the encoder-only architecture that introduced it — demonstrating that foundational innovations transcend their originating organization and even their originating architectural context.
• IN google-model-lineage-transformer-to-gemini
  Google's core language model evolution: Transformer (2017) → BERT (2018) → T5 (2019) → LaMDA (2021) → PaLM (2022) → Gemini (2023)
• IN google-paradox-validated-uncontainable-innovation-then-exported-entire-ecosystem
  Google's paradox encompasses the full innovation lifecycle: it validated the Transformer at unprecedented web scale (Search and Translate simultaneously), demonstrated its universal cross-domain applicability (through lineage spanning Transformer, BERT, CoT, and Gemini), and then watched these contributions become the universal foundation for all frontier competitors — the strongest validator became the strongest proof that validation cannot be captured.
• IN google-research-lineage-spans-transformer-to-frontier
  Google's research lineage connects the original Transformer (2017), BERT (2018), Chain-of-Thought prompting (2022), and the model evolution through to Gemini — a continuous thread from foundational architecture to frontier capability.
• IN google-simultaneously-deployed-patented-and-published
  Google both deployed the Transformer internally (Google Translate transitioning from LSTM-based architectures by 2020) and sought proprietary protection (US Patent 10,452,978, filed 2019), illustrating that the company pursued parallel strategies of practical adoption and intellectual property containment for the same architecture.
• IN google-strongest-validator-couldnt-contain-what-it-validated
  Google provided the strongest empirical validation of the Transformer at web scale — deploying BERT to Search (every English query by October 2019, 70+ languages by December) and Translate simultaneously — yet was the institution that most demonstrated innovation's resistance to proprietary containment, as its inventions became the universal foundation for all frontier competitors including direct rivals.
• IN google-translate-lstm-to-transformer-2020
  Google Translate transitioned from statistical MT to LSTM-based seq2seq (2016) to a transformer-encoder/RNN-decoder architecture by 2020.
• IN google-validated-transformer-at-web-scale-across-task-types
  Google's parallel deployment of Transformer-based models to both Search (BERT for ranking, October 2019, processing every English query) and Translate (replacing LSTM architecture, 2020) validated the Transformer at web scale across fundamentally different NLP task types — understanding/ranking vs. generation — providing the strongest early evidence that architectural generality extends to production-scale deployment, not just benchmark performance.
• IN gpt-1-introduced-june-2018
  GPT-1 was introduced by OpenAI on June 11, 2018, as the first application of generative pre-training to the transformer architecture.
• IN gpt-architecture-uses-decoder-only-for-next-token-prediction
  GPT models use only the decoder portion of the transformer architecture, trained to predict the next token in a sequence, distinguishing them from encoder-only (BERT) or encoder-decoder (T5) variants.
• IN gpt1-117m-params-bookcorpus-june-2018
  GPT-1 (June 2018, OpenAI) had 117 million parameters and was the first decoder-only generative pre-training LLM, using BookCorpus for pre-training followed by discriminative fine-tuning.
• IN gpt1-released-june-2018
  GPT-1 was introduced by OpenAI on June 11, 2018, trained on BookCorpus using decoder-only transformer with generative pre-training
• IN huggingface-transformers-emnlp-2020-demo
  The HuggingFace Transformers library (Wolf et al.) was first presented as a demo at EMNLP 2020 and is the de-facto Python toolkit for pre-trained Transformer models.
• IN innovation-frontier-shifted-from-architecture-to-configuration
  The LLM innovation frontier has shifted from macro-architecture selection (settled: decoder-only transformer via data-driven dominance) to micro-architecture configuration (unsettled: tokenization, positional encoding, normalization) and data strategy optimization.
• IN knowledge-circuits-memory-context-heads
  In LLM interpretability, 'memory heads' recall internal parametric knowledge while 'context heads' retrieve from external context, and they can have opposite effects in later transformer layers (Chughtai et al. 2024; Huang et al. 2023).
• IN llama-core-architecture-rope-rmsnorm-glu
  Llama's transformer architecture uses Rotary Position Embeddings (RoPE) for positional encoding, RMSNorm (not standard LayerNorm) for normalization, and GLU-based feed-forward layers.
• IN llama-decoder-only-autoregressive
  Llama is a decoder-only autoregressive transformer architecture (like GPT-3), not encoder-decoder or encoder-only
• IN llama2-unembedding-not-universal-concept-encoder
  Not all concepts are linearly encoded in LLaMA-2's unembedding space (Γ); 'thing ⇒ part' is explicitly cited as a counterexample where the unembedding representation fails to capture the concept, establishing a boundary condition for the Linear Representation Hypothesis.
• IN llava-only-linear-projection-layer-finetuned
  In the LLaVA architecture, the ViT-L/14 vision encoder and Vicuna-13B language model are bridged by a single linear projection layer, and only that projection layer is finetuned while both backbone models remain frozen
• IN llm-lost-in-middle-serial-position-effect
  Transformer-based language models exhibit a U-shaped performance curve in long contexts: accuracy is highest when the relevant information is at the beginning or end of the input, and lowest when it is in the middle.
• IN local-storage-distributed-acquisition
  Factual knowledge is acquired through distributed corpus exposure (Kandpal's log-linear document-count dependence) but stored in a locally addressable MLP slot (ROME's single-layer FFN edit), revealing a two-phase knowledge pipeline.
• IN lstm-1997-vanishing-gradient
  LSTM (Hochreiter & Schmidhuber, 1997) solved the vanishing gradient problem in RNNs using multiplicative gating units and was the dominant sequence modeling architecture until Transformers in 2017.
• IN lstm-dominant-1997-to-2017
  LSTM (1997) was the dominant sequence modeling architecture until transformers replaced it in 2017, a 20-year reign.
• IN lstm-hochreiter-schmidhuber-1997-standard-reference
  The standard LSTM reference is Hochreiter & Schmidhuber, 1997, published in Neural Computation 9(8):1735-1780 (doi:10.1162/neco.1997.9.8.1735).
• IN mallen-2023-decoder-only-models-only
  Mallen et al. (2023) evaluate only decoder-only LMs; encoder-only models using [MASK] fill-in-the-blank probing are explicitly excluded.
• IN mamba-ssm-based-non-transformer-alternative
  Mamba is a state-space model (SSM)-based architecture that serves as a non-Transformer alternative for sequence processing, while GPT is a Transformer variant.
• IN memit-scales-to-10000-edits
  MEMIT scales to up to 10,000 simultaneous factual edits in a transformer LM, whereas prior state-of-the-art (SERAC) reached only 75
• IN model-compression-reveals-significant-parameter-redundancy
  Multiple independent compression techniques — knowledge distillation retaining 95% performance at 60% of parameters (DistilBERT), cross-layer parameter sharing (ALBERT), and weight tying between input/output embeddings — demonstrate that transformer models contain significant parameter redundancy exploitable without meaningful quality loss.
• IN openai-finetune-transformer-lm-2018
  OpenAI published a reference implementation for fine-tuning Transformer language models in 2018 (`finetune-transformer-lm`), predating the HuggingFace ecosystem.
• IN original-paper-row-vector-convention
  The original 2017 Transformer paper uses the row-vector convention, writing linear layers as xW (multiply on the right) rather than Wx (multiply on the left).
• IN original-transformer-100m-parameters
  The original Transformer model from 'Attention Is All You Need' had approximately 100 million parameters.
• IN original-transformer-relu-gpt-bert-gelu
  The original 2017 transformer used ReLU as its feedforward activation function; GPT-1 and BERT use GELU; Llama and PaLM use SwiGLU (a gated linear unit variant)
• IN original-warmup-linear-scale-2pct-training-steps
  The original 2017 Transformer paper recommended linearly scaling the learning rate from 0 to its maximum over the first approximately 2% of total training steps, then decaying.
• IN park-2023-testbed-llama2-7b-decoder-only
  Park et al. (2023) use LLaMA-2 7B (a decoder-only Transformer with 32K context window) as their testbed model for empirically validating the linear representation hypothesis and causal inner product.
• IN prefix-tuning-continuous-vector-per-layer
  Prefix-tuning (Li & Liang, 2021) learns a continuous prefix vector P ∈ ℝ^{L×d} in each Transformer layer, optimizing only that vector while keeping all model weights frozen.
• IN prefix-tuning-vs-prompt-tuning-difference
  Prefix tuning prepends learned continuous vectors to all transformer layers (Li & Liang 2021), while prompt tuning prepends them only to the input layer (Lester et al. 2021)
• IN reward-model-embodies-surviving-pretrain-finetune-paradigm
  The RLHF reward model — a pretrained language model with its final layer replaced by a regression head — is itself an instance of BERT's surviving pretrain-finetune paradigm, meaning the decoder-only era's core alignment component is architecturally defined by an encoder-era methodology that outlived the architecture it was designed for.
• IN ridge-regression-requires-od-squared-hidden-space
  Implementing ridge regression via iterative Sherman-Morrison matrix inversion in a transformer requires O(d²) hidden space due to matrix-matrix multiplication, compared to O(n+d) for the SGD chain.
• IN ripple-edits-rome-mechanism
  ROME performs rank-one updates to Transformer MLP layer weights to modify specific factual associations in a language model.
• IN rome-applicable-autoregressive-decoder
  ROME is applicable to autoregressive (decoder-only) transformers, in contrast to several prior editing methods that focus on masked LMs (BERT) or seq2seq models (BART, T5).
• IN rome-edits-wv-value-projection
  ROME edits W_V (the value projection) in the FFN layer, where the FFN computes f(h) = W_U·g(W_V·h + b) and h is treated as the key in the key-value memory interpretation.
• IN rome-single-layer-ffn
  ROME localizes factual knowledge to a specific Transformer layer and updates only the feedforward network in that layer
• IN rwkv-hybrid-linear-attention-architecture
  RWKV (Peng et al., 2023, arXiv:2305.13048) is a hybrid linear-attention architecture described as 'Reinventing RNNs for the Transformer Era.'
• IN sae-universality-across-models
  SAEs applied to different transformer models produce mostly similar features—more similar to each other than to their own model's neurons—suggesting features reflect data structure rather than architecture
• IN saey-target-architecture-one-layer-512-mlp
  The Bricken et al. 2023 monosemanticity paper uses a one-layer transformer with a 512-neuron MLP layer as the base model for SAE training.
• IN saey-target-needs-relu-mlp-for-decomposition
  A one-layer transformer with an attention layer plus a ReLU-activated MLP is identified as the simplest architecture that requires a feature decomposition strategy, unlike attention-only networks which can be analyzed without one.
• IN sbert-clustering-65-hours-vs-5-seconds
  Hierarchical clustering of 10,000 sentences takes ~65 hours with BERT cross-encoder (~50M pair computations) but ~5 seconds with SBERT pre-computed embeddings.
• IN sbert-nli-large-sts-spearman-76-55
  SBERT-NLI-large achieves 76.55 average Spearman correlation on unsupervised STS tasks, beating Universal Sentence Encoder (71.22) and InferSent-GloVe (65.01).
• IN sbert-outperforms-use-6-of-7-sts
  SBERT outperforms Universal Sentence Encoder on 6 of 7 unsupervised STS datasets; the sole exception is SICK-R where USE's diverse training data is better matched.
• IN sbert-senteval-avg-87-41-vs-use-85-10
  SBERT achieves SentEval average accuracy of 87.41 (base) / 87.69 (large), outperforming InferSent-GloVe (85.59) and Universal Sentence Encoder (85.10) by ~2 percentage points.
• IN sbert-sickr-only-loss-to-use
  The only STS dataset where SBERT underperforms Universal Sentence Encoder is SICK-R (SBERT 72.91 vs. UER 76.69).
• IN sbert-sts-benchmark-gains
  SBERT outperforms InferSent by 11.7 STS points and Universal Sentence Encoder by 5.5 STS points on 7 STS tasks.
• IN sgd-implies-on-layers-and-on-plus-d-hidden-space
  A single SGD update w′ = w − 2α(x(wᵀx − y) + λw) decomposes into 9 sequential RAW operations, and generalizing to n examples requires O(n) transformer layers and O(n+d) hidden space.
• IN skip-thought-vectors-kiros-2015
  Skip-Thought Vectors (Kiros et al., 2015) use an encoder-decoder architecture that trains sentence representations by predicting neighboring sentences in a document.
• IN t5-excluded-due-to-qa-pretraining
  Encoder-decoder models such as T5 are excluded from Mallen et al. (2023) evaluation because their supervised pretraining already includes QA, confounding the memorization analysis.
• IN t5-text-to-text-framing
  T5 (Raffel et al., 2020) is an encoder-decoder Transformer that frames all NLP tasks (classification, translation, summarization) as text-to-text sequence generation.
• IN tensor2tensor-google-transformer-nmt-library
  Tensor2Tensor is Google's open-source library for building and training Transformer-based models, authored by Kaiser (2017) and presented at WMT 2018 (aclanthology.org/W18-1819).
• IN three-training-task-types-masked-autoregressive-prefixlm
  The three main Transformer training task types are masked language modeling (BERT-style), autoregressive/causal language modeling (GPT-style), and prefixLM (T5-style, where a prefix provides context and the model autoregressively predicts the target).
• IN token-id-bijective-mapping-reversible
  The token-to-integer-ID mapping in transformer tokenization is bijective (reversible), which is critical for the decoder to convert output integer IDs back into legible text without loss.
• IN transformer-architecture-generalizes-beyond-nlp
  The Transformer architecture demonstrates domain generality far beyond NLP — solving protein structure prediction (AlphaFold), playing grandmaster-level chess without search, and recasting reinforcement learning as sequence modeling (Decision Transformer) — revealing it as a general-purpose sequence processing architecture rather than a language-specific one.
• IN transformer-chess-grandmaster-no-search
  A Transformer using only static evaluation (no Minimax search) achieved grandmaster-level chess play with an Elo rating of 2895.
• IN transformer-eight-google-authors
  The original Transformer paper had eight authors, all from Google: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin.
• IN transformer-foundation-for-bert-and-llms
  The Transformer architecture is the foundation for BERT, large language models, and modern neural machine translation
• IN transformer-introduced-2017-attention-is-all-you-need
  The transformer architecture was introduced in the 2017 paper 'Attention Is All You Need' by Google researchers, with the original model having approximately 100M parameters.
• IN transformer-is-mature-and-over-parameterized
  The transformer architecture has reached maturity: post-2017 refinements (Pre-LN, SwiGLU, RMSNorm, GQA) improve stability and efficiency without changing fundamentals, while compression evidence (95% performance at 60% parameters, cross-layer sharing, weight tying) reveals substantial redundancy — future gains come from scaling strategy and efficiency engineering, not architectural revolution.
• IN transformer-maturity-is-split-between-macro-and-micro-architecture
  The transformer exhibits split-level architectural maturity: macro-architecture (encoder/decoder choice, attention mechanism, layer structure) is settled and demonstrably over-parameterized, while micro-architecture (positional encoding strategy, tokenization scheme) remains actively contested with no convergent solution — suggesting the fundamental computation is solved but its interface to input representation is not.
• IN transformer-mlp-privileged-basis-correspondence
  Transformer MLP layers and convolutional network neurons correspond to the privileged-basis model, while word embeddings and the transformer residual stream correspond to the no-privileged-basis model.
• IN transformer-original-implementation-tensorflow
  The original Transformer reference implementation (Vaswani et al., 2017) was written in TensorFlow.
• IN transformer-patent-google-2019
  The Transformer is covered by US Patent 10,452,978 ('Attention-based sequence transduction neural networks'), assigned to Google LLC, issued 2019-10-22.
• IN transformer-patent-title-and-issue-date
  The Transformer patent (US 10,452,978) was issued on 2019-10-22, titled 'Attention-based sequence transduction neural networks,' assigned to Google LLC, and names all eight authors.
• IN transformer-superseded-lstm-via-parallelism
  The Transformer ended LSTM's 20-year dominance (1997–2017) by replacing sequential recurrence with parallel self-attention, enabling massive scaling.
• IN transformer-three-architecture-variants
  The three main Transformer architecture variants are encoder-only (e.g., BERT, bidirectional), decoder-only (e.g., GPT, autoregressive/causal), and encoder-decoder (e.g., T5, original Transformer).
• IN transformer-universal-turing-machine-simulation
  Phuong & Hutter (2022, arXiv:2207.09238) proved that transformers with finite precision can simulate universal Turing machines, linking the architecture to theoretical computer science.
• IN two-stage-nli-sts-training-improvement
  Two-stage training (NLI then STS) yields a 1–2 point Spearman improvement for SBERT and 3–4 points for the BERT cross-encoder over STS-only training.
• IN ul2-unifies-three-training-paradigms-single-objective
  UL2 (Tay et al., 2023, arXiv:2205.05131) unifies masked language modeling, causal language modeling, and span-corruption objectives into a single training paradigm within one Transformer model.
• IN use-benchmark-67-cpu-1318-gpu
  Universal Sentence Encoder achieves 67 sentences/second on CPU and 1318 sentences/second on GPU in the SBERT paper's benchmark.
• IN use-trec-outperforms-sbert
  Universal Sentence Encoder outperforms SBERT on TREC question classification (93.2 vs. 89.6/87.4), attributed to USE's pre-training on question-answering data.
• IN vaswani-transformer-2017
  Vaswani et al. published the original Transformer architecture paper in 2017.
• IN vit-16x16-patches-image-tokens
  Vision Transformer (ViT) (Dosovitskiy et al., 2021) divides images into 16x16 patches and treats them as token sequences, extending the Transformer architecture to image recognition.
• IN whisper-trained-680k-hours-weak-supervision
  Whisper (Radford et al., 2022, arXiv:2212.04356) is an encoder-decoder Transformer trained on 680,000 hours of weakly supervised speech data for robust automatic speech recognition.