residual-stream-universal-substrate
OUT derived (depth 1)
Created 2026-08-25T03:00:40+00:00 · Reviewed 2026-08-25T04:28:09+00:00
SAE (middle-layer residual stream), ROME (mid-layer MLP value projection), and Park (final-layer unembedding) all identify the residual stream at different depths as the primary locus of interpretable geometric structure.
Justifications
This belief has 3 justifications — it is IN if any one holds.
SL — Three independent interpretability frameworks target the same architectural object (the residual stream) at different layers, confirming it as the universal substrate carrying both factual content and geometric structure in transformer architectures.
Antecedents (all must be IN):
- IN sae-middle-layer-residual-stream-target — The SAE is applied to residual stream activations at the middle layer specifically to reduce dimensionality versus MLP output and to mitigate cross-layer superposition.
Unless (any of these IN defeats this justification):
- IN residual-stream-universal-substrate-v2 — SAE targets the middle-layer residual stream, ROME edits the mid-layer FFN value projection (W_V), and Park's whitening applies only to the final-layer unembedding matrix. Each approach selects a specific depth as its intervention or representation target, but they do not jointly identify the residual stream as the primary locus of interpretable geometric structure; in particular, Park explicitly leaves internal-layer geometry as an open problem.
SL — Three independent interpretability frameworks target the same architectural object (the residual stream) at different layers, confirming it as the universal substrate carrying both factual content and geometric structure in transformer architectures.
Antecedents (all must be IN):
- IN rome-edits-wv-value-projection — ROME edits W_V (the value projection) in the FFN layer, where the FFN computes f(h) = W_U·g(W_V·h + b) and h is treated as the key in the key-value memory interpretation.
Unless (any of these IN defeats this justification):
- IN residual-stream-universal-substrate-v2 — SAE targets the middle-layer residual stream, ROME edits the mid-layer FFN value projection (W_V), and Park's whitening applies only to the final-layer unembedding matrix. Each approach selects a specific depth as its intervention or representation target, but they do not jointly identify the residual stream as the primary locus of interpretable geometric structure; in particular, Park explicitly leaves internal-layer geometry as an open problem.
SL — Three independent interpretability frameworks target the same architectural object (the residual stream) at different layers, confirming it as the universal substrate carrying both factual content and geometric structure in transformer architectures.
Antecedents (all must be IN):
- IN park-2025-whitening-final-layer-only — The whitening transformation used to define the canonical representation space applies only to the final layer's unembedding matrix, leaving internal-layer geometry as an open problem.
Unless (any of these IN defeats this justification):
- IN residual-stream-universal-substrate-v2 — SAE targets the middle-layer residual stream, ROME edits the mid-layer FFN value projection (W_V), and Park's whitening applies only to the final-layer unembedding matrix. Each approach selects a specific depth as its intervention or representation target, but they do not jointly identify the residual stream as the primary locus of interpretable geometric structure; in particular, Park explicitly leaves internal-layer geometry as an open problem.
Dependents
These beliefs depend on this one:
- OUT covariance-geometry-unifies-analysis-and-editing — The mathematically principled framework for both interpreting (SAE feature extraction, Park polytope analysis) and modifying (ROME rank-one edits) LLM representations is second-moment covariance geometry applied to the residual stream, since C = KKᵀ whitening defines the canonical coordinate system in which all three operations become linear algebra on the same substrate.
- OUT feature-level-editing-reliability — SAE-identified features can serve as interpretable, causally-grounded targets for knowledge editing—specifying edits in semantic feature space rather than raw weight matrices—because the residual stream is the universal substrate and knowledge is locally stored, contingent on feature activations being causally meaningful rather than mere statistical correlates.
- OUT multi-model-editing-transferability — Knowledge editing strategies (ROME-style local rank-one updates) are transferable across model families and architectures because the geometric structure of the residual stream is universal, provided that the geometric structure fully captures compositional (multi-hop) knowledge rather than only single-hop associations.
- OUT superposition-necessitates-covariance-whitening — Over-complete superposition in the shared residual-stream substrate is the precise structural condition that necessitates covariance/whitening (second-moment projection) as the canonical tool for isolating individual features and performing targeted rank-one edits; without superposition, raw Euclidean geometry would suffice and the entire covariance-geometry framework would be unnecessary.