Alignment & Safety

48 beliefs (48 IN, 0 OUT)

• IN alignment-performance-trend-holds-for-language-benchmarks-fails-for-reasoning
  The alignment–language-performance linear trend from Huh et al. (2024) holds for HellaSwag (R²_new = 0.297) and Wikitext (R²_new = 0.489) but fails for ARC (R²_new = −0.575), GSM8K (R²_new = −1.753), MMLU (R²_new = −0.662), and LogiQA2 (R²_new = −1.414) across 36 recent LLMs, where negative R² means the regression line predicts worse than the mean.
• IN authority-hierarchy-is-both-security-mechanism-and-attack-surface
  The LLM instruction authority hierarchy — where system prompts carry higher weight than user messages — is a dual-use security architecture: the same structured privilege boundary that constrains model behavior (defense) creates the privileged channel that prompt injection attempts to impersonate (attack surface), making the security mechanism and the vulnerability architecturally inseparable.
• IN bijective-relaxation-decreases-mutual-knn-alignment
  Relaxing the one-to-one image-text correspondence assumption consistently decreases mutual kNN alignment: on non-synthetic WIT data, 7.1% of captions map to >1 image and 24.6% of images map to >1 caption; on synthetic CycleReward data (11 captions/image or 12 images/caption), mutual kNN still decreases even with a generous 'same source' matching criterion.
• IN claude-opus-22-firefox-vulns-march-2026
  Claude Opus identified 22 security vulnerabilities in Mozilla Firefox in March 2026, demonstrating autonomous bug-hunting capability.
• IN claude-opus-4-safety-level-3
  Opus 4 was classified Level 3 on Anthropic's four-point safety scale, described as 'significantly higher risk.'
• IN claude-tiered-strategy-manages-capability-safety-spectrum
  Claude's model strategy creates a structured capability-safety spectrum: three public tiers (Haiku/Sonnet/Opus) with safety classification scaling by capability (Opus 4 at Level 3), plus a restricted tier (Mythos) for the highest-capability models — systematically linking access to risk.
• IN constitutional-ai-75-guidelines-2023-to-23000-words-2026
  Anthropic's Constitutional AI constitution expanded from 75 guidelines in the 2023 version to approximately 23,000 words with detailed behavioral justifications in the 2026 version, drawing roots from the 1948 UN Universal Declaration of Human Rights.
• IN constitutional-ai-codified-in-23k-word-constitution
  Anthropic's Constitutional AI alignment framework is codified in a public ~23,000-word 'constitution' document (updated January 2026) that guides Claude's behavior toward helpfulness, honesty, and non-destruction of humanity.
• IN constitutional-ai-principles-not-per-example-feedback
  Constitutional AI (Anthropic) uses a set of written principles for alignment rather than requiring individual human feedback for each training example.
• IN constitutional-ai-rlaif-anthropic
  Anthropic's Constitutional AI is the primary example of RLAIF (Reinforcement Learning from AI Feedback), where AI-generated feedback based on constitutional principles replaces human preference labels
• IN cross-modal-alignment-drop-is-model-agnostic-across-3b-to-65b
  The cross-modal mutual kNN alignment degradation pattern holds across DINOv2-base, DINOv2-giant, Pixio-ViT-B/16, and CLIP-B/16 (vision axis) paired with OpenLlama-3b, OpenLlama-13b, OpenLlama-65b, Gemma-7B, and Mistral-7B (language axis), with embedding dimensions ranging from 768 to 8192.
• IN cross-modal-alignment-finding-generalizes-to-text-video-audio
  The cross-modal mutual kNN alignment drop with gallery scale generalizes beyond text-image to text-video (PVD-100k, VideoMAE-v2) and text-audio (LAION-Audio-100k, Dasheng audio encoder) pairs, where the LLM-performance alignment trend is weak (video) or flat (audio), and the finding is verified across DINOv2/Pixio/CLIP vision encoders and OpenLlama/Gemma/Mistral/LLaMA language models.
• IN cross-modal-mutual-knn-drops-with-gallery-scale
  Mutual kNN alignment between DINOv2-base and OpenLlama-3b drops from 0.135 (k=10) / 0.058 (k=1) on WIT-1024 to 0.008 (k=10) / 0.001 (k=1) on LAION-15M, while within-modality mutual kNN (OpenLlama-3b vs 13b at k=1) stays in [0.59, 0.62] and DINOv2-base vs giant in [0.35, 0.45] across all gallery sizes.
• IN daa-beta-controls-deviation-from-reference
  The β parameter in all Direct Alignment Algorithms (DPO, IPO, KTO) serves as KL regularization strength controlling deviation from the reference SFT policy — higher β keeps the policy closer to the reference
• IN dual-use-properties-make-security-scaling-superlinear
  LLM security challenges tend to scale with capability growth because dual-use properties like memorization — which serves as both a knowledge source and an attack vector — mean that mechanisms driving capability improvements can simultaneously deepen vulnerabilities. Evidence from training data memorization patterns and the expansion of multiple independent security surfaces (poisoning, prompt injection, architectural vulnerabilities) suggests that security surfaces don't merely grow alongside capabilities but may become structurally harder to defend, though this pattern is characterized at limited scales rather than confirmed as a universal structural law.
• IN esmfold-removes-msa-runs-10x-faster-than-alfold2
  ESMFold (protein structure prediction) removes the multiple sequence alignment (MSA) requirement used by AlphaFold2, using embeddings instead, and runs approximately 10× faster with fewer parameters.
• IN fable-5-first-public-mythos-class
  Fable 5, released June 9, 2026, is the first publicly available 'Mythos-class' model, with additional safety guardrails in high-risk domains.
• IN frontier-agents-validated-in-high-stakes-domains
  Frontier model agents demonstrate production-grade capability in domains where errors carry severe consequences: 16 Opus 4.6 agents writing a C compiler in Rust capable of compiling the Linux kernel, and Mythos Preview identifying 271 security vulnerabilities in Firefox — validating agentic AI for both systems programming (correctness-critical) and security engineering (adversarial-critical) at production scale.
• IN gpt2-withheld-disinformation-concern-2019
  GPT-2 (2019) was withheld from public release by its authors due to concern about its potential misuse for generating disinformation content, representing an early AI-safety governance case
• IN grokking-implies-training-traverses-security-vulnerable-memorization-phase
  Grokking's memorize-then-generalize dynamic — where models first memorize training data before suddenly discovering generalizable patterns — combined with memorization's dual-use nature (knowledge source and extraction attack surface) suggests that model training may traverse a memorization-heavy phase where training data is more extractable before generalization emerges. However, the evidence for this comes from mechanistic interpretability of modular arithmetic models and memorization measurements at the GPT-2 scale, so whether this phase constitutes a universal security-vulnerable window across all training regimes, and whether earlier checkpoints systematically embed more extractable data than later ones, remains an extrapolation beyond what the current antecedents directly establish.
• IN imagenet-decomposition-shows-coarse-but-not-fine-grained-alignment
  On ImageNet at 49 images per class, DINOv2 retrieves correct-class neighbors 46.1% of the time and OpenLlama 58.0%, but strict cross-modal mutual kNN alignment (retrieving the same exact gallery item) remains flat at approximately 11%, indicating models share category-level structure but organize within-class structure differently.
• IN ipc1-alignment-exceeds-both-models-correct-class-rate
  On ImageNet at 1 image per class (ipc=1), strict cross-modal alignment (23.1%) exceeds the rate at which both DINOv2 and OpenLlama retrieve correct-class neighbors (11.7%), meaning the models frequently agree on the same incorrect item (e.g., both matching a bookstore query to a library image).
• IN llama-guard-prompt-guard-safety-models-on-huggingface
  Llama Guard and Prompt Guard are dedicated safety/guardrail models distributed alongside base Llama weights under the meta-llama organization on Hugging Face (huggingface.co/meta-llama).
• IN llm-safety-is-multi-layered-unsettled-challenge
  LLM safety operates across multiple interdependent layers — capability risk classification (Opus 4 at Level 3), architectural vulnerabilities (prompt injection as inherent design flaw), regulatory intervention (Fable 5/Mythos 5 suspension), and behavioral calibration trade-offs (Opus 4.7 over-refusal complaints) — with no single layer providing comprehensive coverage and each layer creating tensions with the others.
• IN llm-security-requires-defense-across-three-independent-surfaces
  LLM security threats operate across three independent attack surfaces requiring distinct defenses: training data poisoning (deliberate grooming of web content), architectural prompt sensitivity (40%+ accuracy shifts from formatting, instruction-input confusion), and inference-time injection — and the architectural vulnerabilities are fundamental, not solvable by engineering or scale.
• IN llm-security-spans-training-and-inference-surfaces
  LLM security threats operate across both training time (deliberate data poisoning via grooming) and inference time (prompt injection), requiring defense at multiple pipeline stages rather than a single chokepoint.
• IN many-to-many-correspondence-reduces-measured-alignment
  Densifying one modality in the CycleReward dataset (11 captions per image for I2T, 12 images per caption for T2I) consistently decreases mutual kNN scores for both k=1 and k=10, demonstrating that relaxing the bijective one-to-one pairing assumption further reduces measured cross-modal alignment even when retrieved neighbors remain semantically valid.
• IN memorization-is-dual-use-capability-and-vulnerability
  Training data memorization exhibits dual-use characteristics: the same retention mechanism that contributes to model knowledge also creates an attack surface for deliberate data poisoning, as memorization rates serve as a quantitative proxy for poisoning vulnerability. GPT-2's early demonstration of both measurable memorization (1-7% exact duplicates) and capability-related safety concerns suggests this tension scales with model capability, though the evidence characterizes the pattern at one scale rather than confirming it as a universal structural property.
• IN model-accessibility-constrained-at-technical-legal-and-safety-levels
  Model accessibility is constrained at three independent, compounding levels — technical (comprehensive optimization stack required for deployment), legal (licensing restrictions and unresolved open-source definitions), and safety (government suspension directives, capability classification) — creating a widening gap between what frontier models can do and who can practically use them.
• IN mozilla-271-vulnerabilities-firefox-mythos
  Mozilla found and patched 271 security vulnerabilities in Firefox using Mythos Preview.
• IN mutual-knn-metric-computation-method
  The mutual kNN alignment metric is computed by L2-normalizing representations of n shared samples, retrieving the k nearest neighbors for each query point (excluding self) via dot-product similarity independently in each modality's space, and scoring as the mean of |N_a^k(i) ∩ N_b^k(i)| / k across all queries; the metric is invariant to orthogonal transformations and isotropic scaling.
• IN mutual-knn-scores-drop-sharply-with-gallery-scale
  Cross-modal mutual kNN alignment scores between DINOv2-base and OpenLlama-3b fall from 0.135 (k=10) and 0.058 (k=1) on WIT-1024 to 0.008 (k=10) and 0.001 (k=1) on LAION-15M, demonstrating a sharp degradation of measured alignment as gallery size increases by ~15,000x.
• IN prompt-injection-is-architectural-vulnerability-not-bug
  Prompt injection is an architectural vulnerability inherent to instruction-following LLMs — arising from the fundamental inability to distinguish developer-defined instructions from user-supplied inputs — rather than a fixable implementation bug, making it the primary and potentially irreducible security concern for deployed applications.
• IN prompt-injection-primary-security-concern
  Prompt injection is the primary security concern for deployed LLM applications
• IN recaptioning-does-not-prevent-cross-modal-alignment-decline
  Recaptioning WIT-1M using gemini-3-flash-preview to produce ~500-word descriptions raises absolute mutual kNN alignment scores but does not prevent the cross-modal alignment decline as gallery scale increases, ruling out caption quality as the primary driver.
• IN sachan-2022-followup-paper-identifiers
  The follow-up paper 'Polysemanticity and Capacity in Neural Networks' by Kshitij Sachan is available as Arxiv 2210.01892 and Alignment Forum post kWp4R9SYgKJFHAufB.
• IN safety-and-licensing-independently-constrain-model-availability
  Model availability is constrained by two orthogonal forces operating simultaneously: safety concerns (Level 3 classification, government suspension directives, refusal calibration failures) and licensing/definitional tensions (non-open-source status, usage restrictions, training data disclosure requirements) — neither alone determines what users can access, and resolving one does not resolve the other.
• IN safety-classification-triggers-immediate-regulatory-action
  The Fable 5 case — released June 9, 2026 as the first public Mythos-class model with additional safety guardrails, then suspended June 12 per a Department of Commerce directive restricting access to foreign nationals — illustrates that capability classification (Mythos-class) can trigger regulatory action on a days-scale timeline, suggesting that safety assessment may function not only as an internal research concern but also as a potential regulatory trigger with rapid production consequences.
• IN safety-deficit-untargetable-due-to-unpredictable-frontier-v2
  The structural safety deficit is not merely widening but faces significant targeting difficulties: because the field's empirical-discovery development pattern (engineering maturity outpacing theoretical understanding) makes systematic capability forecasting inherently challenging, safety investment is structurally harder to direct at specific future threats — the deficit tends to grow in directions that are difficult to anticipate in advance, placing proactive safety assurance under severe structural strain rather than rendering it routine.
• IN safety-feature-cross-modal-activation
  Safety-relevant features such as 'unsafe code' and 'backdoor' activate on both text prompts and image inputs (e.g., hidden cameras, keylogger ads), indicating shared representational structure across modalities.
• IN security-expertise-gap-widens-with-each-capability-advance-v2
  LLM capability advances tend to expand dual-use security surfaces — where mechanisms driving capability improvements can simultaneously deepen vulnerabilities — in ways that may scale superlinearly, while the operational expertise required for reliable deployment can only be acquired through hands-on experience with the deployed system itself, creating a recursive barrier. Together, these dynamics suggest a compounding security and reliability challenge that intensifies across generations, though the superlinear scaling pattern is characterized at limited scales rather than confirmed as a universal structural law.
• IN security-surfaces-expand-with-capability-scaling
  The early evidence of dual-scaling tensions between capabilities and risks (as illustrated by GPT-2's memorization and misuse concerns) compounds the challenge posed by three independent security surfaces — training data poisoning, prompt injection, and architectural vulnerabilities — since architectural vulnerabilities in particular appear fundamental rather than solvable by scale alone, suggesting that LLM security may be a persistently difficult problem rather than one that straightforward engineering progress will resolve.
• IN security-validation-capability-is-inherently-dual-use
  AI systems powerful enough to find 271 real security vulnerabilities defensively (Mozilla/Mythos) while demonstrated to be resistant to safety training constraints (sleeper agents surviving standard training) establish that security validation capability is inherently dual-use — the same code analysis capability that discovers vulnerabilities for patching could discover them for exploitation, and the model performing the analysis cannot be unconditionally trusted.
• IN sigma-threshold-significance-boundary
  At RBF bandwidth σ=0.1, calibrated alignment fails the significance test (p > 0.05) for multiple model families, while at σ≥0.5 significance is recovered but the scaling trend flattens after calibration
• IN sleeper-agents-resistant-to-safety-training
  Anthropic research demonstrated that sleeper agents (models with hidden behaviors triggered by specific conditions) are difficult to detect or remove via standard safety training techniques.
• IN solu-activation-neuron-basis-alignment
  The SoLU activation function (Elhage et al.) was designed to replace ReLU to push features toward neuron-basis alignment, but may make other neurons less interpretable
• IN stronger-llm-alignment-trend-refuted-with-negative-r2
  The previously reported trend that stronger LLMs exhibit higher alignment with DINOv2 vision features does not continue for the most recent model generations, with R² values of −1.41 to −0.58 on broader reasoning benchmarks (ARC, GSM8K, MMLU, LogiQA2), indicating the extrapolated scaling line is worse than a flat prediction.
• IN video-modality-confirms-local-global-pattern
  Video-language alignment on the PVD test set (1024 samples) with VideoMAE (fine-tuned on Kinetics) and frame-level baselines (DINOv2, CLIP) shows the same local-global dissociation: increasing calibrated neighborhood alignment with scale while calibrated spectral scores drop