llama2-7b-architecture-specs

IN premise — summaries/2026/08/24/park-2023-linear-representation-sR-references.md

Created 2026-08-24T17:11:04+00:00

LLaMA-2 7B (meta-llama/Llama-2-7b-hf) uses a 32,000-token SentencePiece vocabulary with 4,096 embedding dimension, pre-trained on 2 trillion tokens (90% English).

Summary

This establishes the practical capability profile of the 7B LLaMA-2 model: it recognizes roughly 32,000 distinct word-chunks, encodes each into a vector of about 4,000 numbers, and was trained overwhelmingly on English text, so it will perform well in English but noticeably weaker in other languages. For the system, this sets the expected ceiling on multilingual quality, tokenization fidelity, and internal representational capacity for any task routed to this model.