monosemanticity-dataset-sources

IN premise — summaries/2026/08/24/templeton-2024-scaling-monosemanticity-chunk-10.md

Created 2026-08-25T02:58:36+00:00

The scaling monosemanticity paper's dataset uses The Pile (excluding books3) and Common Crawl for text, and hand-curated images from Wikimedia Commons, explicitly excluding Human/Assistant finetuning data.

Summary

The dataset behind this monosemanticity work is built from general web text (The Pile minus books3, plus Common Crawl) and carefully selected Wikimedia images, deliberately leaving out any chat or instruction-tuning data. That means the representational findings describe how a base model organizes knowledge from natural text and images, not how a conversational fine-tune reshapes that structure, so any conclusions about monosemantic features shouldn't be assumed to carry over to deployed chat models.