width-confounder-cka-null-baseline-o-d-over-n

IN premise — summaries/2026/08/24/aristotelian-2026-s3-problem-setup.md

Created 2026-08-24T17:10:50+00:00

Under the null hypothesis, E_H₀[‖Ĉ‖²_F] = d_x·d_y/(n−1), giving a leading-order CKA null baseline of O(d/n) that scales with representation dimensionality relative to sample size.

Summary

When two representations are actually unrelated, CKA won't read zero by default; it has a built-in noise floor that grows with the dimensions of the representations and shrinks as you add more data points. This means that high-dimensional models tested on small datasets will show inflated CKA values purely from chance, so any significance test has to account for this dimensionality-to-sample-size ratio rather than comparing against a fixed threshold.