sub-model-search-computationally-infeasible

IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s8-limitations-and-future-opportunities.md

Created 2026-08-25T02:58:34+00:00

Shen et al. (2023) could not exhaustively search sub-models to identify which parameters correspond to cGD's 'updated' weights due to computational infeasibility at LLM scale.

Summary

At the scale of modern large language models, it is computationally impossible to check every possible combination of parameters to figure out which ones actually changed as a result of the cGD training step. This means researchers can trace that weights were "updated" in aggregate, but they cannot point to specific parameters and say "these are the ones cGD touched," which leaves a gap in our ability to verify exactly what the training procedure did inside the model.