von-oswald-2023-training-hyperparameters
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s0-ix-chunk-1.md
Created 2026-08-24T17:11:04+00:00
The von Oswald et al. (ICML 2023) experiments use Adam optimizer (lr=0.001 for K<3, 0.0005 for K≥3), batch size 2048, gradient clipping (global norm=10), Haiku fan-in truncated normal init (std=0.002/K), no regularization, token clipping to [−10,10] for K>2, W₀=0 initialization, and the Optax library.
Summary
This records the exact training recipe behind the von Oswald et al. 2023 experiments, pinning down the optimizer, learning rates, initialization, clipping thresholds, and software library so that the loss-landscape results they report are tied to a specific, reproducible setup. It matters because any attempt to reproduce, extend, or compare against those findings has to start from these exact conditions rather than guessing what was used.