dai2023-experimental-setup-gpt13b-27b-fairseq-v100-sgd-32demos
IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s4-experiments.md
Created 2026-08-24T17:10:53+00:00
The ICL vs. finetuning experiments used GPT 1.3B and GPT 2.7B (fairseq implementations) on NVIDIA V100 32 GB GPUs, with a maximum of 32 demonstration examples, one epoch, and SGD (not Adam) as the finetuning optimizer.
Summary
This records the exact hardware, model sizes, and training settings used to compare in-context learning against finetuning, so anyone evaluating the results knows the comparison was run under a tight budget of at most 32 examples, a single training pass, and a less commonly used optimizer (SGD) rather than the more standard Adam. Those specific choices matter because they constrain how generalizable the findings are and because switching to Adam or more examples could shift the ICL-versus-finetuning balance.