ginc-transformer-training-config

IN premise — summaries/2026/08/24/xie-2021-icl-bayesian-sR-references-chunk-2.md

Created 2026-08-25T02:58:57+00:00

GINC Transformer models (GPT-2 based) use 4/12/16 layers, 12 attention heads, 768-dim embeddings, dropout 0.1, context window 1024, AdamW optimizer, batch size 8, LR 8e-4, 5 epochs, and 16-bit precision on Titan Xp GPUs.

Summary

This pins down the exact training recipe for the GINC Transformer models so that any result can be reproduced, audited, or traced back to a specific parameter or hardware difference. It also reveals the compute budget was deliberately modest — a single consumer GPU with a small batch size — which sets a hard ceiling on how large the models can be and how much data they can process per training step.