code-llama-training-token-recipe

IN premise — summaries/2026/08/24/wiki-LLaMA-chunk-1.md

Created 2026-08-24T17:11:14+00:00

Code Llama was fine-tuned on Llama 2 using 500B tokens of general code, then 20B long-context tokens, then 5B instruction-following tokens; the Python variant used 100B Python-only tokens followed by long-context tokens.

Summary

Code Llama was built by layering training on top of Llama 2 in a specific order: first absorbing half a trillion tokens of mixed code, then stretching to handle longer files, and finally learning to follow user instructions, with the Python version getting an extra dedicated pass on Python code. This staged recipe explains where its code-understanding strengths and weaknesses come from and sets the baseline for any claim about its context limits, instruction-following quality, or language-specific performance.