quantization-static-vs-dynamic
IN premise — entries/2026/06/21/wiki-Large_language_model.md
Created 2026-06-21T09:50:09+00:00
LLM quantization reduces precision post-training to shrink model size; static quantization uses a calibration phase while dynamic quantization adjusts at inference time
Summary
Quantization is the practical way to fit a large language model into less memory by trading a small amount of numerical precision for a big reduction in size. The key fork is whether you decide how to shrink those numbers using a sample of data before deployment (static) or let the model work it out on the fly with each request (dynamic), and that choice determines whether you need representative calibration data up front and how much flexibility you keep during inference.