Model Quantization

Model quantization reduces the numerical precision used to store weights or perform calculations, such as converting 32-bit values to 8-bit or lower formats. Smaller representations can reduce memory, speed up inference, lower energy use, and enable models to run on phones, browsers, vehicles, or edge hardware. Quantization may be applied after training or included during training so the model learns to tolerate reduced precision. The efficiency gain can come with accuracy loss, especially for sensitive layers or unusual inputs. Hardware support, calibration data, activation ranges, and task requirements determine the best method. Every quantized version should be evaluated independently rather than assumed equivalent to its original model.