Tokenization

Tokenization converts raw input into smaller units a model can process. For language models, a tokenizer may represent whole words, word fragments, punctuation, individual characters, or bytes as numerical identifiers from a fixed vocabulary. The same sentence can use very different token counts across models, affecting context limits, latency, and price. Tokenization also shapes how a model handles spelling, numbers, code, multilingual text, and rare names. It is not identical to linguistic word splitting, and a token does not necessarily have meaning on its own. Multimodal systems extend the idea by converting regions of images, audio segments, or video frames into model-ready units.