Token

In artificial intelligence and natural language processing (NLP), a token is a single unit of text used by language models to process and generate information. Tokens can represent words, subwords, characters, or even punctuation marks, depending on how the model is designed. During training and inference, AI models break text into tokens to understand structure, meaning, and context. For example, a short sentence might be divided into several tokens that the model analyzes sequentially to predict the next word or generate coherent text. The concept of tokens is essential for managing input length, optimizing performance, and calculating costs in large language models.