text processing

Token

Token

By

A basic text unit — such as a word, symbol, or character – that AI language models use to process and generate text. Tokens define how language models interpret and structure responses.