Key Notes
- A published FLORES-200 comparison found that Russian required about 42% more tokens than English with one named tokenizer, rather than across every AI model.
- CLEAR measured a gap reaching 17x between the most and least efficiently encoded languages within a tested tokenizer.
- Higher token consumption can affect API costs and effective context capacity, but it does not establish an equivalent increase in subscription prices or a fixed loss of answer quality.
Writing to an AI in Russian can consume substantially more tokens than expressing the same content in English. In one published comparison, equivalent Russian text required about 42% more tokens. For services billed by usage, that difference can raise the cost before the model has generated a single word of its answer.
The phenomenon is known as the “tokenizer tax.” A September 29 analysis by the Coalition for Language Equity in AI Research, or CLEAR, found a gap reaching 17x between the most and least efficiently encoded languages in one tested tokenizer. Its comparison covered roughly 200 languages. That is a worst-case spread within a particular system, rather than a universal markup on non-English conversations.
What the Russian-Language Test Actually Measured
AI Capital Factory published a separate comparison using 1,012 aligned sentences from the FLORES-200 benchmark and the o200k_base tokenizer. It counted 26,873 tokens for English and 38,188 for Russian, giving a ratio of approximately 1.42. The benchmark uses translations, allowing the comparison to focus on equivalent content rather than unrelated writing samples.
The same publisher reported a much larger Russian-to-English ratio, about 2.46, with the older cl100k_base encoding. Those figures describe named tokenizers and a particular dataset. They should not be treated as a measurement of every current ChatGPT, Claude or Gemini model, or as a prediction for every Russian-language prompt.
Tokenization converts text into units a model can process. Depending on the vocabulary and the text, those units may be whole words, fragments, punctuation or smaller pieces. Two translations can communicate the same idea while breaking into very different numbers of units; counting characters or words alone will not reliably predict the bill.
Why the Biggest Penalties Go Beyond Russian
CLEAR identified Burmese, Tibetan, Khmer and several other less-represented writing systems as particularly disadvantaged. Its headline ratio compares each tokenizer’s highest and lowest language token counts; English is not automatically the cheapest language in every system. The pattern reflects vocabulary design and digital representation, not simply how many people speak a language.
This is an established research problem. A NeurIPS 2023 paper by Aleksandar Petrov and colleagues documented differences reaching 15x for translated text. It connected those differences to access costs, processing time and the amount of content that fits into a model’s context window.
The context effect is easy to illustrate. If one language needs 1.4 tokens for material that another encodes in one, a fixed token budget holds about 71% as much equivalent material, assuming that ratio remains constant. That is an arithmetic example, not a measured result for a particular product. In a document assistant, it could mean less room for source material or conversation history.
More tokens can also mean more processing work, but response time does not necessarily increase by the same percentage. Hardware, batching, network delays, caching and how an application generates its answer all affect what the user experiences.
A Higher Token Count Is Not a Subscription Surcharge
The “tax” is a description of an unequal technical cost, not a government levy or a separate language fee. If equivalent input requires 40% more tokens and every token carries the same rate, that input component costs 40% more. The total task may differ because output length, retries, cached material and other charges have their own effects.
It also does not follow that a monthly subscription becomes 40% more expensive. A flat-rate plan and an application paying for API consumption are different products. Our coverage of OpenAI’s Terra pricing highlighted the broader distinction between a token rate and the cost of an acceptable finished result.
Nor is token efficiency a ranking of intelligence. The multilingual long-context ONERULER benchmark, discussed in our Polish-language coverage, evaluated task performance across 26 languages. A language performing well on retrieval or reasoning tasks does not establish that it is the cheapest to encode.
For a multilingual service, the useful comparison is therefore the cost of completing representative tasks correctly in each supported language. Counting actual inputs and outputs with the intended model, then checking answer quality and completion time, is more informative than assuming an English-language budget transfers unchanged. Translating everything into English may introduce its own costs and errors, especially when local terminology matters.
Disclaimer: AIstify is an independent media brand owned and operated by NuvexMedia LLC, publishing news, research, and insights on artificial intelligence, emerging technologies, automation, and related industries. NuvexMedia LLC invests in and collaborates with companies across the AI, technology, software, and digital innovation sectors. These relationships do not influence AIstify’s editorial coverage, and the publication maintains full editorial independence to provide accurate, timely, and objective information. © 2026 NuvexMedia LLC. All rights reserved. This content is for informational purposes only and should not be considered legal, tax, investment, financial, or other professional advice.
