AI’s ‘Tokenizer Tax’ Can Make Non-English Text More Expensive
Russian text used about 42% more tokens in one test, while a wider study found a 17x gap between languages. Results depend on the tokenizer.
Sequence-to-sequence, or Seq2Seq, describes models that map an input sequence to an output sequence whose length may differ. Machine translation converts words in one language into another, speech recognition maps audio frames to text, and summarization produces a shorter sequence from a longer document. Many Seq2Seq systems use an encoder to represent the input and a decoder to generate output step by step, often with attention between them. Transformer encoder–decoder architectures largely replaced earlier recurrent designs. Training typically uses paired examples and token-level loss, while inference relies on greedy, beam, or sampling-based decoding. Quality depends on alignment, context limits, data coverage, and evaluation beyond surface overlap.
Russian text used about 42% more tokens in one test, while a wider study found a 17x gap between languages. Results depend on the tokenizer.
The Netherlands plans €3.5 billion in annual savings and roughly 25,000 fewer civil-service positions. AI is one lever, with specific targets due in 2027.
Nate Soares says he has not added to retirement savings for a decade over AI extinction fears. His personal forecast contrasts with Musk’s vision of abundance.
Thieves are targeting data-center deliveries through compromised freight accounts and reported attacks on escorts. Industry estimates put losses above $100 million.
Robert Martin’s ub-mode pairs engineering rules with complexity, duplication and mutation-testing tools to help AI coding agents produce maintainable code.
OpenAI and Anthropic claim AI-generated proofs refute the Exponential Time Hypothesis. Lean checks support the claims, while expert scrutiny remains essential.
Arthur Hayes wants FLOP to anchor a global market for AI compute, with futures for hedging and speculation. The network and token remain in development.
Goose pairs Wouter van Oortmerssen’s language design with an implementation largely written by Claude Fable. Carmack praised its memory model; benchmarks need context.
GPT-6.1 Terra targets visual reasoning and computer use at half Sol’s standard API rates, with access through ChatGPT Work, Codex and the API.
Cathedral says AI agents helped researchers access Anthropic’s internal data in under 12 hours. Anthropic says Claude chats, code and model weights were unaffected.