JupiteX Get the app
Science & Technology16 Nov 2026 · about 7 min

‘Tokeniser tax’: Why AI is more expensive for non-English users

The brief

The “tokeniser tax” is the extra burden that arises when an AI model needs more tokens to process one language than another. Tokens are the units of text or data that models process. If a task uses more tokens, it can cost more, even when the user is doing comparable work. The study calls this gap a hidden tax because users may face it without seeing a separate charge named for their language. The gap appeared even among the 20 most popular internet languages: the most expensive used between 1.5 and 5 times as many tokens as the cheapest. Across 200 languages, the token cost gap in one model grew to 17 times. In practice, this could mean a startup in Bengaluru pays more per prompt and completion than a competitor in Austin. More tokens also bring other penalties. The study says they can increase response latency and reduce the context available for documents or conversation history. Researchers say that can affect both service quality and who can participate in the AI economy, especially where computing and connectivity are limited.

01

What is the “tokeniser tax,” and how can it make AI more expensive for some language users?

The “tokeniser tax” is the extra burden that arises when an AI model needs more tokens to process one language than another. Tokens are the units of text or data that models process. If a task uses more tokens, it can cost more, even when the user is doing comparable work. The study calls this gap a hidden tax because users may face it without seeing a separate charge named for their language.

The gap appeared even among the 20 most popular internet languages: the most expensive used between 1.5 and 5 times as many tokens as the cheapest. Across 200 languages, the token cost gap in one model grew to 17 times. In practice, this could mean a startup in Bengaluru pays more per prompt and completion than a competitor in Austin.

More tokens also bring other penalties. The study says they can increase response latency and reduce the context available for documents or conversation history. Researchers say that can affect both service quality and who can participate in the AI economy, especially where computing and connectivity are limited.

02

How large can the gap in token use be between languages?

The size of the gap depends on which languages and model are being compared. In the study, the most expensive languages among the 20 most popular languages on the internet used between 1.5 and 5 times more tokens than the cheapest. That means the imbalance was present even among widely used online languages, not only the least represented ones.

When researchers expanded their comparison to 200 languages, the token cost gap in one AI model grew by 17 times. The article does not say that every language pair, or every model, has exactly this difference. It reports the largest gap found in that expanded analysis. The script used by a language also mattered, so speaker numbers alone did not predict token use.

These figures matter because token counts are tied to the amount of processing a model must do. Higher counts can mean higher costs and slower responses, while leaving less room for conversation history. The study argues that such disparities can limit participation in AI, particularly where computing and connectivity are already constrained.

03

What happens to AI costs, response speed, and the amount of conversation history the model can use when a language needs more tokens?

When a language takes more tokens to represent the same task, the model has more units to process. According to the study, higher token counts translate into higher costs and greater latency, meaning responses can take longer. The article also says more tokens reduce effective context capacity: less of the model’s available space remains for documents or earlier conversation.

That lost space can matter in a practical exchange. Pratyush Choudhury gives the example of a tutor that needs to retain a student’s earlier mistakes and explanations. If that history uses up the context window faster in one language, the model has less room to use it while responding. He says this can reach into the quality of the service.

The study says these penalties are especially serious where computing power and connectivity are already limited. Token prices are falling quickly, which Reddy says should make the direct cost impact less severe over time. But cheaper tokens alone do not fix the tokenisation gap; labs would still need to design tokenisers to handle more languages and scripts well.

04

Why can a language’s writing script matter as much as how many people speak it?

A language’s number of speakers is not the only thing that influences how many tokens its text uses. The study found that the writing script plays a major role. A tokeniser handles text using the patterns and units it can represent, and scripts that are less familiar to it can be split into more tokens. That can make a language costly to process even if speaker numbers are not the main obstacle.

Asturian is one example. It has relatively few speakers and no Common Crawl data, but the study says it “benefits from the Latin-script advantage,” as well as representation from related Romance languages such as Italian. By contrast, languages written in scripts a tokeniser rarely encountered were consistently among the most disadvantaged.

This finding means that improving representation is not just about collecting more text for a language. Tokenisers also need to handle its script effectively. The researchers say that disparities can affect cost, response time, and context capacity, so script-aware design matters to making AI more usable across languages.

05

Which languages and scripts in the study faced the greatest tokenisation disadvantages, and what did they have in common?

The study found the greatest tokenisation disadvantages among languages written in scripts the tokenisers rarely saw. It names Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki as being at the greatest disadvantage consistently across nearly all models in the analysis. The shared issue was script familiarity, rather than simply how many people speak each language.

The article contrasts these cases with Asturian. Although Asturian has relatively few speakers and no Common Crawl data, its Latin script and representation through related Romance languages such as Italian were an advantage. This contrast shows why speaker numbers or a language’s rarity alone cannot explain token use: a tokeniser’s exposure to a writing system also matters.

Many current tokenisers are disproportionately optimised for English and other high-resource languages. The study’s findings suggest that scripts with less representation can be processed less efficiently across models. The article says better data can help, but closing the gap will also require labs to deliberately design future tokenisers to handle these languages and scripts better.

06

How could better language data and deliberately redesigned tokenisers help close the gap?

Better language data gives AI systems more examples of how people actually speak and write. Raoul Nanavati points to RESPIN, a collaborative initiative with the Indian Institute of Science, funded by the Gates Foundation. It collected text from native speakers of 38 dialects along with read speech. Such efforts can address the lack of high-quality digital data that researchers identify as an underlying problem.

However, more data alone does not guarantee efficient tokenisation. The study found that scripts rarely seen by tokenisers were often disadvantaged across models. Dheemanth Reddy warns that better training data does not itself fix the tokeniser. The mechanism needs attention too: labs must deliberately design future tokenisers to handle underserved languages and scripts better.

Reddy says token prices are falling quickly, and better multilingual and synthetic data should improve model quality. Falling prices may soften direct costs, but they do not remove the underlying token-use gap. Choudhury also calls for language efficiency to be treated as a product requirement and for major labs to publish language-by-language scorecards and measurable improvement plans.

07

Why do AI models break text into tokens, and how does a tokeniser decide where one token ends and another begins?

The article defines tokens as units of text or data that AI models process. Breaking input into these units lets a model work with a sequence of representable pieces rather than treating a whole document as one item. The article says tokenisers can break down raw text, images, or audio, but it does not explain their detailed internal rules.

In general, text tokenisers use a vocabulary of units and rules or learned patterns to split text into pieces. A piece may be a whole word, part of a word, punctuation, or another character sequence. The tokenizer maps the pieces to identifiers the model can process. Boundaries therefore do not always fall neatly between words, and different tokenisers can split the same text differently. This is general background, not a mechanism described in the article.

That difference is central to the reported gap: a language can take more tokens if a tokeniser represents its text less efficiently. The article says script familiarity and high-quality digital data matter, and calls for labs to design tokenisers that handle more languages and writing systems better.

This brief was written by AI from the original reporting and checked by other models. Names, figures and quotes come from the source; read it for full context.

Read more in the JupiteX app

Pulse is free. New stories every 4 hours, each one broken into the questions that explain it.

Or read more news on the web