Raw note · dev only 10 min read

Raw: tokenisation, the questions

A notebook page, transcribed as written, with an overview and a reading list attached.

Pratyush August 10, 2026

This is a raw note. It is a notebook page with an overview bolted on, not a finished argument. Nothing here is a claim the lab is making — the questions are open and some of them are probably malformed, which is the point of writing them down. The page is noindex and only renders on the board in dev mode.

The page as written

Transcribed from the notebook, wording kept, punctuation lightly repaired. Numbering is mine.

  1. What is the best tokenisation method for English, Hindi? Or what is the best language for BPE tokeniser and other tokenisation methods? What's the best pair? And what does "best" even mean?
  2. And what if we translate any language into that best language, would that help?
  3. What information exactly is the tokeniser throwing away?
  4. What if we had per-session tokenisation — like, say I am in a coding session, and the repo name is thehallucinatedlab?
  5. What if the vocabulary size is big? What issues does it bring?
  6. Somehow the copying problem in tokenisation feels analogical to how even the same object in C has a different address — like a = b, but both of them having different addresses.
  7. Can we do gradient descent in tokenisation? If yes, how? If no, why?

Overview

Six of the seven questions turn out to have a literature; the seventh is the interesting one.

On "best" (1). The word is doing too much work, and separating its senses is most of the progress available here. There are at least three definitions in play. Compression: bytes per token, or its inverse, fertility — how many tokens a word costs. Distributional: how evenly the vocabulary is used, which Zouhar et al. formalised as Rényi efficiency and showed correlates with downstream BLEU far better than raw compression does. Downstream: task score after pre-training, which is the only definition anyone actually cares about and the only one you cannot measure cheaply. These three disagree. A tokeniser that compresses best is not the one that scores best, because compression buys you sequence length while an unbalanced vocabulary spends capacity on tokens the model barely sees.

English versus Hindi is not a symmetric question. Devanagari is an abugida with heavy inflection and a script that most multilingual vocabularies under-allocate, so Hindi text typically costs several times more tokens per unit of meaning than English does. Petrov et al. and Ahia et al. both measured this and gave it a name — the token premium — and its consequences are concrete, not aesthetic: you pay more per API call, you exhaust the context window sooner, and you get less effective compute per sentence. Rust et al. is the direct experiment for the "which tokeniser for language X" half: a dedicated monolingual tokeniser recovers most of the gap between a multilingual model and a monolingual one, which says the tokeniser, not the model, was carrying the deficit.

On translating into the "best" language (2). Tempting and mostly a trap, but not entirely. It is empirically true that many multilingual models reason better after an implicit pivot through English, and that translate-then-solve beats native prompting on several benchmarks. The costs are that translation is lossy in exactly the places that matter — named entities, code-switching, honorifics, morphology with no English equivalent — and that you have now put a second model's errors upstream of your first. Worth reading the "language models are multilingual chain-of-thought reasoners" line of work before deciding, because the effect is real and the mechanism is not what it looks like.

On what is thrown away (3). This is the sharpest question on the page. Concretely: case and whitespace structure (recoverable only if the vocabulary happens to encode them), morpheme boundaries (BPE splits by frequency, not by morphology, so unhappiness may not split at un|happy|ness), character identity inside a token (the model cannot see the letters in a token it has memorised as an atom, which is the whole explanation for the strawberry-r counting failure), and digit structure unless the tokeniser was built to keep digits separate. Singh and Strouse showed arithmetic performance moving substantially with digit tokenisation alone. Nothing about the model changed; the input representation did.

On per-session vocabularies (4). This is a real research direction and the correct name for it is tokeniser transfer or vocabulary adaptation. Adding thehallucinatedlab as a token is cheap; giving it an embedding the model understands is not, because the embedding matrix and the output head were trained against a fixed vocabulary. Minixhofer et al.'s zero-shot tokeniser transfer trains a hypernetwork to produce embeddings for a vocabulary the model has never seen, which is the closest thing to a general answer. The practical version already exists in a smaller form: Dagan et al. showed retraining the tokeniser for a code domain is worth real points. The open part of your question is whether it can be done per session, in seconds, without a fine-tune — and there the honest answer is that nobody has shown it.

On vocabulary size (5). The trade is: larger vocabulary means shorter sequences (cheaper attention, more text per context) but a larger embedding matrix and softmax, more parameters spent on tokens seen rarely, and a longer tail of under-trained tokens. That tail is not theoretical — it is where glitch tokens live, the ones that make a model behave strangely because they appeared in the tokeniser's training corpus and essentially never in the model's. Tao et al. argue vocabulary should scale with model size and that most models are under-vocabularised; Gowda and May found the optimum for NMT is where the rarest tokens are still seen often enough to learn. Both are the same insight from opposite ends.

On the C-pointer analogy (6). The analogy is close enough to be worth keeping, with one correction. Your intuition is that two things that are semantically the same object end up with different identities. In tokenisation the mechanism is segmentation ambiguity: " the", "the", "The" and "THE" are four unrelated integers with four unrelated embeddings, and the model has to learn from scratch that they mean the same thing. Where the analogy breaks is that in C, a and b having different addresses is a fact about storage that the language guarantees you can see through by dereferencing. There is no dereference here. The model has no operation that recovers "these two ids are the same string modulo case" — it can only learn the equivalence statistically, from data, at a cost. That is why copying is hard: a copy is exact at the character level and approximate at the token level, and the model is only ever shown the token level.

On gradient descent (7). Not on BPE as it stands: merge operations are discrete, the merge table is built by a greedy counting procedure with no loss to differentiate, and segmentation is an argmax. So "no" for the algorithm as written. But "yes" for the goal, along three routes that already exist. Relax the segmentation and learn it jointly — Charformer's gradient-based subword tokenisation scores candidate subword blocks and takes a soft combination, so the block scorer trains with the model; MANTa does the same with a learned segmenter. Keep the discreteness but make it stochastic — BPE-dropout randomises merges during training so the model sees many segmentations of the same string, which is regularisation rather than optimisation, but it attacks the same brittleness. Or delete the tokeniser — ByT5, MegaByte, MambaByte and the Byte Latent Transformer all operate on bytes and learn the grouping internally, with BLT's dynamic patching being the closest thing to a tokeniser that is trained end-to-end. The reason none of these has displaced BPE is cost, not principle: bytes make sequences four to five times longer, and the compute saved by a good tokeniser is enormous.

What I would do next. The cheapest experiment that would teach you the most: take one paragraph of Hindi and its English translation, run both through three tokenisers (GPT-4o's, Llama 3's, a SentencePiece unigram you train yourself on Hindi), and measure tokens per character, fertility per word, and how many tokens survive round-tripping. It is an afternoon, it needs no GPU, and it turns question 1 from a definition argument into a number. The lab's own tokenise tool already does most of this in the browser.

Reading list

Ordered by how directly each one answers a question above. Every arXiv identifier below was checked against arXiv on 10 August 2026 and resolves to the paper named. The entries with no identifier — Schuster and Nakajima, and the Karpathy material — were not machine-checked, and no identifier here is a substitute for opening the paper.

What "best" means, and how to measure it

  • Zouhar, Meister, Gastaldi et al. (2023), "Tokenization and the Noiseless Channel", ACL. Introduces Rényi efficiency as a tokeniser quality metric and shows it predicts downstream BLEU better than compression does. The direct answer to "what does best even mean". arXiv:2306.16842
  • Rust, Pfeiffer, Vulić, Ruder, Gurevych (2021), "How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models", ACL. Isolates how much of the multilingual penalty is the tokeniser rather than the model. arXiv:2012.15613
  • Gowda & May (2020), "Finding the Optimal Vocabulary Size for Neural Machine Translation", Findings of EMNLP. Frames vocabulary size as a frequency-of-observation problem. arXiv:2004.02334

The cost of not being English

  • Petrov, La Malfa, Torr, Bibi (2023), "Language Model Tokenizers Introduce Unfairness Between Languages", NeurIPS. Measures the token premium across many languages. arXiv:2305.15425
  • Ahia, Kumar, Gonen et al. (2023), "Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models", EMNLP. The same effect priced in API terms. arXiv:2305.13707
  • Limisiewicz, Balhar, Mareček (2023), "Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages", Findings of ACL. Vocabulary allocation as the mechanism. arXiv:2305.17179
  • Shi, Suzgun, Freitag et al. (2022), "Language Models are Multilingual Chain-of-Thought Reasoners". The evidence behind question 2 — pivoting through English helps, and the paper is careful about when. arXiv:2210.03057

The foundations, if you want the algorithms exactly

  • Sennrich, Haddow, Birch (2016), "Neural Machine Translation of Rare Words with Subword Units", ACL. BPE, as introduced. arXiv:1508.07909
  • Kudo (2018), "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates", ACL. The unigram LM tokeniser, and the first serious argument that segmentation should be non-deterministic. arXiv:1804.10959
  • Kudo & Richardson (2018), "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing", EMNLP demo. Why treating the input as raw bytes with no pre-tokenisation matters for Indic scripts. arXiv:1808.06226
  • Schuster & Nakajima (2012), "Japanese and Korean Voice Search", ICASSP. WordPiece, and the oldest statement of the problem.
  • Karpathy (2024), "Let's build the GPT Tokenizer" and the minbpe repository. The fastest way to hold the algorithm in your head; the video spends real time on exactly the failure modes in question 3.

What the tokeniser throws away

  • Singh & Strouse (2024), "Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs". Digit segmentation alone moves arithmetic accuracy. arXiv:2402.14903
  • Land & Bartolo (2024), "Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models". The under-trained tail of a large vocabulary, found systematically. Read alongside the original SolidGoldMagikarp write-up. arXiv:2405.05417
  • Chen, Tworek, Jun et al. (2021), "Evaluating Large Language Models Trained on Code" (Codex). Contains the whitespace-token change made specifically because GPT-2's BPE handled Python indentation badly — evidence for question 3, and the whole answer to a question on the other raw note. arXiv:2107.03374

Learned, adaptive and absent tokenisers

  • Tay, Tran, Ruder et al. (2022), "Charformer: Fast Character Transformers via Gradient-based Subword Tokenization", ICLR. The direct "yes, here is how" for question 7. arXiv:2106.12672
  • Godey, Castagné, de la Clergerie, Sagot (2022), "MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling", Findings of EMNLP. A differentiable segmenter trained jointly with the model. arXiv:2212.07284
  • Provilkov, Emelianenko, Voita (2020), "BPE-Dropout: Simple and Effective Subword Regularization", ACL. Keeps BPE discrete but stops the model relying on one segmentation. arXiv:1910.13267
  • He, Haffari, Norouzi (2020), "Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation", ACL. Segmentation chosen by the downstream objective rather than by frequency. arXiv:2005.06606
  • Minixhofer, Ponti, Vulić (2024), "Zero-Shot Tokenizer Transfer". The nearest thing to per-session vocabulary from question 4. arXiv:2405.07883
  • Dagan, Synnaeve, Rozière (2024), "Getting the most out of your tokenizer for pre-training and domain adaptation". Retraining the tokeniser for code, measured. Relevant to the coding-session half of question 4. arXiv:2402.01035

Doing without one

  • Xue, Barua, Constant et al. (2022), "ByT5: Towards a token-free future with pre-trained byte-to-byte models", TACL. arXiv:2105.13626
  • Clark, Garrette, Turc, Wieting (2022), "CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation", TACL. arXiv:2103.06874
  • Yu, Simig, Flaherty et al. (2023), "MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers", NeurIPS. arXiv:2305.07185
  • Pagnoni, Pasunuru, Rodriguez et al. (2024), "Byte Latent Transformer: Patches Scale Better Than Tokens". Entropy-driven dynamic patching — the current strongest argument that the tokeniser can be learned rather than fitted. arXiv:2412.09871