Tokenize
How big is this, and what will it cost to embed? Drop in a document or a folder of chunks and find out — including which pieces are too long to survive the model's context window intact.
This is a measuring instrument, not a pipeline stage
It is tempting to picture the pipeline as extract → chunk → tokenize → embed, with the tokenizer converting text into something the embedding model then consumes. That is not how it works, and building it that way produces garbage.
An embedding model tokenizes internally, with its own vocabulary. BGE-M3 is built on XLM-RoBERTa: SentencePiece, roughly 250,000 pieces. A tiktoken id stream is not a head start for it — it is a different alphabet. There is no file you can hand an embedding model that is "already tokenized".
What a tokenizer is genuinely for is the two questions on this page: how big should a chunk be so it fits the context window, and what will this corpus cost. The chunk tool calls the same counter internally to size its cuts. This page just lets you ask directly.
Count tokens
Drop a document or a JSONL of chunks
A .jsonl file is counted per record, so you can see which chunks overflow
The context window to measure against. 8192 is BGE-M3's. Anything longer is reported as an overflow, because at embed time it would be truncated and the tail would simply not be in the vector.
Counting happens in this tab. The file is never uploaded, and no vocabulary is downloaded.
Arguments
Generated from the tool spec the code itself reads, so it cannot drift out of date.
Which One Is Right Depends On The Question
| Counter | What it is | Download | Use it when |
|---|---|---|---|
| bge | BGE-M3's own SentencePiece vocabulary | ~17MB — the tokenizer, not the 2.3GB model | You are embedding with BGE-M3. This is the only exact answer. |
| openai | tiktoken, cl100k_base | a few MB | You are costing against an OpenAI model. Wrong for this pipeline by construction. |
| estimate | a heuristic, biased high | none | You want a number now. This is what the browser uses. |
Use It From Python
The package can load BGE-M3's real tokenizer, so the counts are exact rather than estimated.
pip install "thehallucinatedlab[chunk]"from thehallucinatedlab import tokenize
report = tokenize("chunks.jsonl", tokenizer="bge")
print(report)
print(report.over_limit, "chunks would be truncated")thl tokenize chunks.jsonl --tokenizer bge
# plain english works too
thl "count the tokens in chunks.jsonl"