Chunk
Split a document along its own headings, not at every 512th character. Size is a constraint here, not the organising principle — which is why the previous stage went to the trouble of keeping the structure.
Chunk a document
Drop a Markdown document here
The output of Extract, or any .md file
Bigger chunks carry more context but blur what a match actually matched. The ceiling is BGE-M3's context window.
Tokens repeated across a boundary, so a sentence cut in two is still whole somewhere. Must be smaller than max tokens.
Prepend the heading path to each chunk, so a fragment from deep in a document embeds as what it is rather than as a stray paragraph.
Chunking is arithmetic and string handling. It runs here, offline, with nothing uploaded and nothing installed.
The Browser Counts By Estimate
BGE-M3's real vocabulary is a 17MB download — out of proportion to a page that budgets 40KB for its own script. So this page sizes chunks with a heuristic and writes "tokenizer": "estimate" on every record. The Python package loads the real tokenizer and re-checks.
The estimate deliberately runs high
An under-count is the expensive mistake. A chunk that exceeds the model's context is not rejected at embed time — it is silently truncated, and the part past the limit simply never reaches the vector. You would not see an error; you would see slightly worse retrieval, months later, for no visible reason.
Over-counting only costs slightly smaller chunks. So the heuristic adds a margin and rounds up, and errs the one direction that is cheap to be wrong in.
Arguments
Generated from the tool spec the code itself reads, so it cannot drift out of date.
One Record Per Line
JSONL, the same shape a LangChain Document carries: text plus metadata. Greppable, streamable, and diffable — which a folder of numbered text files is not.
{
"text": "Chapter 3 > Methods\n\nWe measured things carefully.",
"source": "report.pdf",
"heading_path": ["Chapter 3", "Methods"],
"page": 12,
"chunk_index": 4,
"token_count": 63,
"tokenizer": "estimate"
}heading_path and page are what turn a retrieved fragment into a citation. Without them a match is an anonymous paragraph, and the only honest thing you can tell a user is "it came from somewhere in this document".
Use It From Python
The package counts with BGE-M3's own tokenizer, so the sizes are exact rather than estimated.
pip install "thehallucinatedlab[chunk]"from thehallucinatedlab import chunk
result = chunk("report.md", max_tokens=512, overlap=64)
print(result.path, len(result.chunks), result.total_tokens)thl extract report.pdf
thl chunk report.md --max-tokens 512 --overlap 64
thl tokenize report.jsonl