Pipelines
A tool does one thing. A pipeline runs several of them in the order that actually gets you somewhere — and writes every stage to disk on the way, so the middle is something you can read rather than something you have to trust.
RAG
A document in, a searchable index out. Extract, chunk, embed, index — the same four tools you can run yourself, in the order that gets you there.
- one command thl pipeline rag report.pdf
- every stage on disk read the chunks, not just the answer
- resumable a stop keeps what it made
EDA
A table in, a report you can hand to someone out — with the figures, a replayable recipe, and the Python script that produced all of it.
- one command thl pipeline eda sales.csv
- reproducible the recipe replays every choice
- five primitives each callable on its own
extract → chunk → embed → index
The pipeline owns no logic of its own. It calls the same four tools the Tools page documents, so a fix to the chunker is a fix to both and the two cannot drift apart. What it removes is the typing, not the transparency.
extract
The document becomes markdown, headings intact — because headings are what the next stage splits on.
chunk
Markdown becomes JSONL you can open and read. This is the stage worth looking at when an answer is wrong.
embed
Chunks become vectors, with the model recorded alongside them so a later query cannot use the wrong one.
index
Vectors become a portable store, with a query script written beside it.
Running it
thl pipeline rag report.pdf
thl pipeline rag notes.md --max-tokens 512 --tokenizer estimateNeeds Python 3.10 or newer and pip install "thehallucinatedlab[rag]". On an older Python, pip reports No matching distribution found rather than naming the version — the real reason is one line above it in the output.
Why every stage is a file
The intermediate files are not debris to be cleaned up afterwards. They are the output.
The middle is where it goes wrong
A bad answer out of a RAG system is nearly always a bad chunk. You cannot see a bad chunk in a vector store. You can read one in a JSONL file.
Redoing the expensive step is cheap
Embedding is the slow stage and the one people redo, because the model is the thing they are experimenting with. With the chunks already written, a second model costs one command, not a re-parse of the PDF.
A partial run is still a result
If a stage fails — a missing extra, usually — the pipeline stops, keeps everything the earlier stages produced, and tells you which command to resume from. Nothing is deleted to tidy up after a failure.
What a stopped run looks like
$ thl pipeline rag notes.md
ok extract 0.01s markdown, 4 headings
-> notes.rag/extracted.md
ok chunk 0.02s 9 chunks, estimate (estimated)
-> notes.rag/chunks.jsonl
FAIL embed 0.00s holding vectors needs numpy, which is not installed
thl: stopped at embed. What ran is in notes.rag;
fix the cause and resume with: thl tool embed ...