Embed
Turns chunks into vectors locally, preparing them for similarity search.
Reference
Arguments
All arguments are documented here, in the Assistant, and the Python package. The table below is generated from the tool spec.
Model comparison
Embedding models
Two models are supported. An index built with one cannot be queried with the other.
| Model | Model ID | Dimensions | Context | Size | Notes |
|---|---|---|---|---|---|
| bge | BAAI/bge-m3 | 1024 | 8192 | ~2.3GB | Multilingual, no instruction prefix needed |
| minilm | sentence-transformers/all-MiniLM-L6-v2 | 384 | 512 | ~90MB | English‑centric, truncates >512 tokens |
Technical note
L2 normalisation
Vectors are L2‑normalised by default, which makes cosine similarity identical to dot product. BGE‑M3 does not require an instruction prefix, unlike the older bge‑large‑en model.
Same tool, other doors
Use It From Python
The Python tier reads the JSONL produced by the chunk stage and writes a .npy of vectors.
From the THL library
Install
pip install "thehallucinatedlab[embed]"
Embed a chunk file
from thehallucinatedlab import embed
embed("chunks.jsonl", model="bge")
Or from the command line
thl embed chunks.jsonl