Index
Writes a portable database directory that you own and can query with your own code.
Reference
Arguments
All arguments are documented here, in the Assistant, and the Python package.
Why the sidecar matters
Metadata sidecar
A folder of vectors without a record of which model produced them is unopenable. Chroma’s default embedding function is all‑MiniLM‑L6‑v2 (384‑dim). Pointing it at 1024‑dim BGE‑M3 vectors either raises a dimension error or returns nonsense.
- model
- model_id
- dimensions
- normalized
- count
- store
- collection
- created
Store options
Comparing stores
Choose between a built‑in nearest‑neighbour store or a simple portable file.
| Store | What it provides | Portability | Notes |
|---|---|---|---|
| chroma | SQLite‑backed collection with NN search | Medium – needs the Chroma library | Default, convenient for quick queries |
| numpy | .npy vectors + JSONL metadata, no DB | High – plain files readable anywhere | Use when you want to load into another system |
FAISS is deliberately omitted because its wrapper pickles the docstore and requires unsafe deserialization.
Same tool, other doors
Use It From Python
Build an index from vectors and metadata produced by the embed stage.
From the THL library
Install
pip install "thehallucinatedlab[index]"
Create an index
from thehallucinatedlab import index
index("chunks.npy", "chunks.jsonl", "mydb", store="chroma")
Or from the command line
thl index chunks.npy chunks.jsonl mydb --store chroma