Exploratory Data Analysis
Point it at a data file. Get back a report you can read, the figures behind it, a recipe that replays every choice, and the Python script that produced all of it. It runs on your machine and nothing uploads.
One Command, No Flags
This is a command-line tool, not a page widget. Profiling a 200 MB CSV is pandas work — the browser is the wrong place for it, and pretending otherwise would waste your time before failing.
Install and run
The base package is 20 kB and pulls only Pillow. pandas, numpy, matplotlib and scipy live behind an extra, so converting a PNG never costs you a scientific stack.
pip install thehallucinatedlab[eda]thl eda sales.csvsales.eda/
├── report.md the profile, in Markdown so it diffs in a pull request
├── recipe.json every decision, replayable
├── analysis.py regenerates everything above — and is meant to be edited
├── summary.json the same numbers, machine-readable
└── figures/ 01_revenue_histogram.png, ...The Report Is A Starting Point, Not An Artefact
Every run emits an analysis.py that reproduces the report it came from, imports nothing from this package, and is meant to be edited. Nothing else in this space hands you the code.
Run the script it gave you
Same figures, and a summary.json identical to the original. That is not a claim in a README — a test executes the generated file in a clean interpreter and diffs the output, which passes because the script contains the same code that ran, copied in rather than reimplemented.
python sales.eda/analysis.py
# then open it. It is ordinary pandas and matplotlib.
# Delete what you do not need; the report was the starting point.Design once, replay forever
Everything a run decides is written to a recipe: column types, chart selections, the sampling seed, the top-N cut-off. A team standard is a recipe checked into a repository.
thl eda sales.csv -i --save-recipe team.json
thl eda new_month.csv --recipe team.jsonEvery Column Gets A Type And A Confidence
Every failure you actually notice in a profiler is a misclassification: a zip code read as a continuous quantity, a 0/1/2 label encoding given a mean, dates in three formats silently coerced, a 99%-null float described as though it had a distribution.
Nothing is asserted without a score
Anything below 0.70 is flagged in three places — the review screen, the report's caveats, and result.warnings — and every verdict is overridable from the session, from --types, or by editing the recipe. An override is recorded as a decision, so a replay uses it rather than re-inferring and disagreeing with the report it is meant to reproduce.
| Column | Type | Confidence | Null |
|----------|----------------------|------------|-----------|
| revenue | Numeric - continuous | 0.95 | 90 (10%) |
| zip | Numeric - discrete | 0.85 | 0 (0%) |
| us_date | Datetime | 0.60 ! | 0 (0%) |That last row is a date column where day-first and month-first both parse and disagree about the dates. The tool says so instead of quietly picking one.
Sampling is never silent
Under 200 MB everything is exact. Above it the file is streamed: counts, nulls, minimum, maximum and cardinality stay exact, and only the figures come from a seeded sample. When that happens it is stated in the pre-run plan, as a banner atop the report, as a caption on every affected figure, in the warnings, and in the recipe with its seed. A report that does not say it sampled is a report that gets screenshotted into someone's real deliverable with the wrong numbers on it.
Arguments
Five primitives, each independently callable. These tables are generated from the same tool spec the code validates against, so the page cannot document an argument the package rejects.
describe_dataset
Shape, and an inferred type with a confidence for every column.
profile_column
The statistics that fit one column's type. The outlier rule always travels with the count.
plot_column
One figure, written to one file. A chart that does not apply to the column's type is an error that names the types it does.
relate_columns
Correlation, missingness, duplicates, or everything against a target.
eda_report
The whole report, from a recipe. This is the seam a session, a CI job or a future front-end all reach.
From Python, Or A Whole Folder At Once
The command line is an application over five ordinary functions. Reach for whichever fits what you are doing.
From the THL library
Non-interactive by definition — it never prompts, never blocks on input and never opens a window, which is what keeps it working over SSH, in Docker, in CI and inside a notebook.
from thehallucinatedlab import eda
result = eda("sales.csv", target="churn", tier2=True)
print(result.report) # sales.eda/report.md
print(result.script) # sales.eda/analysis.py
print(result.warnings) # sampling, low-confidence typesfrom thehallucinatedlab import describe_dataset, relate_columns
describe_dataset("sales.csv").types()["zip"]
# 'numeric_discrete'
relate_columns("sales.csv", kind="target", target="churn")More of the command line
A folder is treated as N independent datasets, each profiled into its own directory plus an index linking them. One unreadable file does not abort the run — it is recorded, and the exit code says the run was partial.
thl eda sales.csv -i # the nine-screen session
thl eda data/ --pattern "*.csv" --out reports/ # a folder of datasets
thl eda sales.csv --target churn --tier2 all # correlation, missingness, ranking
thl eda sales.csv --format html --self-contained # one file to email
thl eda --list # every chart and statisticThe session walks nine screens — source, load options, type review, columns, charts, statistics, relationships, output, confirm — and every screen prints the flag that would have produced the same choice. Walk it once and the next run is one line with no session at all.
Where this came from
Contributed by @06pratyush. It was built as its own project and folded into the toolkit here; the standalone repository, its tests and the design notes behind the type-inference rules live at thl-exploratory_data_analysis_in_CLI. Issues and pull requests are welcome there.
Not from the Assistant
The Assistant recognises these requests and tells you the command, rather than offering to run them. A browser tab is the wrong place to load a million rows, and finding that out after attaching a file would be worse than being told up front.