Honesty toolkit

The whole honesty toolkit in a single call. It cross-validates the model against a naive baseline, scans for target leakage, and checks class balance (and fairness, if you pass a sensitive column), then returns a Report whose verdict is SHIP, WARN, or STOP. Read report.verdict for the decision and report.reasons for why. When to use: as the last gate before you deploy or export anything. Pitfall: a SHIP verdict means no blocking issues were found, not that the model is perfect, so still read the warnings. Press Run: the clean model ships.

Give the data a column that secretly contains the answer, and report() refuses to ship it. This is the guardrail that keeps a too-good-to-be-true score from reaching production. Press Run to watch it say STOP.

Scans your raw table for the traps that quietly ruin models: target leakage (a feature that secretly contains the answer), duplicate rows, ID-like columns, constants, and class imbalance. It returns a dict whose 'ok' flag is False when a critical issue is found, and prints a readable summary when show=True. When to use: before you train, and as a CI gate (the CLI exits 1 when it fails). Pitfall: audit reads the data, not the model, so run report() after training as well.

Checks whether rows leaked between your train and test sets, the sneakiest way to fool yourself with a great score. It compares the two tables for exact and near-duplicate rows and returns a dict describing the overlap. When to use: whenever you split data by hand instead of letting BreezeML do it. Pitfall: a model that memorized rows it also gets tested on looks brilliant and is worthless in production.

import breezeml

breezeml.contamination(train_df, test_df)

Measures whether your model treats groups differently. Pass the name of a sensitive column and it returns per-group accuracy, the demographic parity ratio, and a four-fifths-rule verdict, so you learn who the model is worse for. When to use: any model that touches people. Pitfall: overall accuracy can look great while one group is badly served, which is exactly what this surfaces.

import breezeml

breezeml.fairness.report(model, df, sensitive="gender")

Measures how far new data has drifted from the data the model was trained on, using the population stability index (PSI), and flags unseen categories and out-of-range values. It returns a dict of per-feature drift scores with a flag when any crosses the threshold. When to use: on a schedule in production, and it runs live in every deployed API. Pitfall: a model does not tell you when the world changed under it, so silent drift is how good models quietly go stale.

import breezeml

breezeml.drift.check(model, new_df)

Is model A's 2% edge real or luck? McNemar's test and a paired cross-validated t-test compare two models on the same rows and return a p-value instead of a hunch. When to use: before you replace a working model with a slightly better looking one. Pitfall: small score gaps are often noise, and shipping the wrong winner costs you real accuracy.

import breezeml

breezeml.significance.mcnemar(model_a, model_b, df, "target")

Distribution-free prediction sets with a guaranteed coverage level. Instead of one guess you get 'the answer is in {A, B} with 90% coverage', honest uncertainty on every prediction. alpha sets the miss rate you tolerate, so alpha=0.1 targets 90% coverage. When to use: high-stakes calls where you need to know when the model is unsure. Pitfall: coverage is only honest on a held-out calibration set, so never calibrate on the training rows.

import breezeml

sets = breezeml.conformal.conformal_classifier(
    model, calib_df, "target", alpha=0.1
)

For rare-event problems like fraud or disease, where a lazy model can score 99% by always guessing the majority. Summarize the imbalance, move the decision threshold to the metric you care about, calibrate the probabilities, and cost out false positives against false negatives. When to use: any time one class is much rarer than the other. Pitfall: the default 0.5 threshold is almost never right for rare events, so tune it to your real costs.

import breezeml

breezeml.imbalance.summary(df["target"])
breezeml.imbalance.tune_threshold(model, df, "target", metric="f1")