A statistical layer for LLM eval runs. It answers one question: which of these differences are real?
Prompt evaluation harnesses tend to end at a table of means. Variant A scored 0.76, variant B scored 0.84, ship B. At the sample sizes anyone actually runs, a dozen items or maybe two dozen, that table cannot support the decision being made on it.
This package reads the output of an existing eval harness and adds the part that licenses a conclusion: interval estimates, paired significance tests, correction for running many tests at once, a power analysis that says how many items the next run needs, and judge validation against human labels.
It also implements the step that should come before all of that. The four dimensions this harness scores were designed top-down; error analysis builds the taxonomy from traces instead. Those commands are in Error analysis.
Built against the summarization eval for We the Compton, a civic intelligence platform. Everything below comes from a real run, not a demo fixture.
pip install -e .
compton-eval analyze path/to/run --out report/The harness compares three prompt variants across twelve council meetings on four dimensions. Read as means, V1 and V2 both look like large wins over V0, and V2 looks like a modest further gain over V1.
Twelve comparisons, Holm-corrected. Four survive.
V0 to V1 is a large improvement and it holds up. Tone gains 2.17 points on a 1–5 scale, top-dollar recall gains 0.72, and both survive correction comfortably. That prompt change worked.
V1 to V2 is a different story. Top-dollar recall is identical on every single meeting, not close but identical. Tone, identical. Anomaly surfacing differs by 0.06 with an interval running from −0.38 to +0.31. If V2 costs more tokens than V1, it is buying nothing I can measure.
One result sits right on the line, and the distinction is the whole reason this package exists. Factual precision from V1 to V2 comes to +0.073, bootstrap interval [+0.023, +0.142], excluding zero. Read alone, that is a win. Corrected for the twelve tests in the family, p climbs to 0.22. Four dimensions times three variant pairs means you should expect roughly one spurious "significant" result per run from chance alone, and this one is not distinguishable from that.
So the chart encodes three states rather than two. "The interval excludes zero" and "this survives multiplicity correction" are different claims, and this run contains rows where they disagree.
Every dot is one meeting. At n=12 the individual observations belong on the chart, because an aggregate mark hides the case where a single meeting is carrying the entire delta.
Simulated from the observed noise distribution, against effects small enough to matter: 0.05 on factual precision, 0.10 on the bounded rates, 0.25 on the tone scale.
| Dimension | Minimum effect | Power at n=12 | Meetings for 80% |
|---|---|---|---|
| anomaly_surface_rate | 0.10 | 55% | 40 |
| tone_score | 0.25 | 38% | 40 |
| factual_precision | 0.05 | 36% | 40 |
A run of twelve meetings has roughly a one-in-three chance of noticing a regression. Nothing in the means table hinted at that.
reliability unavailable: no repeated judgments in this run. Every
(meeting, variant) was scored once. Re-run the tone judge k>=3 times over
the same manifest, tagging each pass, then re-run this check.
Tone is scored by an LLM against fixed calibration anchors. Anchoring pins the scale. It does not establish that the judge returns the same answer twice. Until the judge is scored repeatedly, every tone comparison above rests on an unverified instrument, and no sample size fixes that.
So the module refuses to compute a coefficient from single-pass data. Returning one anyway would be worse than returning nothing.
Self-consistency is only the floor, though. A judge can return the same answer every time and have that answer be wrong, and the coefficients above would call it excellent. The question that decides whether any of this means anything is whether the judge agrees with a human.
Following ch. 3 and ch. 10.
Everything above measures four dimensions that were designed top-down, from what seemed important before anyone read a trace. Grounded theory runs the other way: read traces, write down what actually went wrong, and let the categories fall out of that. The four commands below implement that loop.
compton-eval sample <run-dir> # blended review queue + open-coding worksheet
compton-eval saturation codes.json
compton-eval axial codes.json
compton-eval prevalence labels.json --taxonomy taxonomy.json
Sampling is blended, and the blend matters. Ch. 10 suggests 50% failure-driven, 30% uncertainty, 20% random. The random fifth is the one everyone drops, and it is the only slice that estimates the real failure rate; the other 80% is selected to over-represent failure on purpose.
36 traces available, 24 queued
failure-driven 12 (50%)
uncertainty 7 (29%)
random 5 (21%)
Uncertainty here means the deterministic signals disagree with each other, which flags borderline traces. Only deterministic scores drive the selection. Letting the LLM judge choose what a human reviews would fold the judge's blind spots into the taxonomy meant to expose them.
Open coding is not automated, deliberately. sample emits a worksheet and stops. Ch. 3 recommends an LLM for axial coding and explicitly not for open coding, because that step is your judgment about what counts as a failure. The module enforces the shape rather than the content: an empty note raises, and a note without a binary accept/reject raises with "pick a side." Likert scales are the third pitfall the chapter names, since underspecified rubrics produce lower agreement and higher variance.
Saturation is gated. Ch. 3 says keep reading until at least 20 problematic traces are in hand and new failure modes have stopped appearing. The first half is checkable and the tool checks it:
coded 24/24 problematic: 18
Keep reading. 2 more problematic traces before the taxonomy is worth building.
Axial coding emits the prompt rather than running it. axial fills the chapter's clustering prompt with your notes and prints it. The book is direct that the model's groupings should not be accepted blindly, since it does not know how the system works or which failures need different fixes. Reviewing and editing the taxonomy is the work, not a formality.
Taxonomy gets structurally checked. Fewer than five modes usually means things were merged that have different root causes; more than eight gets hard to apply consistently. Modes without examples drift. And generic categories get called out by name:
! 'hallucination' is a generic category from LLM research, not something
that emerged from your traces. Ch. 3 calls this out as the second most
common mistake in error analysis.
Prevalence reports two numbers, not one.
spokesperson_voice 0.67 [0.46, 0.83] (16/24) random-only: 0.40 [0.00, 0.80] (n=5)
anomaly_not_surfaced 0.42 [0.25, 0.62] (10/24) random-only: 0.00 [0.00, 0.00] (n=5)
The left number is the rate across the review queue, which is biased upward by construction. The right one uses only the randomly sampled traces and is the one that answers "how often does this happen." Reporting the queue rate as the system's failure rate is an easy and expensive mistake, so the tool computes both and labels them.
Round-1 error analysis found a misspelled proper name in 47% of summaries, spread evenly across prompt variants. It is the one failure mode that needs no judge: the correct spellings are in the database.
compton-eval names <run-dir> --extra round1_garbles.json
FAIL 4-v1 alvarez madruga -> Olivarez Madruga; norseman -> Nossaman
FAIL 1885-v1 darton -> Darden x3
FAIL 4219-v0 maxie d filer -> Maxcy D. Filer; matheson -> Matthisen
17/36 fail the check (47%)
Two tiers with different error profiles. Known garbles are an exact lookup against a curated map and cannot produce false positives, since something is only in the map because a human asserted it is wrong. Near misses flag a capitalized token close to a canonical name but not equal to it, which is how the map grows; they are reported for review and never gate the metric.
The near-miss tier needs hard guards or it becomes noise. A token matching any known entity is never flagged, so Sanders Roberts the law firm is not reported as a misspelling of Treasurer Sanders. Distance-2 matches are only trusted after a title word, because without that context they pair Penal (from Penal Code) with Bernal.
Measured against 36 human-labeled traces:
| Vocabulary | TPR | TNR |
|---|---|---|
| As shipped in prod | 0.167 | 1.000 |
| Plus 12 round-1 garbles | 1.000 | 1.000 |
That second row is in-sample and should not be read as accuracy. The garbles were discovered by reading these same traces, so the checker is being tested on its own training data. The honest estimate comes from the next eval run, on names nobody has looked at yet. The first row is the real measurement: the lexicon as it existed caught one garble in six.
Building it surfaced three things worth more than the checker:
Nothing reads the lexicon. asr_corrections (43 rows) and officials.aliases (153 aliases) have no consumers anywhere in the platform codebase. Migration 039 seeded them and nothing has ever queried them.
Phrase entries miss. The map held Attorney Eric Paradin while a summary said Mr. Paradin, so the correction never fired. Entries should be surname-level.
It caught a labeling error of mine. I had applied a garble label to all three variants of one meeting without checking each; the checker reported one as a false negative, and it was right: that variant never mentions the name. The label is corrected, with provenance.
Following Husain and Shankar, Evals for AI Engineers, ch. 5.
Split labeled traces three ways, refine the judge prompt against dev only, and read the test split exactly once after freezing the prompt. Then measure two rates on that frozen split: how many real passes the judge catches, and how many real fails it catches.
compton-eval split labels.json # stratified train/dev/test
compton-eval align labels.json # TPR/TNR, then correct a batch
── Judge accuracy (frozen test set) ───────────────────────────
TPR=0.839 (52/62) TNR=0.914 (53/58) informedness=0.753
! TPR 0.84 below 90% — keep refining the prompt
── True rate over the unlabeled batch ─────────────────────────
raw judge pass rate : 0.6768 (3262/4820)
bias-corrected : 0.7848 [0.6982, 0.9018] (95% CI)
correction moved it : +0.1080
That gap is the point. Counting the judge's own "pass" labels over 4,820 traces gives 0.68, and running it over ten times as many traces would still give 0.68. The bias does not shrink with sample size, so more unlabeled data just buys precision around the wrong number. Correcting with the measured error rates moves the estimate eleven points.
Why TPR and TNR rather than precision and recall. The goal is estimating the true pass rate, and a judge can only get that wrong two ways: missing real passes, or passing real fails. Those two rates name exactly those two errors.
Why binary. TPR and TNR are undefined on a 1–5 scale, so a Likert judge cannot be validated this way at all. That, rather than tidiness, is the argument for one binary evaluator per failure mode.
The correction is Rogan and Gladen (1978):
theta = (p_obs + TNR - 1) / (TPR + TNR - 1)
with a bootstrap over the labeled test set for the interval: resample the (human label, judge prediction) pairs, recompute both rates on each draw, re-apply the correction, take the 2.5th and 97.5th percentiles.
Two deliberate departures from the reference implementation. It rejects the correction when TPR + TNR - 1 <= 0; this uses a small positive floor instead, because a judge with informedness of 0.01 passes a > 0 test and then divides by 0.01, amplifying noise a hundredfold into a confident-looking number built from nothing. And include_batch_uncertainty=True additionally resamples the unlabeled batch, which matters when that batch is small; the book holds p_obs fixed, which is right when it is large. Default behavior matches the book.
Verified against the book's own reference implementation on shared inputs: identical point estimate to machine precision, CI bounds agreeing to four decimals.
These numbers are synthetic. examples/alignment-demo.json exists so the interface is runnable and the output above is real output rather than a mockup. No human labels exist for the actual corpus yet. Producing them is the next piece of work, and nothing here invents them.
Five decisions, each of which changes the answer.
Tests are paired, not independent. Every variant is scored on the same meetings, which makes this a within-subjects design. Analysing it as two independent groups throws away the pairing along with most of the power the design was built to provide.
Wilcoxon signed-rank rather than a paired t-test. Three dimensions are proportions on [0, 1], and the fourth is an ordinal 1–5 judgment with heavy ties and a floor: V0 scores exactly 0.000 on anomaly surfacing for every meeting. Normality is not on offer at n=12.
BCa bootstrap intervals rather than normal-theory ones, because a t-interval on a proportion near its ceiling will hand back an upper bound above 1.0 for a quantity that cannot exceed 1.0. The bias-corrected and accelerated bootstrap handles both the skew and the median bias. Constant vectors return a degenerate interval instead of a NaN from a zero-variance acceleration term.
Cliff's delta rather than Cohen's d. Cohen's d divides a mean difference by a standard deviation, which presumes an interval scale. Cliff's delta asks something ordinal data can actually answer: draw one meeting under each variant, how much more often does one win?
Holm-Bonferroni across the whole family. Four dimensions times three pairs is twelve tests, and Holm controls family-wise error without Bonferroni's conservatism.
Two things are deliberately absent.
There is no post-hoc power calculation. Power computed from an observed effect is a monotone transform of the p-value; it tells you nothing new while sounding like it does. Power here is prospective only: given an effect you would want to detect, how many meetings.
There is no imputation of incomplete pairs. Meetings missing a variant get dropped and the count is reported (anomaly_surface_rate has 8 complete pairs of 12). Filling them in would manufacture precision.
Ground truth lives in predicates, not IDs. Inherited from the upstream harness and repeated here because it is the detail people get wrong: the RAG test set stores relevance as SQL predicates rather than chunk IDs, since chunk IDs change on every re-chunk while "the chunks belonging to this agenda item" survives re-ingestion.
The power simulation pins method="asymptotic". scipy's wilcoxon auto-selects the exact signed-rank distribution at small n and the normal approximation above it. Left on auto, that switch puts a discontinuity in the middle of the power curve, which is exactly the region being read. Pinning one method keeps the curve comparable across sample sizes, at the cost of being mildly conservative below n≈20. It also runs about four orders of magnitude faster, since the exact path does not vectorise over the simulation axis.
The curves go non-monotonic below n≈16, and that is real rather than simulation noise. The signed-rank statistic is discrete, so its attainable p-value grid is coarse at small n, and adding one item can move the critical threshold the wrong way. The chart labels this instead of smoothing it away.
src/compton_eval/
load.py tidy loader; verifies the design is actually paired
bootstrap.py BCa intervals, paired and unpaired
compare.py signed-rank, Cliff's delta, Holm correction
power.py simulation-based prospective power
coding.py blended sampling, open/axial coding, taxonomy, prevalence
entities.py deterministic proper-name check against a DB vocabulary
align.py TPR/TNR against human labels, Rogan-Gladen correction, splits
reliability.py ICC(2,1) and Krippendorff's alpha; self-consistency only
plots.py forest, interval, and power charts
cli.py `analyze` `align` `split` `sample` `saturation` `axial`
`prevalence` `names`
data/ entity vocabulary exported from prod, plus round-1 garbles
tests/ 93 tests, property-based rather than golden-number
examples/ run output, plus synthetic labels for the align demo
Requires Python 3.11+, numpy, pandas, scipy, matplotlib.
pip install -e ".[dev]"
pytestThe loader expects a scores.json shaped like this:
{
"run_id": "2026-04-15T23-05-34-891Z",
"scores": [
{"meeting_id": 1, "variant": "v0", "top_dollar_recall": 0.4, "tone_score": 2}
]
}Rename meeting_id to whatever your items are, then adjust DIMENSIONS in load.py with each metric's scale. Everything downstream is generic. Nothing in it is specific to civic data or to summarization.
To make the reliability check work, score each item k >= 3 times and tag the passes in a pass column.
For judge alignment, supply human labels alongside judge predictions:
{
"failure_mode": "summary_omits_top_fiscal_impact",
"test_set": [{"trace_id": "1885", "human": 1, "judge": 1}],
"unlabeled": {"n": 4820, "judged_pass": 3262}
}Both label fields are binary, 1 for pass. unlabeled is the batch the frozen judge ran over without human review.
Built by O.M. Miles. QA Automation Engineer, M.S. Analytics candidate at Georgia Tech. MIT licensed.