Count the grades your rubric never hands out
Shipped
0.43.0 added two things to the workout report card: a calibration script that regrades a trailing window of real sessions and fails when a grade distribution goes degenerate, and a set of verdict evals that assert the letter a run deserves rather than the float the rubric computed.
Both exist because of a defect that the module’s own test suite could not have caught. The report card had more than 140 unit tests. Every one of them asserted that the rubric computes what it says it computes: a deviation float, a band boundary, a display string. Not one could fail when the rubric’s answer was wrong. So every grading defect in that module’s history was found the same way, by a person reading a rendered card and saying “that grade is wrong.”
The technique below is the general version. If you own anything that turns a measurement into a category, a letter, a risk tier, a health status, a lead score, you can build both halves in an afternoon, and the first run usually tells you something.
Why unit tests structurally miss this
A band table is a function from a number to a label. Its unit tests pin the boundaries: 0.05 is an A, 0.051 is an A-. Those tests keep passing no matter how badly the boundary is placed, because they were written from the same understanding that placed it.
The measurable signature of a bad table is that it stops discriminating. In the case that prompted this work, a heart-rate ceiling was graded on the fraction of the run spent above it, and that fraction was fed into a table calibrated for relative magnitudes. Over 90 days the metric emitted 32 F, 7 A and 4 D, with B and C never used at all. A five-band scale with two live bands is a threshold wearing a rubric’s clothes, and it was visible in one query against data that had been sitting on disk the whole time.
Psychometrics has named this for a long time. In classical test theory, an item that everyone answers the same way is uninformative, because it does not differentiate between the people taking the test; variability in the responses is precisely what makes an item worth scoring. The same is true one level up, of the band table itself.
It is also the shape Google’s data-validation group described in Data Validation for Machine Learning: a defect confined to one slice can leave aggregate metrics looking fine, so the check has to be on the distribution, not on the average or on any single example.
Prerequisites: separate the grader from the report
You need one entry point that turns a record into a grade, and everything else must call it. Save this as rubric.py.
"""The grader under test: one deviation in, one letter out."""
# Each row is (max relative deviation, letter). First row that fits wins.
GRADE_BANDS = (
(0.02, "A+"), (0.05, "A"), (0.08, "A-"),
(0.12, "B+"), (0.16, "B"), (0.20, "B-"),
(0.26, "C+"), (0.32, "C"), (0.38, "C-"),
(0.40, "D"),
)
GRADE_POINTS = {"A": 4.0, "B": 3.0, "C": 2.0, "D": 1.0, "F": 0.0}
def base_letter(grade):
"""'B-' -> 'B'. The modifier is presentation; the band is the signal."""
return grade[0] if grade else None
def grade_from_deviation(deviation):
"""Relative deviation (>= 0) -> letter. Anything past the last band is F."""
for edge, letter in GRADE_BANDS:
if deviation <= edge:
return letter
return "F"
def distance_deviation(actual_m, target_m):
"""Two-sided: a target distance is a point, missable either way."""
return abs(actual_m - target_m) / target_m
def pace_deviation(actual_s_per_km, target_s_per_km, intent):
"""One-sided on easy days. An easy run is *supposed* to be slow, so only
running too FAST is a miss; on the free side the deviation is exactly 0."""
over = target_s_per_km - actual_s_per_km # positive == too fast
if intent == "easy" and over <= 0:
return 0.0
return abs(over) / target_s_per_km
def hr_deviation(avg_hr, cap_bpm):
"""Grade a prescribed ceiling in the unit it was prescribed in: bpm over.
NOISE absorbs sensor jitter; SCALE sets how many bpm over is an F."""
NOISE, SCALE = 1.5, 28.0
return max(0.0, (avg_hr - cap_bpm) - NOISE) / SCALE
def grade_record(rec):
"""The production path. The calibration gate must call THIS, not a copy."""
metrics = {
"distance": grade_from_deviation(
distance_deviation(rec["distance_m"], rec["target_distance_m"])),
"pace": grade_from_deviation(
pace_deviation(rec["pace_s_per_km"], rec["target_pace_s_per_km"],
rec["intent"])),
"hr": grade_from_deviation(hr_deviation(rec["avg_hr"], rec["cap_bpm"])),
}
points = [GRADE_POINTS[base_letter(g)] for g in metrics.values()]
gpa = sum(points) / len(points)
overall = min(GRADE_POINTS, key=lambda k: abs(GRADE_POINTS[k] - gpa))
# An F on any single metric caps the overall at C: a composite must not
# average away a hard failure.
capped_by = next((k for k, g in metrics.items() if base_letter(g) == "F"), None)
if capped_by and GRADE_POINTS[overall] > 2.0:
overall = "C"
return {"metrics": metrics, "overall": overall, "capped_by": capped_by}
You also need history. In a real system this is your production database; for a runnable example, seed.py fabricates a season with a realistic spread of compliance.
"""Fabricate a season of history so the gate has something to grade."""
import sqlite3
from pathlib import Path
SCHEMA = """
CREATE TABLE IF NOT EXISTS sessions (
id INTEGER PRIMARY KEY, day TEXT NOT NULL, intent TEXT NOT NULL,
distance_m REAL, target_distance_m REAL,
pace_s_per_km REAL, target_pace_s_per_km REAL,
avg_hr REAL, cap_bpm REAL
);
"""
# (intent, distance multiplier, seconds faster than target, bpm over cap)
PATTERN = [
("easy", 1.00, 12, 0.5), ("easy", 1.01, 6, -3.0), ("quality", 0.98, -4, 6.0),
("easy", 1.02, 20, 2.0), ("long", 0.95, 30, -1.0), ("easy", 1.00, -2, 12.0),
("quality", 1.09, 2, 4.0), ("easy", 0.99, 40, 0.0), ("long", 1.15, 8, 9.0),
("easy", 1.00, 1, 18.0), ("quality", 0.70, -110, 1.0), ("easy", 1.45, 160, 24.0),
("quality", 1.62, -220, 3.0),
]
def build(path):
Path(path).parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(path)
conn.executescript(SCHEMA)
conn.execute("DELETE FROM sessions")
for i in range(48):
intent, dist_mult, pace_delta, hr_over = PATTERN[i % len(PATTERN)]
target_d, target_p, cap = 8000.0, 360.0, 140.0
conn.execute(
"INSERT INTO sessions (id, day, intent, distance_m, target_distance_m,"
" pace_s_per_km, target_pace_s_per_km, avg_hr, cap_bpm)"
" VALUES (?,?,?,?,?,?,?,?,?)",
(i, f"2026-{1 + i // 28:02d}-{1 + i % 28:02d}", intent,
target_d * dist_mult, target_d,
target_p - pace_delta, target_p, cap + hr_over, cap))
conn.commit()
conn.close()
return path
if __name__ == "__main__":
print(f"seeded {build('data/history.db')}")
Build the gate: two signatures, and only two
The gate regrades the window and asks two questions of each metric’s letter counts.
Punitive skew. More than 60% of records land in D or F. A rubric measures compliance with an instruction somebody is trying to follow, so heavy compliance is the expected state. Heavy failure means either the athlete is missing constantly, which something else is already shouting about, or the yardstick is wrong.
Dead bands. Two or more letters are never used. The table cannot reach those grades on this data, so they are decoration.
Note what is deliberately not checked: concentration in a passing grade. That asymmetry was the correction to this check’s own first draft, which failed any letter above a flat 60% share and promptly flagged healthy metrics. Distance grades 79% A on the live data because the distances are being hit; that is the system working. The SRE book’s rule for alerting applies directly here, that every page should be actionable, because when alerts fire too often people skim or ignore them. A gate that cries about a metric doing its job gets muted, and then it protects nothing.
Save this as calibrate.py.
#!/usr/bin/env python3
"""Regrade real history through the production grader and gate on the shape
of the letter distribution. Exit 0 clean, 1 degenerate, 2 could not run."""
import argparse
import sqlite3
from collections import Counter
from rubric import base_letter, grade_record
LETTERS = ("A", "B", "C", "D", "F")
FAILING = ("D", "F")
MIN_SAMPLE = 10
DEFAULT_MAX_FAIL_SHARE = 0.60 # above this share of D/F, suspect the yardstick
DEFAULT_MAX_EMPTY = 2 # this many unused letters means the scale collapsed
# Naming the knobs per metric turns a failure into a next step instead of a grep.
GOVERNING_CONSTANTS = {
"distance": ("GRADE_BANDS", "distance_deviation"),
"pace": ("GRADE_BANDS", "pace_deviation"),
"hr": ("GRADE_BANDS", "NOISE", "SCALE"),
}
def open_readonly(path):
"""mode=ro is enforced by SQLite, not by this script remembering to SELECT."""
conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
conn.row_factory = sqlite3.Row
return conn
def grade_window(conn):
"""Grade every historical record through the SAME entry point production
uses. A reimplementation here would be free to disagree with the grader,
and you would not know which of the two was wrong."""
rows = conn.execute("SELECT * FROM sessions ORDER BY day")
return [grade_record(dict(r)) for r in rows]
def collect(cards):
tally = {name: Counter() for name in GOVERNING_CONSTANTS}
for card in cards:
for name, grade in card["metrics"].items():
tally[name][base_letter(grade)] += 1
return tally
def verdict(counts, *, max_fail_share, max_empty):
"""(status, reason) for one metric's distribution.
Concentration in a PASSING letter is never a failure. A rubric measures
compliance with an instruction somebody is trying to follow, so heavy
compliance is the expected state; heavy failure is the surprise.
"""
n = sum(counts.values())
if n < MIN_SAMPLE:
return "skip", f"only {n} graded; need {MIN_SAMPLE}"
fail_share = sum(counts[x] for x in FAILING) / n
empty = [x for x in LETTERS if not counts[x]]
if fail_share > max_fail_share:
return "FAIL", (f"{fail_share:.0%} graded D/F (max {max_fail_share:.0%})"
" - punitive skew")
if len(empty) >= max_empty:
return "FAIL", f"dead bands - {', '.join(empty)} never used"
top, count = counts.most_common(1)[0]
return "ok", (f"{len(LETTERS) - len(empty)}/5 bands used, "
f"{top} {count / n:.0%}, D/F {fail_share:.0%}")
def report(tally, cards, *, max_fail_share, max_empty):
header = f"{'metric':<12}{'A':>4}{'B':>4}{'C':>4}{'D':>4}{'F':>4}{'n':>6} verdict"
out = [f"Calibration - {len(cards)} graded records", "", header, "-" * len(header)]
failed = False
for name in GOVERNING_CONSTANTS:
counts = tally[name]
status, reason = verdict(counts, max_fail_share=max_fail_share,
max_empty=max_empty)
failed = failed or status == "FAIL"
cells = "".join(f"{counts[x]:>4}" for x in LETTERS)
out.append(f"{name:<12}{cells}{sum(counts.values()):>6} {status} - {reason}")
if status == "FAIL":
out.append(f"{'':<12}governed by: {', '.join(GOVERNING_CONSTANTS[name])}")
# Informational, never gated: `overall` is derived from the rows above, so
# failing it too would report one defect twice.
overall = Counter(c["overall"] for c in cards)
capped = sum(1 for c in cards if c["capped_by"])
out += ["", "overall (informational, not gated)",
" letters: " + " ".join(f"{x}={overall.get(x, 0)}" for x in LETTERS),
f" F-cap fired on {capped}/{len(cards)} ({capped / len(cards):.0%})"]
return "\n".join(out), failed
def main(argv=None):
ap = argparse.ArgumentParser()
ap.add_argument("--db", default="data/history.db")
ap.add_argument("--max-fail-share", type=float, default=DEFAULT_MAX_FAIL_SHARE)
ap.add_argument("--max-empty", type=int, default=DEFAULT_MAX_EMPTY)
args = ap.parse_args(argv)
# sqlite3.connect() is lazy: a file that is not a database opens fine and
# only raises on the first read, so the guard has to cover the query too.
try:
conn = open_readonly(args.db)
try:
cards = grade_window(conn)
finally:
conn.close()
except sqlite3.Error as exc:
print(f"ERROR - could not read {args.db}: {exc}")
return 2
if not cards:
print(f"SKIPPED - no records in {args.db}")
return 2
text, failed = report(collect(cards), cards,
max_fail_share=args.max_fail_share,
max_empty=args.max_empty)
print(text)
if failed:
print("\nFAIL - a band table has stopped discriminating. Recalibrate the"
" named constants against this distribution before shipping.")
return 1
print("\nOK - every graded metric still uses its bands.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
The read-only connection is worth a sentence. SQLite’s URI filename mode parameter opens the database read-only, so a stray write is refused by the engine rather than avoided by the script remembering to only SELECT. This thing points at production data; make that structural.
Run it, then prove it bites
python3 seed.py && python3 calibrate.py; echo "EXIT=$?"
Calibration - 48 graded records
metric A B C D F n verdict
-----------------------------------------------
distance 31 8 3 0 6 48 ok - 4/5 bands used, A 65%, D/F 12%
pace 31 8 3 0 6 48 ok - 4/5 bands used, A 65%, D/F 12%
hr 26 8 8 0 6 48 ok - 4/5 bands used, A 54%, D/F 12%
overall (informational, not gated)
letters: A=24 B=15 C=3 D=3 F=3
F-cap fired on 9/48 (19%)
OK - every graded metric still uses its bands.
EXIT=0
A clean run proves nothing until you have watched it fail. Open rubric.py, change SCALE in hr_deviation from 28.0 to 200.0, which makes the heart-rate ceiling so forgiving that nothing can breach it, and re-run:
hr 42 6 0 0 0 48 FAIL - dead bands - C, D, F never used
governed by: GRADE_BANDS, NOISE, SCALE
Then swing it the other way, SCALE = 2.0, so a single beat over the cap is catastrophic:
hr 19 0 4 0 25 48 FAIL - dead bands - B, D never used
governed by: GRADE_BANDS, NOISE, SCALE
Both directions caught, and the second one is instructive: at 52% D/F it slid under the punitive-skew threshold and the dead-band signature caught it anyway. Two loose checks covering each other beat one tight one you have to tune.
Set SCALE back to 28.0 before continuing.
The other half: assert the verdict, not the float
The gate finds a scale that has collapsed. It cannot tell you that a specific run got the wrong letter, because it never looks at any single record. That is what verdict evals are for, and the design decision that makes them survive is to declare bounds rather than exact letters. A bound is the real contract, “this run deserves at least a B”, and it survives ordinary recalibration; an exact letter fights every tuning pass and gets deleted within a month.
Save as test_verdicts.py:
"""Verdict evals: what grade does this record DESERVE."""
import pytest
from rubric import GRADE_POINTS, base_letter, grade_record
BASE = {"distance_m": 8000.0, "target_distance_m": 8000.0,
"pace_s_per_km": 360.0, "target_pace_s_per_km": 360.0,
"avg_hr": 130.0, "cap_bpm": 140.0, "intent": "easy"}
SCENARIOS = {
"obedient_clean": BASE,
# The one that started this: the average obeys the ceiling, but the run
# spent most of its time a beat or two over it.
"straddling_the_cap": {**BASE, "avg_hr": 141.0},
"cap_blown_hard": {**BASE, "avg_hr": 158.0},
# An easy day run slower than target is compliant, not a miss.
"easy_run_slow": {**BASE, "pace_s_per_km": 402.0},
"distance_way_short": {**BASE, "distance_m": 4200.0},
}
EXPECTED_VERDICTS = {
"obedient_clean": {
"min": "A",
"why": "Distance, pace and heart rate all on prescription with room to "
"spare. There is no defensible reading in which this is not an A.",
},
"straddling_the_cap": {
"min": "B",
"why": "One beat over a stated 140 ceiling is sensor noise, not "
"disobedience. Anything below a B here means the metric is "
"punishing measurement error.",
},
"cap_blown_hard": {
"max": "C",
"why": "18 bpm over a stated ceiling, sustained. The cap has to bite "
"here, or the straddling fix has simply disabled the metric.",
},
"easy_run_slow": {
"min": "A",
"why": "An easy day is supposed to be easy. A one-sided expectation "
"must not charge for landing on the free side of the bound.",
},
"distance_way_short": {
"max": "C",
"why": "Roughly half the prescribed distance. A composite that still "
"returns a B is averaging away the only thing that was asked for.",
},
}
def points(letter):
return GRADE_POINTS[base_letter(letter)]
def test_every_scenario_declares_a_verdict():
"""A scenario nobody has judged is a fixture, not an eval. This is what
stops the suite drifting back into asserting mechanics only."""
assert set(EXPECTED_VERDICTS) == set(SCENARIOS)
for name, spec in EXPECTED_VERDICTS.items():
assert spec.get("min") or spec.get("max"), f"{name} bounds nothing"
assert len(spec.get("why", "")) > 40, f"{name} does not say why"
@pytest.mark.parametrize("name", sorted(SCENARIOS))
def test_scenario_meets_its_declared_verdict(name):
spec = EXPECTED_VERDICTS[name]
overall = grade_record(SCENARIOS[name])["overall"]
if "min" in spec:
assert points(overall) >= points(spec["min"]), (
f"{name}: got {overall}, expected at least {spec['min']} - {spec['why']}")
if "max" in spec:
assert points(overall) <= points(spec["max"]), (
f"{name}: got {overall}, expected at most {spec['max']} - {spec['why']}")
def test_the_cap_scenarios_are_strictly_ordered():
"""Obeying >= straddling > blowing it.
An ordering, not three fixed letters, because the failure being guarded is
COLLAPSE: the original rubric scored straddling and blowing it identically.
Any over-correction that makes the cap stop biting fails the second half.
"""
clean = grade_record(SCENARIOS["obedient_clean"])["overall"]
straddled = grade_record(SCENARIOS["straddling_the_cap"])["overall"]
blown = grade_record(SCENARIOS["cap_blown_hard"])["overall"]
assert points(clean) >= points(straddled) > points(blown)
test_every_scenario_declares_a_verdict is the piece that keeps the file honest over time. Without it, somebody adds a scenario, forgets the bound, and the suite quietly goes back to asserting mechanics.
Run it with python3 -m pytest test_verdicts.py -q and you should see 7 passed.
Gotchas
The eval will fail the first time you run it, and it will be right. Writing this example, the last band edge was (0.50, "D"). A run that covered 4.2 km of a prescribed 8 km is a 47.5% deviation, which landed inside D, which meant the F-cap never fired, which meant the composite averaged a near-total miss into an overall B. Every unit test still passed, because 0.475 really does map to D under that table. The escape is to treat the eval as the authority over the constant, not the reverse: moving the last edge to 0.40 produced 7 passed. Then re-run the calibration gate to check that the new edge did not wreck the distribution somewhere else, which is the loop these two tools are designed to form.
Do not gate on concentration in a passing letter. The first draft of the real script failed any letter above a flat 60% share, and it immediately flagged three of five healthy metrics. Distance grades 79% A on live data because the prescribed distances are being hit. Symmetry feels principled and produces a gate people mute in a week.
Do not put this in CI, and say so out loud. It needs a populated production database, and a fabricated one would only ask the fixture whether it agrees with itself. The real repo goes as far as a test named test_the_gate_is_not_wired_into_ci that fails if someone adds it to a workflow, which sounds pedantic until you picture the alternative: a green check next to a corpus that proves nothing.
A dead band on live data is a weaker claim than a wrong grade. It says the metric is unproven over this window, not broken. Running the real gate against 42 sessions today, the prescribed-cap heart-rate metric passes with four of five bands used, while the other heart-rate regime, the rolling reference band used when no ceiling was prescribed, fails with C and F never issued across 32 graded runs. That is a genuine open finding, and it may well turn out to be a thin window rather than a bad table. The value is that it is now a question somebody has to answer before touching a constant, instead of a thing nobody had looked at.
Grade through the production entry point, never a copy. It is tempting to reimplement the scoring inline in the calibration script, because the production path needs a database connection and context objects you have to wire up. Then the two drift, and a clean report means the copy is healthy. Wire up the real thing.
Sources
- Data Validation for Machine Learning (Breck, Polyzotis, Roy, Whang, Zinkevich; MLSys 2019) — the argument for validating distributions rather than examples, including how a slice-local defect hides behind healthy aggregate metrics
- Overview of Classical Test Theory and Item Response Theory (Cappelleri, Lundy, Hays) — items everyone answers identically are uninformative, and variability is what makes a scored item useful
- Google SRE Book, Monitoring Distributed Systems — every alert must be actionable, because over-firing trains people to skim and ignore
- SQLite URI filenames — the
mode=roquery parameter that makes the read-only guarantee the engine’s job
Changelog
- release: 0.43.0 — report-card grading fix, one-page PDFs, read-cache, and a calibration gate (dev → main) (#185) (29ac1b8)
- chore(brand): press v0.7.2 — BRAND VALUES CHANGED (#174) (afc16ec)
- chore(brand): adopt press v0.7.1 (#172) (a0f6798)
- refactor(branding): generate the PRESS token block from the shared brand (#166) (#167) (2a0f229)