Nate Swenson

Your new scale can inherit the old one's calibration

Local Fitness · No. 109 · 12 min read ·v0.50.0

Shipped

The report card that grades a run against its prescription stopped handing out letters and started scoring 1 to 5 with fractional precision. Every compliance metric still reduces to one non-negative relative deviation through one shared grader; that grader now returns a number instead of one of five letters plus a plus-or-minus modifier. The interesting part is not the stars. It is that the new curve was built on the old rubric’s own band edges, so a scale that took months of tuning to trust carried its calibration across the swap intact.

That is the technique this walks through: how to replace a discrete scale with a continuous one without re-earning the trust you already have in it.

Measure the scale before you redesign it

The honest reason this change happened is that I looked at a stack of my own report cards and noticed most of them said the same thing. A quarter of all cards scored a perfect 4.00. Across 240 real cards spanning 730 days, the plus-or-minus modifier appeared on 545 of 749 graded rows, and A+ alone was 63% of distance rows, 90% of heart-rate rows and 76% of continuity. The modifier was telling me a deviation sat in the bottom third of a band whose bottom third held about seventy percent of everything. That is decoration.

Before you touch the curve, write the measurement that says whether you have this problem. It is short, and it is the thing you will re-run afterwards to prove the change did something.

"""The discrete grader you already have, plus the measurement that judges it."""
from collections import Counter

# Each band is (upper edge of relative deviation, label). Anything past the last
# edge is the floor. These edges are the calibrated part; the letters are not.
GRADE_BANDS = ((0.05, "A"), (0.10, "B"), (0.20, "C"), (0.35, "D"))
FLOOR_LABEL = "F"


def grade_from_deviation(d, widen=1.0):
    """Relative deviation -> one of five labels. None in, None out."""
    if d is None:
        return None
    for edge, label in GRADE_BANDS:
        if d <= edge * widen:
            return label
    return FLOOR_LABEL


def occupancy(scores):
    """How much of your scale the data actually uses."""
    graded = [s for s in scores if s is not None]
    counts = Counter(graded)
    top = counts.most_common(1)[0]
    return {
        "n": len(graded),
        "levels_used": len(counts),
        "most_common": top[0],
        "most_common_share": round(top[1] / len(graded), 3),
    }

Two numbers decide it. levels_used tells you how many rungs your data actually reaches, and most_common_share tells you how much of the mass piles on one of them. A five-level scale where one level holds 60% of the rows is a two-level scale wearing a costume.

Categorising a continuous measurement is a known-expensive move, and the clinical statistics literature has quantified the bill. Altman and Royston’s note on dichotomising continuous variables puts it bluntly: cutting a variable at the median “reduces power by the same amount as would discarding a third of the data”, and individuals sitting close together on opposite sides of a cut point “are characterised as being very different rather than very similar”. A grading rubric is that same trade, made on purpose for readability. It is worth making only while the buckets are carrying information.

Lift the boundaries out of the old rubric

Here is the move that saves the calibration. Do not design a new curve. Take the band edges you already have, drop the labels, and use the edges as the knots of a piecewise-linear curve.

"""The continuous curve. Its knots ARE the old band edges."""
from grader import GRADE_BANDS

SCORE_MAX = 5.0
SCORE_FLOOR = 1.0

# The old edges, with the labels dropped. Changing these recalibrates the
# rubric; that is the whole reason they are lifted from GRADE_BANDS rather
# than retyped.
KNOTS = tuple(edge for edge, _label in GRADE_BANDS)

# (normalized position, score at that position). Derived, so a change to KNOTS
# cannot leave the anchors stale.
ANCHORS = ((0.0, SCORE_MAX),) + tuple(
    (k / KNOTS[-1], SCORE_MAX - 1 - i) for i, k in enumerate(KNOTS)
)

SCALE = KNOTS[-1]          # normalizer: z == 1.0 at the old floor edge
NOISE = {"distance": 0.0, "pace": 0.0, "hr": 0.0}


def score_from_deviation(d, metric, widen=1.0):
    """Relative deviation -> a continuous score in [SCORE_FLOOR, SCORE_MAX]."""
    if d is None:
        return None
    z = max(0.0, d - NOISE[metric]) / (widen * SCALE)
    if z <= 0.0:
        return SCORE_MAX
    if z >= 1.0:
        return SCORE_FLOOR
    for (z0, s0), (z1, s1) in zip(ANCHORS, ANCHORS[1:]):
        if z <= z1:
            return s0 + (s1 - s0) * (z - z0) / (z1 - z0)
    return SCORE_FLOOR


def display_score(score, step=0.25):
    """Score -> the value the glyphs draw. Rounding is display, never math."""
    if score is None:
        return None
    return round(score / step) * step

KNOTS is imported, not retyped. That single line is what makes the property survive a future retune: if someone moves a band edge, both scales move together, and nobody can quietly reshape the curve while believing the old validation still applies.

Deriving ANCHORS rather than writing them out matters for the same reason. A hand-written anchor table is a second copy of the boundaries, and second copies go stale exactly when you are busy thinking about something else.

The clamping at both ends is deliberate and it is worth copying. It is the same behaviour NumPy’s own linear interpolation gives you: numpy.interp returns fp[0] for anything below the first sample point and fp[-1] for anything above the last, by default. Past the floor, “how much worse” stops being a useful question. One of my metrics has a live maximum almost seven times its own floor, and an unsaturated linear extension would have scored that card somewhere around minus twenty, dragging any average containing it into nonsense.

Make the safety rail arithmetic

The old rubric had a rule: a failing grade on any weighted metric pinned the overall to a C. That rule exists so a card cannot print a comfortable summary directly above a row that failed. It was a threshold, which means it either fired or it did not.

A continuous scale lets the same rule become arithmetic.

"""Aggregation, with the safety rail expressed as arithmetic."""
from stars import SCORE_FLOOR

# SCORE_FLOOR + HEADROOM is the value the old discrete cap pinned to, so this
# reproduces that rule exactly at the old boundary and degrades linearly on
# either side of it instead of stepping.
HEADROOM = 2.0


def overall(rows, weights):
    """Weighted mean, never allowed to sit more than HEADROOM above the worst row.

    Returns (capped, uncapped) so the card can say why they disagree.
    """
    graded = {m: s for m, s in rows.items() if s is not None}
    if not graded:
        return None, None
    total_weight = sum(weights[m] for m in graded)
    mean = sum(s * weights[m] for m, s in graded.items()) / total_weight
    capped = min(mean, min(graded.values()) + HEADROOM)
    return round(capped, 2), round(mean, 2)


assert overall({"a": SCORE_FLOOR, "b": 5.0}, {"a": 1, "b": 1})[0] == 3.0

SCORE_FLOOR + HEADROOM is 3.0, which is exactly the C the old cap pinned to. That is the second place the migration inherits rather than invents: the new rule reproduces the old one at the old boundary, then degrades smoothly on either side of it. On the real corpus it catches all ten cards the discrete cap caught, plus sixteen more whose worst row was merely bad rather than floored, which the stepping version ignored completely.

Returning both numbers is worth the extra tuple element. When the capped and uncapped values disagree, the card can say so, and a summary that explains why it is lower than its own average is a summary people believe.

Run it and confirm the scale opened up

Do not take the change on faith. Assert the knots did not move, then re-run the occupancy measurement from the first block on the same inputs.

"""Prove the knots survived the swap, then compare how much scale each uses."""
import random

from grader import grade_from_deviation, occupancy
from stars import ANCHORS, KNOTS, display_score, score_from_deviation
from overall import overall

# 1. Every old band edge still lands exactly on a whole score.
for i, edge in enumerate(KNOTS):
    got = score_from_deviation(edge, "pace")
    want = 4.0 - i
    assert abs(got - want) < 1e-9, f"knot {edge} moved: {got} != {want}"
print("knots:", KNOTS)
print("anchors:", tuple((round(z, 4), s) for z, s in ANCHORS))
print("every old band edge still lands on a whole score")

# 2. Same deviations, both scales.
random.seed(7)
deviations = [abs(random.gauss(0, 0.09)) for _ in range(240)]

old = [grade_from_deviation(d) for d in deviations]
new = [display_score(score_from_deviation(d, "pace")) for d in deviations]

print()
print("old", occupancy(old))
print("new", occupancy(new))

# 3. The rail is arithmetic, so it degrades instead of stepping.
weights = {"pace": 0.42, "distance": 0.33, "hr": 0.25}
for worst in (1.0, 1.5, 2.5):
    rows = {"pace": worst, "distance": 5.0, "hr": 5.0}
    capped, mean = overall(rows, weights)
    print(f"worst row {worst}: overall {capped} (uncapped {mean})")

Save the three blocks as grader.py, stars.py and overall.py, this one as check_scale.py, and run it. That is the whole dependency list; it is stdlib Python. Here is what it printed for me:

knots: (0.05, 0.1, 0.2, 0.35)
anchors: ((0.0, 5.0), (0.1429, 4.0), (0.2857, 3.0), (0.5714, 2.0), (1.0, 1.0))
every old band edge still lands on a whole score

old {'n': 240, 'levels_used': 4, 'most_common': 'A', 'most_common_share': 0.421}
new {'n': 240, 'levels_used': 14, 'most_common': 2.75, 'most_common_share': 0.125}
worst row 1.0: overall 3.0 (uncapped 3.32)
worst row 1.5: overall 3.5 (uncapped 3.53)
worst row 2.5: overall 3.95 (uncapped 3.95)

Four occupied levels became fourteen, and the pile-up on one value fell from 42% to 12.5%, on identical inputs. The bottom three lines are the rail behaving like arithmetic: a floored row still pins the overall to exactly 3.0, a merely-bad row of 1.5 pulls it down slightly, and a 2.5 leaves the mean alone.

On the real corpus the shape held. The overall went from 4 occupied levels to 12, perfect scores halved from 33% to 17%, and pace, which carries the heaviest weight on easy and quality days, went from 5 letters to 15 levels. Rank agreement with the old score is 0.944 across all 27,730 card pairs, and the 5.6% that reorder are the cards where the modifier was carrying the difference the letter threw away.

Gotchas

Every site that parsed the old representation breaks silently. A letter grade invites grade[0] as a way to get the base letter, and that slice gets re-implemented wherever someone needs it. Four places in this codebase had their own copy. Under a numeric score none of them raised: a distribution quietly emptied while the line that printed it kept printing, a suppression rule stopped firing so a badly-run quality day was promoted back into the coach’s receipts as “done as prescribed”, a CSS class name matched no rule and the card still rendered cleanly, and a calibration script skipped every metric and exited 0. Of everything touching the old scale, only one pinned description failed loudly. The escape is to grep for the parse, not the symbol. Search for the slice, the regex, the startswith, the class-name concatenation. A type checker will not find these and neither will a shape-level test, because the shape did not change. Fowler’s Replace Primitive with Object is the long-term fix; the short-term one is knowing that the slice is the thing you are hunting.

A retune that looks better on paper can delete safety cases. A percentile-derived per-metric normalizer was simulated here, fitting each metric’s own distribution rather than holding one scale across all four. It bought two extra quarter-star buckets on pace, and it lost 2 of the 10 cards the cap catches. Granularity is easy to measure and easy to fall in love with. Before adopting a candidate curve, score it on the cases the old rule caught, and treat losing any of them as a regression regardless of what the histogram looks like.

Pick the headroom that reproduces the old boundary, not the round one. 2.0 is not there because it is tidy; it is there because floor plus 2.0 lands on the exact value the discrete cap pinned to. A headroom of 2.5 was tried and silently stopped capping 4 of the 10 cards, which is the one thing the constant exists to prevent. Whenever you replace a threshold with a formula, solve for the constant that makes the formula agree with the threshold at the point the threshold fired.

Your font may not have the glyph you just designed around. The brand monospace here carries 963 codepoints and no star among them, so a text star would render in whatever the host machine happened to substitute. Drawing the glyphs as inline SVG fixed it, and then partial fills introduced their own trap: clipping a star at 75% of its bounding box leaves 89.6% of the ink, because a star is widest in the middle. The first 4.75 was visually identical to a 5.00 until the cut table was linearised by area instead of width.

Migrate the readers in a deliberate order. Everything above is one instance of what Fowler calls Parallel Change: expand, migrate, contract. The reason the contract phase deserves its own pass is that a partially migrated numeric scale does not crash, it just reports things that are almost right.

Sources

Changelog

  • feat(report-card): continuous 1-5 star rating replaces letter grades (0.50.0) (#198) (#199) (896fdad)