Nate Swenson

Weighting cannot fix a score whose axes measure the same thing

Local Fitness · No. 081 · 12 min read ·v0.40.0

Shipped

This release split a workout report card into two surfaces: three axes that get letter grades, and a block of numbers that get reported without any grade at all. It also gave a prescribed heart-rate ceiling its own database column, and added a fourth graded axis for whether a session ran continuously or contained a break.

The reason is worth more than the feature. The old card graded four axes, and two of them turned out to be the same measurement pointed in opposite directions. That made the score reward the opposite of what the plan asked for. If you maintain any weighted score, a code review rubric or an LLM eval composite, this failure mode is available to you, and a small weight will not save you from it.

Set up a scorer you can interrogate

Start with a scorer that takes normalised deviations and returns one letter. Nothing here is specific to fitness; a deviation is just “how far off target was this axis, as a fraction”.

# scorer.py
"""A weighted composite scorer: deviations in, one letter out."""
BANDS = ((0.05, "A"), (0.10, "B"), (0.20, "C"), (0.35, "D"))
POINTS = {"A": 4.0, "B": 3.0, "C": 2.0, "D": 1.0, "F": 0.0}
CUTS = ((3.5, "A"), (2.5, "B"), (1.5, "C"), (0.5, "D"))


def grade(deviation):
    """Non-negative relative deviation -> letter. One band table for every axis."""
    for hi, letter in BANDS:
        if deviation <= hi:
            return letter
    return "F"


def composite(deviations, weights, f_cap="C"):
    """Weighted GPA over the axes named in `weights`; axes absent from `weights`
    are ignored entirely. An F on any WEIGHTED axis caps the result at `f_cap`."""
    scored = [(weights[k], POINTS[grade(d)])
              for k, d in deviations.items() if k in weights]
    if not scored:
        return {"grade": "n/a", "gpa": None, "capped": False}
    total_w = sum(w for w, _ in scored)
    gpa = sum(w * p for w, p in scored) / total_w
    letter = "F"
    for cut, candidate in CUTS:
        if gpa >= cut:
            letter = candidate
            break
    hit_f = any(grade(d) == "F" for k, d in deviations.items() if k in weights)
    if hit_f and POINTS[letter] > POINTS[f_cap]:
        return {"grade": f_cap, "gpa": round(gpa, 2), "capped": True}
    return {"grade": letter, "gpa": round(gpa, 2), "capped": False}

Two details matter later. Membership in weights is what decides whether an axis counts, and the f_cap rule can override the weighted average entirely.

Now the two axes. One grades a stated ceiling; you are asked to stay under a number, and going under it is full marks. The other grades accumulated effort against an expectation, and falling short of that expectation costs you.

# axes.py
"""Two axes that look independent and are not.

`effort_load` is a stand-in for any accumulated-intensity metric: it rises
steeply with the same input the ceiling axis is measuring. The constants are
fitted to be illustrative, not to reproduce any vendor's model.
"""
import math

HR_CEILING = 140.0      # the instruction: stay under this
EXPECTED_LOAD = 60.0    # what a typical session of this type banks


def ceiling_deviation(avg_hr, ceiling=HR_CEILING):
    """Over the stated ceiling only. Under it is full marks, by design."""
    return max(0.0, (avg_hr - ceiling) / ceiling)


def effort_load(avg_hr, minutes):
    """Accumulated intensity: duration times an exponential function of HR."""
    per_minute = 0.5 * math.exp(0.0712 * (avg_hr - 126.0))
    return minutes * per_minute


def load_deviation(load, expected=EXPECTED_LOAD):
    """Shortfall against the expectation. Undershooting costs you."""
    return max(0.0, 1.0 - load / expected)

Read those two functions together and the problem is already visible. ceiling_deviation punishes a high input. effort_load is a monotonic function of the same input, so load_deviation punishes a low one. The score contains a variable and its own negation.

That is not an artifact of my toy model. Firstbeat, whose model sits behind training load on Garmin devices, describes it plainly: they “developed a heart rate based model to estimate EPOC”, and note that EPOC “is modeled based on information on exercise intensity (%VO2max), which is affected by the levels of heart rate and respiration rate”, accumulating “during the course of the exercise” (Firstbeat, EPOC Based Training Effect Assessment). Heart rate times duration. The card was grading heart rate, then grading a function of heart rate.

Audit the score for inversion

Do not reason about this from the weights. Sweep the shared input and look at where the optimum lands.

# audit.py
"""Does the score's optimum sit where you asked people to operate?"""
from axes import HR_CEILING, ceiling_deviation, effort_load, load_deviation
from scorer import composite

MINUTES = 50
INTENDED = (118, 132)   # what "comfortably inside the ceiling" actually means
FIXED = {"distance": 0.0, "pace": 0.0}   # nailed, so only the coupled axes move


def score_at(avg_hr, weights):
    deviations = dict(
        FIXED,
        hr=ceiling_deviation(avg_hr),
        load=load_deviation(effort_load(avg_hr, MINUTES)),
    )
    return composite(deviations, weights)


def audit(weights, label):
    print(f"\n{label}")
    print(f"{'avg HR':>7} {'gpa':>5} {'grade':>6}  {'zone':>10}")
    rows = []
    for avg_hr in range(112, 172, 6):
        out = score_at(float(avg_hr), weights)
        if avg_hr > HR_CEILING:
            zone = "violation"
        elif INTENDED[0] <= avg_hr <= INTENDED[1]:
            zone = "intended"
        else:
            zone = "boundary"
        rows.append((avg_hr, out["gpa"], out["grade"], zone))
        cap = " capped" if out["capped"] else ""
        print(f"{avg_hr:>7} {out['gpa']:>5.2f} {out['grade']:>6}  {zone:>10}{cap}")

    def best(zone):
        vals = [r[1] for r in rows if r[3] == zone]
        return max(vals) if vals else float("-inf")

    intended, violation = best("intended"), best("violation")
    print(f"\n  best while intended : {intended:.2f}")
    print(f"  best while violating: {violation:.2f}")
    if violation > intended:
        print("  VERDICT: INVERTED. Breaking the instruction scores higher than "
              "following it.")
    else:
        print("  VERDICT: aligned. Following the instruction is never outscored.")
    return intended, violation


BOTH_GRADED = {"distance": 0.20, "pace": 0.45, "hr": 0.25, "load": 0.10}
COMPLIANCE_ONLY = {"distance": 0.22, "pace": 0.50, "hr": 0.28}

if __name__ == "__main__":
    audit(BOTH_GRADED, "BEFORE: both axes graded, load weighted just 10%")
    audit(COMPLIANCE_ONLY, "AFTER: load removed from the weight table entirely")

Run it with python3 audit.py. The first table is the one that should worry you:

BEFORE: both axes graded, load weighted just 10%
 avg HR   gpa  grade        zone
    112  3.60      C    boundary capped
    118  3.60      C    intended capped
    124  3.60      C    intended capped
    130  3.60      C    intended capped
    136  3.80      A    boundary
    142  4.00      A   violation
    148  3.75      A   violation
    154  3.75      A   violation
    160  3.50      A   violation
    166  3.50      A   violation

  best while intended : 3.60
  best while violating: 4.00
  VERDICT: INVERTED. Breaking the instruction scores higher than following it.

Doing exactly what you were told earns a C. Ignoring the ceiling earns an A. Note the weight on the offending axis: ten percent. That is the part people get wrong when they meet this bug, because the instinct is to turn the weight down, and the weight was already almost nothing.

Two things conspired. The weighted average was fine on its own, 3.60 is comfortably an A by these cuts. What destroyed it was the f_cap rule, which lets a single F override the average no matter how little that axis weighs. A cap rule is a reasonable thing to want; a card that prints “A” above a row reading “F” really is averaging away a finding. But a cap turns any axis, however light, into a veto.

Partition the metrics instead of reweighting them

The fix is to stop scoring the redundant axis and start reporting it. In this codebase that meant deleting load from every weight table rather than shrinking its number, because membership in the table is what composite iterates:

COMPLIANCE_ONLY = {"distance": 0.22, "pace": 0.50, "hr": 0.28}

The axis is still computed and still printed, with its expectation beside it. It just has no letter, and there is no weight for a cap rule to catch. The guarantee becomes structural instead of numerical. The second table from the same run:

AFTER: load removed from the weight table entirely
 avg HR   gpa  grade        zone
    112  4.00      A    boundary
    118  4.00      A    intended
    124  4.00      A    intended
    130  4.00      A    intended
    136  4.00      A    boundary
    142  4.00      A   violation
    148  3.72      A   violation
    154  3.72      A   violation
    160  3.44      B   violation
    166  3.44      B   violation

  best while intended : 4.00
  best while violating: 4.00
  VERDICT: aligned. Following the instruction is never outscored.

The OECD and the European Commission’s Joint Research Centre describe the underlying arithmetic in their handbook on composite indicators: if two collinear indicators enter a composite with weights w1 and w2, the single dimension they both measure ends up carrying weight w1 + w2 (Handbook on Constructing Composite Indicators). Their guidance on when to act is the more useful half: double counting “should not only be determined by statistical analysis but also by the analysis of the indicator itself vis-à-vis the rest of indicators and the phenomenon they all aim to capture.”

That matters because correlation alone is a bad trigger. Rubric criteria are correlated all the time and it is usually fine. A study modelling holistic marks from analytic rubrics found five criterion pairs correlating above 0.7 and chose an algorithm robust to it rather than removing anything (Frontiers in Education). The trigger is not “these move together”. The trigger is “one is computed from the other, and their signs oppose”.

Prove a new axis is independent before you add it

Removing an axis leaves a gap, and the temptation is to fill it with the first idea you have. Make the candidate earn its weight first. An axis is worth adding only if it separates items your current axes already pass; if everything it flags was failing anyway, it is decoration.

# independence.py
"""Before adding an axis, check it discriminates where the existing ones do not."""
from scorer import grade

PASSING = ("A", "B")


def already_passing(items, weights):
    """Items every currently-weighted axis is happy with."""
    return [it for it in items
            if all(grade(it["deviations"][k]) in PASSING for k in weights)]


def independence_report(items, weights, candidate, threshold):
    """`candidate` maps an item to the candidate axis's raw value."""
    passing = already_passing(items, weights)
    flagged = [it for it in passing if candidate(it) > threshold]
    print(f"  items                        : {len(items)}")
    print(f"  pass every current axis      : {len(passing)}")
    print(f"  ...of those, candidate flags : {len(flagged)}")
    for it in flagged:
        print(f"      {it['id']}: candidate = {candidate(it):.2f} "
              f"(threshold {threshold})")
    if flagged:
        print("  VERDICT: independent. It sees things the current axes miss.")
    else:
        print("  VERDICT: redundant here. Everything it flags already fails "
              "elsewhere; do not add it.")
    return flagged


def item(name, distance, pace, hr, worst_segment_ratio):
    return {"id": name,
            "deviations": {"distance": distance, "pace": pace, "hr": hr},
            "worst_segment_ratio": worst_segment_ratio}


WEIGHTS = {"distance": 0.22, "pace": 0.50, "hr": 0.28}

# A small illustrative set. Substitute your own recent items here.
ITEMS = [
    item("s1-even",        0.00, 0.00, 0.00, 1.01),
    item("s2-even",        0.01, 0.02, 0.00, 1.06),
    item("s3-broke-stride", 0.00, 0.01, 0.00, 1.32),   # passes all three, uneven
    item("s4-broke-stride", 0.02, 0.03, 0.02, 1.41),   # passes all three, uneven
    item("s5-too-hot",     0.00, 0.00, 0.40, 1.08),    # already fails on hr
    item("s6-short",       0.45, 0.01, 0.01, 1.55),    # already fails on distance
]

if __name__ == "__main__":
    print("Candidate axis: worst_segment_ratio (slowest segment / median segment)")
    independence_report(ITEMS, WEIGHTS,
                        candidate=lambda it: it["worst_segment_ratio"],
                        threshold=1.15)

python3 independence.py gives you:

Candidate axis: worst_segment_ratio (slowest segment / median segment)
  items                        : 6
  pass every current axis      : 4
  ...of those, candidate flags : 2
      s3-broke-stride: candidate = 1.32 (threshold 1.15)
      s4-broke-stride: candidate = 1.41 (threshold 1.15)
  VERDICT: independent. It sees things the current axes miss.

The two items it flags are the ones the existing axes were blind to; the two that were already failing do not count as evidence, which is the whole point of filtering to already_passing first.

I ran this against real history before adding the axis. Across 61 sessions with at least three full segments, 50 sat at or under a 1.15 ratio and 11 were above it. Four of those cleared every existing axis while their slowest segment ran 30% or more slower than their own median, which is what made the axis worth its weight rather than a fifth way of saying “you went too hard”.

Pick the raw excess over the threshold as your deviation, not a relative one. Dividing by the tolerance is the obvious move and it is wrong here, because the ratio already lives near 1.0, so the division squashes every genuine outlier into the top bands.

Gotchas

A small weight is not protection when a cap rule exists. The redundant axis carried ten percent of an easy day’s weight, which felt safe. The cap rule ignored the weight completely and let one F rewrite the letter. Symptom: a weighted average that clearly reads as an A printing as a C, with the average still shown next to it. Escape: scope every cap or veto rule to the axes actually present in the weight table, and confirm with a sweep rather than by reading the weights.

Recalibrating the redundant axis will not save it. I re-measured the expectation from real history and moved the intent factor from 0.75 to 0.61, which is genuinely more accurate. The compliant case still deviated 0.58, still an F, because the problem was never the target. Symptom: a better constant, the same inversion. Escape: run the audit sweep after the recalibration; if the verdict has not flipped, the constant was not the bug.

The instruction your score claims to grade may not be stored anywhere. The heart-rate ceiling existed only as prose in a free-text description field, so no code could read it. The score compared against a statistical stand-in instead, which happened to land within a bpm of the real ceiling by coincidence of the training mix. Symptom: a genuine breach of a stated limit registering as a rounding error. Escape: give the instruction a column and grade against it; if a rule is worth grading, it is worth storing.

Counting from a truncated view. I derived the corpus figures in this post from the tail of a long report rather than the whole thing, and published a smaller count than reality in the release notes. Symptom: numbers that look plausible and cannot be reproduced from the data. Escape: aggregate with code that consumes the full set, and re-derive every published number from the source rather than from an earlier summary of it.

Sources

Changelog

  • feat: split report-card compliance from training stimulus, add continuity + prescribed HR cap (#164) (#165) (c2f0e58)