Nate Swenson

The numbers disagreed because they were read a second apart

GitHub Stats · No. 100 · 7 min read ·v0.1.0

Shipped

This release turns a Python profile-metrics CLI into a Claude skill: a deterministic gh and jq core that computes five numbers (commits, followers, stars, pull requests, issues), a repo browser, network-free unit tests over JSON fixtures, and a parity harness that runs the port and the original side by side.

The harness is the part worth copying. Rewriting a tool that reads a live API breaks the usual way you prove a rewrite is faithful, because you cannot diff two outputs that were never supposed to be identical.

Why you cannot diff a port of a live-data tool

The normal move when replacing a tool is Parallel Running: run both, compare. Fowler’s write-up is direct about the ideal, saying you want automated testing that injects known data into “both new and legacy implementations so we can compare outputs for the same set of known inputs.”

Known inputs are exactly what you do not have here. The input is a live account, and it changes while you are reading it. Between the reference call and the candidate call, somebody stars a repo. Now stars differs by one and a strict comparison fails on a port that is perfectly correct.

Some of the numbers are not even stable within a single call. GitHub’s search API caps how long a query may run: “For queries that exceed the time limit, the API returns the matches that were already found prior to the timeout, and the response has the incomplete_results property set to true.” The docs then add the part that matters for a parity check: “Reaching a timeout does not necessarily mean that search results are incomplete. More results might have been found, but also might not.”

So a strict equality check produces failures that mean nothing, and a human learns to ignore the harness. The fix is to make the comparison say what “the same answer” means.

Build the two sides

Give both implementations one shape. Here the two are stubbed so you can run this without either tool installed; in a real harness each body shells out to one implementation and parses its JSON. Save it as metrics.py:

"""Two implementations of the same profile metrics, over a source that moves.

`reference` is the original tool being replaced; `candidate` is the port.
Both read live data, so two calls seconds apart can legitimately disagree.
Swap the bodies for real subprocess calls to your two implementations.
"""
import random


def _live(user):
    rnd = random.Random(user)
    return {
        "followers": rnd.randint(80, 400),
        "following": rnd.randint(20, 200),
        "stars": rnd.randint(200, 4000),
        "prs": rnd.randint(50, 900),
        "pct_closed": rnd.randint(40, 95),
        "top_repo": "atlas",
    }


def reference(user):
    return _live(user)


def candidate(user):
    m = _live(user)
    # Drift: one star landed between the two calls. Not a defect.
    m["stars"] += 1
    # Drift: this estimate is sampled, so it wobbles every call.
    m["pct_closed"] -= 6
    # Defect: the port counts organizations the user follows, too.
    m["following"] += 47
    return m

Three differences, and only one of them is a bug. A harness earns its place by telling them apart.

Compare with a tolerance, and declare what may differ

Two decisions do the work.

The comparison is a predicate, not ==. A count is “the same” when it is within a floor or a percentage, whichever is larger. The floor matters: five percent of 12 is 0.6, so a percentage alone would demand exact equality on every small number, which is where drift is most visible in relative terms.

Every field is either gated or informational, and both lists are written down. Gated fields fail the run. Informational fields are printed on every run and never fail it. This is the same split GitHub’s Scientist library makes with its ignore blocks, which exist for known-acceptable mismatches; the documentation notes those blocks are “only called if the values don’t match,” so expected variation stops generating noise while unexpected variation still surfaces.

The important half is that informational is a declared list, not silence. A sampled percentage you deliberately excluded and a field you forgot to check look identical in a passing run unless the harness prints one and not the other. Save this as parity.py:

#!/usr/bin/env python3
"""Compare a port against the tool it replaces, over data that keeps moving."""
import sys

from metrics import candidate, reference

# Gated: a mismatch here fails the run. These are the numbers a reader quotes.
GATED_COUNTS = ["followers", "following", "stars", "prs"]
GATED_NAMES = ["top_repo"]
# Informational: reported every run, never fails. Sampled or estimated values
# that move on their own. Declaring them is the point; silence would hide them.
INFORMATIONAL = ["pct_closed"]


def within(a, b, floor=2, pct=0.05):
    """True when two live readings are close enough to be the same answer."""
    return abs(a - b) <= max(floor, pct * max(abs(a), abs(b)))


def compare(user):
    ref, cand = reference(user), candidate(user)
    failures = []

    for field in GATED_COUNTS:
        r, c = ref[field], cand[field]
        if not within(r, c):
            failures.append(f"{field}: reference={r} candidate={c}")

    for field in GATED_NAMES:
        if ref[field] != cand[field]:
            failures.append(f"{field}: reference={ref[field]!r} candidate={cand[field]!r}")

    notes = [f"{f}: {ref[f]} vs {cand[f]}" for f in INFORMATIONAL if ref[f] != cand[f]]
    return failures, notes


def main(users):
    failed = 0
    for user in users:
        failures, notes = compare(user)
        status = "FAIL" if failures else "ok"
        print(f"[{status}] {user}")
        for f in failures:
            print(f"       gated  {f}")
        for n in notes:
            print(f"       info   {n}")
        failed += bool(failures)

    print(f"\n{len(users) - failed}/{len(users)} users at parity")
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:] or ["torvalds", "sindresorhus"]))

Note which comparison each field gets. Counts get the tolerance. The name of the top repository gets strict equality, because a different repository is a different answer no matter how close the star counts were.

Run it, and read the failure

python3 parity.py

[FAIL] torvalds
       gated  following: reference=85 candidate=132
       info   pct_closed: 63 vs 57
[FAIL] sindresorhus
       gated  following: reference=35 candidate=82
       info   pct_closed: 79 vs 73

0/2 users at parity

The harness ignored the one-star drift, printed the sampled percentage without failing on it, and caught the real defect on both accounts. It exits non-zero, so it can gate something.

Now fix the port. Delete the two lines in metrics.py that add 47 to following, and run it again:

[ok] torvalds
       info   pct_closed: 63 vs 57
[ok] sindresorhus
       info   pct_closed: 79 vs 73

2/2 users at parity

The informational line is still there, which is the behaviour you want. A field that moves on its own does not become invisible just because the run passed.

Gotchas

This is a first release, so these are the failure modes to watch for rather than traps that have already sprung.

A parity harness depends on the tool you are deleting. This one resolves the original through GITHUB_STATS_CLI_PATH and REFERENCE_PYTHON, and stops with a clear error when that interpreter is missing rather than reporting a false pass. That is the right failure, but it means the check cannot run in CI, cannot run on a fresh clone, and stops working the day the old repository is archived. Escape: treat parity as a migration-time gate with a deadline, and capture its verdict into offline fixture tests that outlive the original. The fixture tests are what still run next year.

A percentage tolerance alone is wrong at both ends. Five percent of 12 is 0.6, so small counts effectively demand exact equality, which is where a single event between two calls does the most relative damage. Five percent of 40,000 is 2,000, which is enough room to hide a real bug. Escape: pair a floor with a percentage as max(floor, pct * magnitude), and set the floor from how fast the underlying number actually moves, not from what makes the current run green.

A count that arrives with incomplete_results: true is not an error. The response is a normal HTTP 200 with a smaller number in it, so nothing in a naive client raises. Since the docs say a timeout does not necessarily mean results were incomplete, you cannot even treat the flag as a clean retry signal. Escape: read the flag explicitly and surface it next to the number, rather than letting a quietly-truncated count flow into a comparison as though it were a measurement.

Tolerance is not a place to put fields you have not thought about. The moment “it’s within tolerance” becomes the answer to every mismatch, the harness stops being evidence. Escape: keep the gated and informational lists short and explicit, and require a reason in the comment when a field moves from one to the other.

Sources

Changelog

  • feat: add github-stats skill (gh-driven, parity-evaluated) (#2) (c92e978)