Nate Swenson

Only the consumer without a version badge got the right answer

Press · No. 097 · 9 min read ·v0.8.1

Shipped

0.8.1 fixed a false positive in a fan-out tool. The tool labels each pull request by what actually moved: a values change gets a loud title and a design review, a version-only adoption gets a quiet title and a one-line diff. On the previous release, three of the four consumer repositories got the loud title for a release in which no design value had changed at all.

The one consumer that got it right was the only one with no version line in its README. That is what pointed at the cause. The emitter for that version line writes the generator’s own version into the body it emits, so a pure version bump moves those bytes, and a comparison against the newly-emitted body reports a content change every single time.

Any generator whose output embeds its own version has this bug available to it. This guide reproduces it, fixes it with a comparison that asks a better question, and pins both directions with tests.

Setup: an emitter that writes its own version into its output

Two emitters, which is the minimum needed to see the problem. One writes the version into the body it produces; the other does not:

// emit.mjs — two emitters. Only one of them writes the version into its output.
export function versionBadge(tokens, { version }) {
  return [
    `**Brand:** kit v${version}, accent ${tokens.accent}.`,
    'Generated from the design tokens; edit tokens.json and re-emit.',
  ].join('\n');
}

export function cssVars(tokens) {
  return `:root {\n  --accent: ${tokens.accent};\n}`;
}

export const EMITTERS = { 'version-badge': versionBadge, 'css-vars': cssVars };
export const emitBody = (name, tokens, ctx) => EMITTERS[name](tokens, ctx);

A version line in a README is a genuinely useful thing to generate. Without one, the only places a consumer’s version of a shared tool appears are a CI pin and a comment marker, and nobody reads either when they land on a repository. The cost is that the emitter now has a run-time input, the version, mixed into output that is otherwise a pure function of the tokens.

The comparison that answers the wrong question

The natural way to ask “did anything change” is to emit fresh and compare:

// compare.mjs — did the VALUES move, or only the version?
import { emitBody } from './emit.mjs';

const trim = (s) => s.replace(/\s+$/, '');

/** v1: compare what is on disk against what we would write now. */
export function valuesMovedNaive(found, emitter, tokens, version) {
  return trim(found.body) !== trim(emitBody(emitter, tokens, { version }));
}

That is correct for css-vars and wrong for version-badge, and the difference is invisible from the call site. Both are just emitters. The question being asked, “are the bytes on disk what I would write now”, quietly includes “was this written by the release I am currently shipping”, which is a different question and always answers no on a release.

Compare at the version the file records

The region on disk records which release wrote it. Use that: re-emit with today’s tokens at the recorded version and compare against that instead. If they match, no value moved, whatever the version substitution did to the bytes.

/**
 * v2: compare what is on disk against what TODAY'S tokens would emit AT THE
 * VERSION THE REGION RECORDS. If those match, no value moved, whatever the
 * version substitution did to the bytes.
 */
export function valuesMoved(found, emitter, tokens, version) {
  if (!found.version) return valuesMovedNaive(found, emitter, tokens, version);
  const atRecorded = emitBody(emitter, tokens, { version: found.version });
  return trim(found.body) !== trim(atRecorded);
}

This is the same trick reproducible builds use for timestamps. The Reproducible Builds project treats build-time metadata as the primary obstacle to bit-for-bit comparison, noting that “Timestamps make the biggest source of reproducibility issues. Many build tools record the current date and time”, and the remedy is SOURCE_DATE_EPOCH, a fixed value substituted in so that “Tools that support it will use its value … instead of the current date and time” (timestamps). Holding the version constant across a comparison is the same move: pin the volatile input so the comparison is about the content.

Use it, then verify it

Three consumer states, one release, both comparisons side by side:

import { emitBody } from './emit.mjs';
import { valuesMoved, valuesMovedNaive } from './compare.mjs';

const tokens = { accent: '#f26722' };
const RELEASE = '0.8.0';
const wroteBy = '0.7.2';

// What each consumer has on disk, written by the previous release.
const cases = [
  ['README badge, no value moved', 'version-badge', tokens],
  ['README badge, accent moved',   'version-badge', { accent: '#e05500' }],
  ['stylesheet, no value moved',   'css-vars',      tokens],
];

console.log('consumer state                 naive   fixed   PR title');
for (const [name, emitter, diskTokens] of cases) {
  const found = { version: wroteBy, body: emitBody(emitter, diskTokens, { version: wroteBy }) };
  const naive = valuesMovedNaive(found, emitter, tokens, RELEASE);
  const fixed = valuesMoved(found, emitter, tokens, RELEASE);
  const title = fixed ? `kit v${RELEASE}: BRAND VALUES CHANGED` : `adopt kit v${RELEASE}`;
  console.log(`${name.padEnd(30)} ${String(naive).padEnd(7)} ${String(fixed).padEnd(7)} ${title}`);
}

node check.mjs

consumer state                 naive   fixed   PR title
README badge, no value moved   true    false   adopt kit v0.8.0
README badge, accent moved     true    true    kit v0.8.0: BRAND VALUES CHANGED
stylesheet, no value moved     false   false   adopt kit v0.8.0

Row three is the diagnostic. Under the naive comparison the stylesheet consumer is the only one reporting false, because it is the only target whose emitter does not embed the version. When one consumer disagrees with all the others, the thing it does differently is the bug.

Pin both directions, not just the one that was wrong

A fix that makes the false positive go away can also make the detector blind. Both properties need a test, and the negative one is the easy one to skip:

import test from 'node:test';
import assert from 'node:assert/strict';
import { emitBody } from './emit.mjs';
import { valuesMoved } from './compare.mjs';

const tokens = { accent: '#f26722' };
const onDisk = (emitter, t, version) => ({ version, body: emitBody(emitter, t, { version }) });

test('a version-only bump of a version-embedding emitter is NOT a values change', () => {
  const found = onDisk('version-badge', tokens, '0.7.2');
  assert.equal(valuesMoved(found, 'version-badge', tokens, '0.8.0'), false);
});

test('a real values change in the same release is still detected', () => {
  const found = onDisk('version-badge', { accent: '#e05500' }, '0.7.2');
  assert.equal(valuesMoved(found, 'version-badge', tokens, '0.8.0'), true);
});

test('a region with no recorded version falls back to a direct comparison', () => {
  const found = { version: null, body: emitBody('css-vars', tokens) };
  assert.equal(valuesMoved(found, 'css-vars', tokens, '0.8.0'), false);
});

node --test compare.test.mjs

✔ a version-only bump of a version-embedding emitter is NOT a values change
✔ a real values change in the same release is still detected
✔ a region with no recorded version falls back to a direct comparison
ℹ tests 3
ℹ pass 3
ℹ fail 0

Confirm the first test is not decorative by reverting the fix, running it, and watching it fail. A regression test you have never seen fail is a test you have not written yet.

The same confusion, twice in one afternoon

The interesting part is that this is not one bug. Within about fifteen minutes of the same release, a sibling tool’s CI went red on an unrelated pull request, and both fixes landed in one commit. That tool’s golden fixture froze an entire rendered region, including the marker line, and the marker line carries the generator’s version. Bumping the generator to 0.8.1, which changed nothing that tool emits, failed its test.

The fix there is the mirror image: freeze the region body, not the rendered block. The marker format belongs to the generator and is covered by the generator’s own tests; a consumer’s golden should assert only what the consumer produces. The general rule underneath both bugs is that a version bump is not a content change, and anything comparing generated output has to be told which parts are content.

Semantic versioning is explicit that a patch release is for “backward compatible bug fixes” (semver.org), so if a patch bump makes your comparison shout, the comparison is measuring the wrong thing.

Gotchas

Only some emitters have this problem, and the call site cannot tell which. A generator with ten emitters where one embeds the version will produce a bug that appears to depend on the consumer rather than on the code. Symptom: a subset of repositories misbehaving, with no pattern in the repositories themselves. Escape: look for what the outlier does differently, not what the majority share.

A comparison that is always true fails silently, in the loud direction. Nothing errors. The pull requests open, the diffs are correct, only the label lies. Symptom: the important label appears on every release, so people stop reading it. Google’s SRE book names the endpoint of that trajectory: when alerts fire too often “employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a ‘real’ page that’s masked by the noise” (Monitoring Distributed Systems). Escape: for any classification your automation emits, write the test that asserts the quiet case stays quiet.

Fixing a false positive can blind the detector. The tempting shortcut is to strip version-looking substrings before comparing, which also strips a genuine change to a value that happens to look like a version. Escape: re-render at the recorded version instead of scrubbing text, and keep a test that a real values change in the same release is still caught.

Regions with no recorded version need an explicit path. A file written before you added the receipt has no version to re-render at, and emitBody(..., { version: undefined }) will happily produce kit vundefined. Symptom: the first run after adding receipts reports every legacy consumer as changed. Escape: branch on the missing receipt and fall back to a direct comparison, as the fixed function does.

Goldens that include a version rewrite themselves on every release. If a frozen fixture contains the generator’s version, every release fails the test, and the standard response of regenerating the golden trains you to regenerate it without reading it. Escape: freeze with a fixed placeholder version, or freeze only the body, and test version substitution in its own test.

Sources

Changelog

  • fix(press): a version-only release no longer claims the brand changed (0.8.1) (#142) (618edc6)