Nate Swenson

A baseline pinned to whatever ran most recently

Eval · No. 102 · 8 min read ·v0.2.0

Shipped

This release renames the skill from assay to eval, and fixes something more interesting than a rename: the command that refreshes its frozen baseline was not reproducing that baseline.

Given no arguments, the refresh picked the most recently modified session transcript on the machine and re-pinned the golden to that. So the declared refresh command swapped the graded run for an unrelated one, wrote a completely different golden, and left the resulting enormous diff looking exactly like drift.

Every frozen-fixture setup has this failure mode available to it, and it is invisible from the passing side.

What a frozen baseline is actually pinned to

The reason to freeze anything is that the real inputs move. Grading a run reads a session transcript, which grows every turn, and a repository, which changes every pull request. Pinned against either of those directly, the golden goes red constantly for reasons that have nothing to do with the code under test.

That is the same problem Bazel calls hermeticity: “when given the same input source code and product configuration, a hermetic build system always returns the same output by isolating the build from changes to the host system.” Freezing the inputs is how a byte comparison becomes a statement about your code rather than about your Tuesday.

But freezing creates a second artifact nobody thinks about: the command that unfreezes and refreezes. Approval testing describes the loop as approving output as a golden master and then reviewing and re-approving when requirements change. That re-approval step is where the rot happens, because it is the one operation nobody tests.

The property you need is the one reproducible builds states for artifacts: “given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.” Applied to a fixture refresh: running it on an unchanged tree must give back exactly what is committed.

Build the thing, and freeze its output

A tiny version of the setup. report.mjs is whatever you are pinning:

// The thing under test: turns a recorded run into a graded report.
export function report(trace) {
  const failed = trace.events.filter((e) => !e.ok);
  const lines = [
    `run: ${trace.runId}`,
    `events: ${trace.events.length}`,
    `failed: ${failed.length}`,
    ...failed.map((e) => `  - ${e.id}: ${e.why}`),
  ];
  return lines.join('\n') + '\n';
}

Two committed files hold the frozen state: fixtures/trace.json, the recorded input, and fixtures/report.golden, the output it produces. Create the input and a couple of decoy session files so the failure mode is reachable:

mkdir -p fixtures sessions
cat > fixtures/trace.json <<'JSON'
{"runId":"run-a91c","events":[{"id":"e1","ok":true},{"id":"e2","ok":false,"why":"published without reading the tag back"}]}
JSON
cp fixtures/trace.json sessions/2026-07-04-first.json
cat > sessions/2026-08-01-newest.json <<'JSON'
{"runId":"run-ffed","events":[{"id":"e1","ok":true},{"id":"e2","ok":true},{"id":"e3","ok":true}]}
JSON

The second session is newer and greener. Remember that, because it is what the broken refresh reaches for.

Give the refresh command one obligation

Here is the refresh, written so re-pinning is something you ask for rather than something that happens to you. Save it as update.mjs:

#!/usr/bin/env node
/**
 * Refresh the baseline. This is the command the failing assertion tells you
 * to run, so it has one obligation above all others: on an unchanged tree it
 * must reproduce the committed bytes, not pick a new subject.
 *
 *   node update.mjs                      # re-render the golden from the frozen trace
 *   node update.mjs --session <path>     # deliberately re-pin to a new run
 *   node update.mjs --session latest     # ...to the newest transcript
 */
import { readFileSync, writeFileSync, readdirSync, statSync } from 'node:fs';
import { join } from 'node:path';

import { report } from './report.mjs';

const TRACE = 'fixtures/trace.json';
const GOLDEN = 'fixtures/report.golden';

function latestSession() {
  const files = readdirSync('sessions')
    .map((f) => ({ f: join('sessions', f), t: statSync(join('sessions', f)).mtimeMs }))
    .sort((a, b) => b.t - a.t);
  if (!files.length) throw new Error('no sessions to pick from');
  return files[0].f;
}

const i = process.argv.indexOf('--session');
const requested = i === -1 ? null : process.argv[i + 1];

// The default is the committed trace. Re-pinning is opt-in, because a refresh
// that silently changes the subject makes every diff read as drift.
let source = TRACE;
if (requested === 'latest') source = latestSession();
else if (requested) source = requested;

if (source !== TRACE) {
  writeFileSync(TRACE, readFileSync(source, 'utf8'));
  console.log(`re-pinned trace from ${source}`);
}

const trace = JSON.parse(readFileSync(TRACE, 'utf8'));
writeFileSync(GOLDEN, report(trace));
console.log(`wrote ${GOLDEN} (run ${trace.runId})`);

latestSession still exists, and picking the newest transcript is genuinely what you want right after doing a run worth grading. The change is that you have to say so. Generate the golden:

node update.mjs && cat fixtures/report.golden

wrote fixtures/report.golden (run run-a91c)
run: run-a91c
events: 2
failed: 1
  - e2: published without reading the tag back

The assertion that makes the baseline real

Now test the refresh, not just the output. Copy the tree to a scratch directory, run the refresh there, and compare. Save it as baseline.test.mjs:

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import { readFileSync, cpSync, mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

const HERE = new URL('.', import.meta.url).pathname;
const read = (p) => readFileSync(join(HERE, p), 'utf8');

test('the golden matches the committed trace', () => {
  const trace = JSON.parse(read('fixtures/trace.json'));
  assert.equal(trace.runId, 'run-a91c', 'the graded run is the one that was approved');
});

// The assertion the whole baseline rests on. `update` is what a failing golden
// tells you to run, so running it on an unchanged tree must give back exactly
// what is committed. If it does not, the golden is pinned to nothing.
test('the refresh command reproduces the committed baseline', () => {
  const work = mkdtempSync(join(tmpdir(), 'baseline-'));
  cpSync(HERE, work, { recursive: true });

  const before = readFileSync(join(work, 'fixtures/report.golden'), 'utf8');
  execFileSync('node', ['update.mjs'], { cwd: work, stdio: 'pipe' });
  const after = readFileSync(join(work, 'fixtures/report.golden'), 'utf8');

  assert.equal(after, before, 'update.mjs changed the golden on an unchanged tree');
});

Running the refresh in a copy matters. Run it in place and a broken refresh overwrites the committed golden as a side effect of testing it, and then the comparison you were about to make has nothing left to compare against.

node --test

✔ the golden matches the committed trace (0.334042ms)
✔ the refresh command reproduces the committed baseline (26.77375ms)
ℹ tests 2
ℹ pass 2
ℹ fail 0

Prove it fails

A green two-line suite proves nothing yet. Reintroduce the original defect by replacing the three source lines with the version that has no default:

let source = requested ?? latestSession();

✔ the golden matches the committed trace (0.311792ms)
✖ the refresh command reproduces the committed baseline (26.07875ms)
ℹ tests 2
ℹ pass 1
ℹ fail 1
✖ failing tests:
✖ the refresh command reproduces the committed baseline (26.07875ms)
  AssertionError [ERR_ASSERTION]: update.mjs changed the golden on an unchanged tree

That is the whole bug, caught in one assertion. Note that the first test still passes: the committed trace is untouched on disk, so a check of the inputs sees nothing wrong. Only running the refresh reveals it.

Gotchas

A default that reads the filesystem is a default that changes. “No argument given, so use the most recent one” feels helpful and is the exact shape of this bug: the command’s behaviour depends on file modification times, which are not in the repository and differ on every machine. Escape: when a command has a committed artifact available, make that the default and make anything else explicit. Convenience belongs behind a flag whose name says what it will do.

Regenerating a derived artifact and rewriting a record are different operations. This same release renamed three skills, and every generated artifact embedding an old name was re-frozen by re-running its own update command. The recorded trace was deliberately left alone, because it is a normalized record of a session that actually happened, and editing names inside it would assert something that was not observed. Escape: before a bulk rename touches a fixture, ask whether the file is produced by your tool or recorded from reality. Produced files get regenerated; recorded files are evidence, and evidence does not get corrected to match today’s vocabulary.

Leaving the record honest leaves a mismatch, and that is the right trade. The frozen report now cites current file paths against events that ran under the old ones. It looks like an inconsistency in the fixture and it is the only version of it that is true. Escape: write down why the mismatch exists next to the fixture, or the next person will helpfully fix it.

A refresh test that runs in place destroys its own evidence. The broken refresh writes the golden before you can compare it, so an in-place test either passes vacuously or leaves the working tree dirty in a way that looks like an unrelated change. Escape: copy to a scratch directory, refresh there, compare across the boundary.

Sources

  • Hermeticity, Bazel — the same inputs returning the same output, and why host state is the enemy
  • Definition of Reproducible Builds — bit-by-bit identical artifacts from identical inputs, the property a refresh command needs
  • ApprovalTests — the golden-master loop, including the re-approval step this post is about