Nate Swenson

An approval that outlives the thing it approved

Issue Flow · No. 142 · 10 min read ·v0.6.0

Shipped

issueflow is the skill I use to walk a GitHub issue to a pull request through a series of stages. Until this release each stage waited for me to type accept. Version 0.6.0 adds start --auto: a dispatched red-team reviewer attacks each delivered artifact, sends it back with cited findings until a round finds nothing blocking, and only then does the run advance. I read the findings tables afterwards instead of sitting at each gate.

The interesting part is not the reviewer. It is what makes its verdict count. A review only registers once its findings parse against a strict grammar, every citation resolves to something that exists, the verdict is derived from the severities rather than declared, and the whole record is bound to the SHA-256 of the artifact it read (and, for code, the commit). accept --auto trusts nothing except that persisted record, and refuses the moment the artifact or the branch differs from what was reviewed. That binding is the technique this guide builds.

Why bind an approval to a hash

GitHub already does a version of this for humans. With the branch-protection option to dismiss stale approvals, the protected branches documentation says that if the diff changes after an approving review, the approval is dismissed as stale and the pull request cannot merge until someone approves again. The approval was for a state, not for the branch name.

An automated gate needs the same property, and it needs it more, because nobody is watching the gap between “reviewed” and “approved”. If the record of the review says only pass, then an edit to the plan after the review, or a new commit pushed after the code review, walks through on a verdict that was about different bytes.

Git’s own storage is the model. The Pro Git chapter on objects describes Git as a content-addressable filesystem: hand it content, get back a key derived from that content, and the key is how you get the content back. A review record that carries the content hash of the artifact is addressing the same way. Same bytes, same hash, approval stands. Different bytes, different hash, approval is void.

Step 1: parse findings, and refuse the ones that cite nothing

Create review.mjs. A finding is one line: a severity in brackets, a citation, a separator, the text. Anything that starts like a finding but does not match the grammar is malformed and refuses the whole review, because a finding the parser cannot read is a finding the gate cannot act on.

// review.mjs
import { createHash } from 'node:crypto';
import { execFileSync } from 'node:child_process';
import { existsSync, readFileSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';

export const sha256OfFile = (path) =>
  createHash('sha256').update(readFileSync(path)).digest('hex');

export function headOf(tree) {
  try {
    return execFileSync('git', ['rev-parse', 'HEAD'], {
      cwd: tree, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'],
    }).trim();
  } catch {
    return null;
  }
}

const FINDING = /^\s*[-*]\s+\[(critical|high|medium|low)\]\s+(\S(?:.*?\S)?)\s+::\s+(\S.*)$/;

export function parseFindings(text) {
  const findings = [];
  const malformed = [];
  for (const line of text.split('\n')) {
    if (!/^\s*[-*]\s+\[/.test(line)) continue;
    const m = FINDING.exec(line);
    if (m) findings.push({ severity: m[1], cite: m[2].replace(/`/g, ''), text: m[3].trim() });
    else malformed.push(line.trim());
  }
  return { findings, malformed };
}

The createHash('sha256') call is the Node crypto module; one update with the file bytes, one digest('hex'), and a Hash object is not reusable after digest, so create a new one per file.

A citation has to resolve. Two shapes are enough for a plan: path:line against the repository, and plan.md § Heading against the artifact’s own headings.

export function resolveCitation(cite, { root, artifactText = '' }) {
  const heading = /^(\S+\.md)\s+§\s+(.+)$/.exec(cite);
  if (heading) {
    const escaped = heading[2].replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
    return new RegExp(`^#{1,6}\\s+${escaped}\\s*$`, 'm').test(artifactText)
      ? { ok: true }
      : { ok: false, reason: `no heading "${heading[2]}" in ${heading[1]}` };
  }
  const loc = /^(.+?):(\d+)$/.exec(cite);
  if (!loc) return { ok: false, reason: 'not path:line or <artifact>.md § Heading' };
  const file = join(root, loc[1]);
  if (!existsSync(file)) return { ok: false, reason: `${loc[1]} does not exist` };
  const total = readFileSync(file, 'utf8').split('\n').length;
  return Number(loc[2]) <= total
    ? { ok: true }
    : { ok: false, reason: `${loc[1]} has ${total} lines, cited line ${loc[2]}` };
}

export const BLOCKING = ['critical', 'high'];

export const deriveVerdict = (findings) =>
  findings.some((f) => BLOCKING.includes(f.severity)) ? 'blocked' : 'pass';

The verdict is derived, never read from the review. A reviewer who writes “Verdict: pass” above a high finding has written a review that disagrees with itself, and the registrar should register nothing.

Step 2: register the round, bound to the bytes

The registrar reads the review, parses it, resolves every citation, and writes a small JSON record. The two fields that make the record an approval rather than an opinion are artifactSha and head.

export function registerReview({ artifact, reviewMd, out, root, codeStage = false }) {
  const { findings, malformed } = parseFindings(readFileSync(reviewMd, 'utf8'));
  if (malformed.length > 0) {
    throw new Error(`malformed finding line, expected "- [severity] <citation> :: <finding>": ${JSON.stringify(malformed[0])}`);
  }
  const artifactText = readFileSync(artifact, 'utf8');
  for (const f of findings) {
    const r = resolveCitation(f.cite, { root, artifactText });
    if (!r.ok) throw new Error(`citation "${f.cite}" does not resolve (${r.reason}); a finding that cites nothing is an opinion`);
  }
  const head = codeStage ? headOf(root) : null;
  if (codeStage && !head) throw new Error('a code review that names no commit cannot say which code it reviewed');
  const record = { verdict: deriveVerdict(findings), findings, artifactSha: sha256OfFile(artifact), head };
  writeFileSync(out, `${JSON.stringify(record, null, 2)}\n`);
  return record;
}

In the real skill the registrar also requires a Not examined section, and refuses a review that reports zero findings with that section empty, on the grounds that “clean” without naming what nobody looked at is indistinguishable from “unreviewed”. I left that out of the walkthrough to keep it short; it is a one-line check worth adding.

Step 3: the gate that trusts only the record

acceptAuto is a narrower gate than the human one, never a bypass. Every check the human path runs still runs first (not shown here); this adds four refusals on top.

export function acceptAuto({ artifact, verdictPath, root }) {
  if (!existsSync(verdictPath)) {
    throw new Error('no red-team review is registered; in an auto run the review IS the approval');
  }
  const v = JSON.parse(readFileSync(verdictPath, 'utf8'));
  if (v.verdict !== 'pass') throw new Error(`the last round is ${v.verdict}; fix the findings, do not bypass the gate`);
  if (v.artifactSha !== sha256OfFile(artifact)) throw new Error('the artifact changed after it was reviewed; review it again');
  if (v.head && v.head !== headOf(root)) throw new Error('the branch moved after the review; review the new head');
  return 'approved';
}

Use it, then try to sneak past it

Save this as demo.mjs beside review.mjs. It builds a scratch repository, registers a review that cites a line that does not exist (refused), registers one that resolves (pass), approves, then edits the artifact and pushes a commit and tries again.

// demo.mjs
import { execFileSync } from 'node:child_process';
import { appendFileSync, mkdirSync, mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { acceptAuto, registerReview } from './review.mjs';

const root = mkdtempSync(join(tmpdir(), 'gate-'));
const git = (...a) => execFileSync('git', ['-c', 'user.name=demo', '-c', 'user.email=demo@example.com', ...a], { cwd: root, stdio: 'ignore' });
mkdirSync(join(root, 'src'));
writeFileSync(join(root, 'src/app.js'), 'export const add = (a, b) => a + b;\n');
writeFileSync(join(root, 'plan.md'), '## Root cause\n\nadd() is never called with strings.\n');
git('init', '-q'); git('add', '.'); git('commit', '-q', '-m', 'plan');

const attempt = (label, fn) => {
  try { console.log(`${label}: ${fn()}`); } catch (e) { console.log(`${label}: REFUSED (${e.message})`); }
};
const args = { artifact: join(root, 'plan.md'), root, out: join(root, 'verdict.json'), codeStage: true };

writeFileSync(join(root, 'r1.md'), '- [high] src/app.js:40 :: add() has no guard\n');
attempt('round 1', () => registerReview({ ...args, reviewMd: join(root, 'r1.md') }).verdict);

writeFileSync(join(root, 'r2.md'), '- [medium] plan.md § Root cause :: rule out the string-input path explicitly\n');
attempt('round 2', () => registerReview({ ...args, reviewMd: join(root, 'r2.md') }).verdict);
attempt('accept  ', () => acceptAuto({ artifact: args.artifact, verdictPath: args.out, root }));

appendFileSync(join(root, 'plan.md'), '\n## Approach\n\nNew section nobody reviewed.\n');
attempt('accept after edit', () => acceptAuto({ artifact: args.artifact, verdictPath: args.out, root }));

git('add', '.'); git('commit', '-q', '-m', 'edit the plan');
writeFileSync(join(root, 'plan.md'), '## Root cause\n\nadd() is never called with strings.\n');
attempt('accept after commit', () => acceptAuto({ artifact: args.artifact, verdictPath: args.out, root }));

Run node demo.mjs. You should see the first round refused for a citation that does not resolve, the second registered as a pass, one approval, and then two refusals for two different reasons:

round 1: REFUSED (citation "src/app.js:40" does not resolve (src/app.js has 2 lines, cited line 40); a finding that cites nothing is an opinion)
round 2: pass
accept  : approved
accept after edit: REFUSED (the artifact changed after it was reviewed; review it again)
accept after commit: REFUSED (the branch moved after the review; review the new head)

The last case is the one worth staring at. The artifact was restored to the reviewed bytes, so artifactSha matches again, and the gate still refuses because the commit moved. For a code stage both bindings have to hold.

Gotchas

The one-line grammar will refuse a real review over formatting. The strict - [severity] <citation> :: <text> line is what makes parsing deterministic, and it is also brittle in the hands of a model. In the dogfood run that followed this release, the grammar refused an entire review over a backtick twice in twenty-four real rounds; the reviewer had wrapped a citation in backticks, which is how code gets written everywhere else it had ever seen code. The replace(/\x60/g, '') on the citation above is the small fix. The larger one, which landed in the next release, was to have reviewers emit findings as JSON and stop parsing prose at all.

A refused stage that re-runs from the same brief will produce the same artifact. Before this release, when a review blocked a stage, the stage was re-dispatched with a byte-identical brief. The refusal existed only in the conversation, so the subagent had no idea what to change. The escape is a feedback seam: the re-brief carries a “Review feedback, round N” section with the blocking findings and the path to the full review. If your gate can say no, the thing it says no to has to be able to read why.

A gate with no cap is a loop. Two reviewers that disagree, or a subagent that cannot satisfy a finding, will go around forever if nothing stops them. Three blocked rounds on one stage stops the run here: a fourth brief and a fourth review are both refused, and the open findings are handed to a person. The auto path never touches the human --force flag; if it could, the cap would be decorative.

Keep timestamps out of the approval record. I keep frozen reference outputs for the registrar and byte-compare them in tests. A timestamp in the verdict file would make every frozen record differ on every run, so round timing lives on the run log instead and the verdict file holds only what the gate reads.

Sources

Changelog

  • issueflow 0.6.0 — auto mode: red-team adversarial review gates (#245) (7985317)