Nate Swenson

Adding a free LLM judge to your build, without trusting it blindly

Resume · No. 016 · 10 min read ·v0.2.0

Shipped

v0.2.0 of the resume skill added npm run benchmark, which runs the real tailoring pipeline against seven cached job postings, five picked as a genuine fit and two deliberately mismatched as controls, and scores every result. Some of that scoring is deterministic (rule checks, a keyword-coverage count), and some of it is an LLM judging the tailored résumé against the job, both judges spawned through the Claude CLI so the run costs nothing beyond the existing subscription. The same release also made retries cheaper: when the only thing wrong with a tailored résumé was its summary, the pipeline now fixes just the summary instead of regenerating the whole document, cutting that retry from about 39 seconds to about 5.3 seconds, and it started dropping “optimized” bullets that came back identical to the original instead of letting them pollute the change log.

The part worth teaching isn’t the benchmark itself, it’s the judge. An LLM scoring your own output for free is an appealing idea the moment you have a Claude subscription, and it’s also a good way to make your CI flaky, slow, or silently wrong if you wire it in carelessly. Here’s the shape that avoids all three: how to spawn one, how to make its failure modes boring, and how to decide what it’s allowed to have an opinion on.

Why the judge runs through the CLI instead of the metered API

The Claude CLI’s non-interactive mode reads a prompt with -p, and --output-format json together with --json-schema gets you a schema-validated object back instead of prose you have to hope is well-formed (Claude Code docs: run Claude Code programmatically). Called this way, from a script, it authenticates through the same session as an interactive claude session, so a benchmark that spawns it dozens of times a day doesn’t add a line item anywhere. If you want to confirm that instead of assuming it, the same --output-format json envelope reports total_cost_usd per call, so you can check it directly rather than trust the mental model.

That’s the whole case for routing a judge through the CLI: the same reasoning capability as the metered API, spawned as a subprocess, at no incremental cost. The interesting engineering starts once you treat that subprocess like the unreliable external dependency it is.

Spawning a bounded judge and parsing what it hands back

A judge is a function: give it a prompt and a schema, get back a scored object. child_process.spawn gets you there, but the reply needs defensive handling on the way out. The schema-validated object lands in structured_output when everything goes as documented; a defensive caller also handles the documented fallback shape, where the object is JSON text sitting in .result, sometimes wrapped in a fenced code block.

// judge.mjs
import { spawn } from "node:child_process";

const SCHEMA = {
  type: "object",
  additionalProperties: false,
  properties: { score: { type: "integer" }, reason: { type: "string" } },
  required: ["score", "reason"],
};

function spawnJudge({ prompt, timeoutMs }) {
  return new Promise((resolve, reject) => {
    const args = ["-p", prompt, "--output-format", "json", "--json-schema", JSON.stringify(SCHEMA)];
    const child = spawn("claude", args, { stdio: ["ignore", "pipe", "pipe"] });

    let stdout = "";
    let stderr = "";
    child.stdout.on("data", (c) => (stdout += c));
    child.stderr.on("data", (c) => (stderr += c));
    child.on("error", (err) => reject(new Error(`spawn error: ${err.message}`)));
    child.on("close", (code) => {
      if (code !== 0) return reject(new Error(`claude exited ${code}: ${stderr.slice(-200)}`));
      try {
        const env = JSON.parse(stdout.trim());
        if (env.structured_output !== undefined) return resolve(env.structured_output);
        const unfenced = String(env.result ?? "").trim().replace(/^```(?:json)?\n?/, "").replace(/\n?```$/, "");
        resolve(JSON.parse(unfenced));
      } catch {
        reject(new Error(`unparseable judge reply: ${stdout.slice(0, 200)}`));
      }
    });
  });
}

This resolves or rejects, nothing more. It has no opinion yet on what happens when it’s slow, or when the CLI isn’t installed, or when the JSON just doesn’t parse. That’s on purpose: a judge that’s tangled together with its own error handling is harder to reason about than one that just tells you plainly what happened.

Making failure boring: a timeout that kills, and a wrapper that shrugs

claude -p is a real process on someone’s machine or CI runner, and any real process can hang. SIGTERM is a request a Node.js process can catch and act on before exiting; SIGKILL cannot be caught by anything and terminates the process unconditionally (Node.js docs: process signal events). So a hard timeout sends SIGTERM first and gives the child a few seconds to exit on its own, then follows with SIGKILL as the backstop that always works.

// judge.mjs (continued): spawnJudge from above, now with a kill timeout
// wrapped around the same spawn + parse logic.
const TIMEOUT_MS = 20_000;
const SIGKILL_GRACE_MS = 3_000;

function spawnJudge({ prompt, timeoutMs }) {
  return new Promise((resolve, reject) => {
    const args = ["-p", prompt, "--output-format", "json", "--json-schema", JSON.stringify(SCHEMA)];
    const child = spawn("claude", args, { stdio: ["ignore", "pipe", "pipe"] });

    let stdout = "";
    let stderr = "";
    let settled = false;
    const finish = (fn, arg) => {
      if (settled) return; // a timeout firing after close() already resolved is a no-op, not a double-settle
      settled = true;
      clearTimeout(timer);
      fn(arg);
    };

    const timer = setTimeout(() => {
      try { child.kill("SIGTERM"); } catch {}
      setTimeout(() => { try { child.kill("SIGKILL"); } catch {} }, SIGKILL_GRACE_MS).unref();
      finish(reject, new Error(`judge timed out after ${timeoutMs}ms (child killed)`));
    }, timeoutMs);

    child.stdout.on("data", (c) => (stdout += c));
    child.stderr.on("data", (c) => (stderr += c));
    child.on("error", (err) => finish(reject, new Error(`spawn error: ${err.message}`)));
    child.on("close", (code) => {
      if (code !== 0) return finish(reject, new Error(`claude exited ${code}: ${stderr.slice(-200)}`));
      try {
        const env = JSON.parse(stdout.trim());
        if (env.structured_output !== undefined) return finish(resolve, env.structured_output);
        const unfenced = String(env.result ?? "").trim().replace(/^```(?:json)?\n?/, "").replace(/\n?```$/, "");
        finish(resolve, JSON.parse(unfenced));
      } catch {
        finish(reject, new Error(`unparseable judge reply: ${stdout.slice(0, 200)}`));
      }
    });
  });
}

// Fail-open: any judge failure becomes a visible, neutral result instead of
// an exception that takes the caller down with it.
export async function judge(prompt, timeoutMs = TIMEOUT_MS) {
  try {
    return await spawnJudge({ prompt, timeoutMs });
  } catch (err) {
    return { score: null, reason: `judge_failed: ${err.message}` };
  }
}

The settled guard matters more than it looks. The timeout and the child’s close event are racing against the same promise; without a guard, a slow close arriving right after the timeout fires can try to resolve a promise that already rejected. The guard makes “first one wins, the other is silently discarded” an explicit rule instead of a race a debugger finds for you at 11pm.

The other deliberate choice is what judge returns on failure: not a thrown exception the caller has to remember to catch, and not a silent default score that looks like a real answer. score: null with a judge_failed reason is unmistakably not a verdict, so a report full of them reads as “the judge was down,” not as “everything passed.”

Deciding what the judge is allowed to fail your build over

None of the above answers the harder question: should a judge’s score ever fail a build on its own? claude -p has no temperature pin, so the same prompt can score differently on two runs of the same input, and independent research on LLM-as-judge reliability backs up treating that variance as real: judges show measurable position and verbosity bias, and even strong models miss a meaningful fraction of deliberately degraded answers, which is why the honest framing is development aid, not ground truth (Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators). Anthropic’s own eval guidance points the same direction from a different angle: give a judge room to return “Unknown” instead of forcing a confident guess, and grade one rubric dimension per judge call rather than asking one call to arbitrate everything at once (Anthropic: Demystifying evals for AI agents).

So the judge’s score is reported, never gating. What gates the build is a small set of deterministic, source-of-truth checks that don’t depend on any model call: does the output violate a hard structural rule, did a job that’s supposed to be a bad fit somehow crash the pipeline outright. The judge’s number sits next to that verdict in the report, informative and disagreement-worthy, but it never gets to fail the build by itself.

Use it: run the judge, then break it on purpose

Wire judge up to a real prompt and it returns a scored, schema-shaped object:

const verdict = await judge(
  'Score this sentence 1-5 for clarity: "The gate reads the treatment score ' +
    'and ignores baseline noise." Return ONLY the JSON object the schema describes.'
);
console.log(JSON.stringify(verdict, null, 2));

{
  "score": 5,
  "reason": "Simple subject-verb-object structure, concrete terms, no ambiguity in what the gate does or doesn't do."
}

Force the timeout path by giving it no time to even start the CLI, and the fail-open wrapper does exactly what it’s supposed to: return a visible, non-crashing failure instead of an unhandled rejection.

const verdict = await judge("Score anything 1-5.", 1); // 1ms: guaranteed timeout
console.log(JSON.stringify(verdict, null, 2));

{
  "score": null,
  "reason": "judge_failed: judge timed out after 1ms (child killed)"
}

That second run is the one that matters. A caller that only ever tested the happy path has no idea what its own judge does under load until the first time it happens in CI, three weeks after launch, at the worst possible time.

Gotchas

Reusing a shared HTTP or CLI adapter class for the judge. If your codebase already has an adapter wrapping claude -p for some other purpose, the instinct is to reuse it for the judge too. Check what it was built for first: an adapter designed for a long-running interactive call may have no cancellation hook and a default ceiling measured in minutes, which means wrapping it in an AbortController looks like it should cancel the child process and doesn’t. The symptom is a judge that appears to hang past the timeout you configured. The fix is to give a judge that needs a hard kill its own direct spawn call, not a shared adapter that was never built for cancellation.

Trusting structured_output to always be there. The documented, schema-validated field is structured_output, but a caller that only reads that field and nothing else will throw on the documented fallback case, where the reply lands as fenced or unfenced JSON text inside .result. Strip the fence and parse .result as the second attempt before giving up.

Fail-open that quietly looks like a pass. It’s tempting to make “the judge is unavailable” resolve to some default numeric score so downstream code doesn’t need a null check. Don’t; a benchmark full of silent default scores reads as “everything’s fine” right up until someone notices the judge binary was never installed on that runner. Return something that’s visibly not a real verdict, and make its reason say so.

Sources

Changelog

  • feat(resume): accuracy + speed benchmark + pipeline improvements (#11) (1262519)