Nate Swenson

Your hallucination detector needs a control group

Local Fitness · No. 072 · 11 min read ·v0.39.0

Shipped

This release was an audit of an AI running coach: sixteen fixes across accuracy, coach voice, tool output and speed. The one worth writing up is the smallest. The app has a grounding check that reads every generated brief and flags numbers the model appears to have invented, and when someone finally measured it against the 63 briefs already on disk, it scored real production output worse than random noise.

That’s a general problem. If you run any heuristic over LLM output, a groundedness check, a policy filter, a lint rule, you probably don’t know its false positive rate either. Here’s how to find out.

The trap in a one-shot pipeline

A toolless single-turn generator has no corrective round trip. It writes prose once, and every number it can legitimately cite already exists in the context you handed it. So the check is simple in principle: pull every numeric token out of the prose, compare each against the pool of known values, and flag the ones that look like a real metric but aren’t it.

That last part matters. You want contradiction, not novelty. A brief that says “45 minutes easy” when no metric is near 45 is quoting a prescription, not inventing a measurement. Flagging it is noise. Ragas defines its faithfulness metric the same way, as supported claims over total claims, scored 0 to 1; the interesting engineering is deciding what counts as a claim in the first place.

The failure mode nobody plans for is a check that fires on everything. It still returns a number, the number still goes in a log, and the dashboard still looks alive. You just can’t tell a regression from a Tuesday.

This is not an unusual outcome. A 2025 evaluation of hallucination metrics across 37 models and 4 datasets found that most of them score between 0.50 and 0.65 weighted-F1, barely above a random baseline, and that different metrics correlate weakly or not at all with each other. The lesson worth stealing is that a groundedness score is itself a model output, and it deserves the same skepticism you’d apply to any other.

Build the checker

Start with the pool and the flags. Two ideas do most of the work here: every known value carries the unit it’s measured in, and comparisons keep their sign.

# grounding.py
from __future__ import annotations

import re
from dataclasses import dataclass

_NUM_RE = re.compile(r"[-+]?\d[\d,]*(?:\.\d+)?")

# A number glued to a time-window word ("14 days", "7-day") is a window, not a
# metric claim. Prose is full of these and they collide with real magnitudes.
_WINDOW_AFTER = re.compile(r"[\s-]*(?:day|week|month|year)s?\b", re.IGNORECASE)

# A written duration like "7h 30m" is two numbers glued to unit letters.
# Neither half is a standalone citation.
_DURATION = re.compile(r"\b\d+h\s*\d+m\b")

EXACT_REL = 0.03    # within 3% of a known value: a faithful citation
NEARBY_REL = 0.12   # close but unequal: a corrupted metric
SIGN_REL = 0.50     # right magnitude, wrong sign: qualitatively wrong


@dataclass(frozen=True)
class Known:
    """One citable number, with the unit it is measured in."""
    name: str
    value: float
    unit: str


@dataclass(frozen=True)
class Flag:
    token: str
    nearest: str
    delta: float
    kind: str  # "value" or "sign"

Now the matching. Note that nearest is chosen on signed distance and the sign check runs before the magnitude check, because a flipped sign is wrong in a way a percentage can’t express.

# grounding.py, continued
def _in_duration(text: str, start: int, end: int) -> bool:
    return any(m.start() <= start and end <= m.end()
               for m in _DURATION.finditer(text))


def _unit_of(token: str, text: str, end: int) -> str:
    """Infer a token's unit from the word right after it."""
    tail = text[end:end + 12].strip().lower()
    if token.endswith("%") or tail.startswith("%"):
        return "pct"
    for word, unit in (("bpm", "bpm"), ("mi", "mi"), ("min", "min"),
                       ("steps", "steps")):
        if tail.startswith(word):
            return unit
    return "none"


def flag(text: str, pool: list[Known]) -> list[Flag]:
    """Flag numeric tokens that look like a known value but aren't it.

    Contradiction-only: a number with no nearby known value is treated as an
    unrelated quantity and ignored, never flagged.
    """
    out: list[Flag] = []
    for m in _NUM_RE.finditer(text):
        raw = m.group()
        if _WINDOW_AFTER.match(text[m.end():]):
            continue
        if _in_duration(text, m.start(), m.end()):
            continue
        try:
            value = float(raw.replace(",", ""))
        except ValueError:
            continue

        unit = _unit_of(raw, text, m.end())
        candidates = [k for k in pool if k.unit == unit]
        if not candidates:
            continue

        nearest = min(candidates, key=lambda k: abs(k.value - value))
        scale = max(abs(nearest.value), abs(value), 1e-9)
        rel = abs(value - nearest.value) / scale

        if rel <= EXACT_REL:
            continue
        # Compare SIGNED values, so +22.4 against a real -22.4 is caught.
        if (value * nearest.value < 0
                and abs(abs(value) - abs(nearest.value)) / scale <= SIGN_REL):
            out.append(Flag(raw, nearest.name, value - nearest.value, "sign"))
        elif rel <= NEARBY_REL:
            out.append(Flag(raw, nearest.name, value - nearest.value, "value"))
    return out


def invention_rate(text: str, pool: list[Known]) -> float:
    """Fraction of checkable numeric tokens that got flagged."""
    checkable = 0
    for m in _NUM_RE.finditer(text):
        if _WINDOW_AFTER.match(text[m.end():]):
            continue
        if _in_duration(text, m.start(), m.end()):
            continue
        unit = _unit_of(m.group(), text, m.end())
        if any(k.unit == unit for k in pool):
            checkable += 1
    if checkable == 0:
        return 0.0
    return len(flag(text, pool)) / checkable

Build four arms from one real record

Here’s the part that turns a heuristic into something you can trust. Take a record whose values you already know and generate four versions of the prose, each with a known correct answer. Research on hallucination detection does the same thing deliberately: one recent approach generates both faithful and hallucinated outputs by rewriting system responses so a detector has honest negatives to train and test against.

# arms.py
from __future__ import annotations

import random

from grounding import Known


def faithful(pool: list[Known]) -> str:
    """Every number cited exactly. Expected score: 0.0."""
    by = {k.name: k for k in pool}
    return (
        f"Resting heart rate {by['rhr'].value:.0f} bpm against a "
        f"{by['rhr_baseline'].value:.0f} bpm baseline. "
        f"Freshness is {by['tsb'].value:.1f}. "
        f"You covered {by['distance'].value:.2f} mi."
    )


def corrupted(pool: list[Known], drift: float = 0.08) -> str:
    """Every number shifted by `drift`. Expected score: high."""
    bumped = [Known(k.name, k.value * (1 + drift), k.unit) for k in pool]
    return faithful(bumped)


def sign_flipped(pool: list[Known]) -> str:
    """Signed values negated, magnitudes intact. Expected score: high."""
    flipped = [Known(k.name, -k.value, k.unit) for k in pool]
    return faithful(flipped)


def unrelated(pool: list[Known], rng: random.Random) -> str:
    """Numbers with no relationship to the record. Expected score: LOW.

    This is the arm most detectors quietly fail. A checker that fires here is
    matching on density, not on contradiction.
    """
    noise = [Known(k.name, rng.uniform(1, 200), k.unit) for k in pool]
    return faithful(noise)

The unrelated arm is the one to think hardest about. It’s a negative control in the classic sense, a condition you expect to produce nothing, used to expose spurious signal. Lipsitch, Tchetgen Tchetgen and Cohen formalised this for observational studies in 2010: a negative control is designed to detect both suspected and unsuspected sources of spurious inference. If unrelated numbers score close to genuinely corrupted ones, your detector is responding to how many numbers are in the pool, not to whether the prose contradicts them.

Run it and read the separation

# check.py
from __future__ import annotations

import random
import statistics

import arms
from grounding import Known, invention_rate

# Stand in your own records here. Each Known is (name, value, unit); the unit
# is what stops a 52 bpm heart rate matching a 52-minute duration.
RECORDS = [
    [Known("rhr", 52, "bpm"), Known("rhr_baseline", 55, "bpm"),
     Known("tsb", -22.4, "none"), Known("distance", 5.00, "mi")],
    [Known("rhr", 48, "bpm"), Known("rhr_baseline", 51, "bpm"),
     Known("tsb", -8.1, "none"), Known("distance", 8.01, "mi")],
    [Known("rhr", 61, "bpm"), Known("rhr_baseline", 57, "bpm"),
     Known("tsb", 4.6, "none"), Known("distance", 3.06, "mi")],
]

ARMS = {
    "faithful":  lambda p, rng: arms.faithful(p),
    "corrupted": lambda p, rng: arms.corrupted(p),
    "sign-flip": lambda p, rng: arms.sign_flipped(p),
    "unrelated": lambda p, rng: arms.unrelated(p, rng),
}


def main() -> int:
    rng = random.Random(7)  # seeded: the run is reproducible
    scores = {name: [] for name in ARMS}

    for pool in RECORDS:
        for name, build in ARMS.items():
            scores[name].append(invention_rate(build(pool, rng), pool))

    print(f"{'arm':<10} {'mean':>6}  {'expected':<10}")
    print("-" * 32)
    means = {}
    for name, values in scores.items():
        means[name] = statistics.fmean(values)
        expected = "~0.0" if name in ("faithful", "unrelated") else "high"
        print(f"{name:<10} {means[name]:>6.2f}  {expected:<10}")

    print()
    failures = []
    if means["faithful"] > 0.05:
        failures.append("faithful arm flags its own true citations")
    if means["corrupted"] < 0.50:
        failures.append("corrupted arm slips past the detector")
    if means["sign-flip"] < 0.50:
        failures.append("sign inversions are invisible")
    if means["unrelated"] > means["corrupted"] * 0.5:
        failures.append("unrelated numbers score near corrupted ones; "
                        "the detector is matching density, not contradiction")

    if failures:
        for f in failures:
            print(f"FAIL: {f}")
        return 1
    print("PASS: all four arms separate")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Put the three files in one directory and run it:

python3 check.py

arm          mean  expected  
--------------------------------
faithful     0.00  ~0.0      
corrupted    0.75  high      
sign-flip    1.00  high      
unrelated    0.08  ~0.0      

PASS: all four arms separate

Four arms, four different answers, and a non-zero exit code when they stop separating. That’s a test you can put in CI.

Now try it with the bug that shipped. Change the two comparisons to run on magnitudes, which is what happens the moment someone writes abs() to “normalise” a distance:

        nearest = min(candidates, key=lambda k: abs(abs(k.value) - abs(value)))
        scale = max(abs(nearest.value), abs(value), 1e-9)
        rel = abs(abs(value) - abs(nearest.value)) / scale

arm          mean  expected  
--------------------------------
faithful     0.00  ~0.0      
corrupted    0.75  high      
sign-flip    0.00  high      
unrelated    0.08  ~0.0      

FAIL: sign inversions are invisible

Three of the four arms look healthy. Faithful is clean, corrupted still gets caught, unrelated stays quiet. And a freshness score cited as +22.4 when the real value is -22.4, which is the difference between telling someone they’re rested and telling them they’re wrecked, scores a perfect zero.

Look at why, because it’s nastier than a wrong answer. The sign-mismatch branch is still sitting there in flag(), untouched. It just never runs: once nearest is chosen on magnitude, +22.4 and -22.4 are separated by a relative distance of zero, so the function hits the EXACT_REL check and returns a faithful citation before it ever reaches the sign test. The code that catches inversions was never deleted, it was made unreachable by a comparison three lines earlier. Without the sign-flip arm you’d never learn that, because the arm you don’t build is the failure you don’t measure.

Gotchas

Composite display strings poison the pool. Prose renders a duration as “7h 30m”, and a naive tokeniser reads that as a 7 and a 30. Once signed matching was in, the bare 30 landed at a relative distance of 0.27 from an unrelated freshness value of -22, comfortably inside the sign band, and got flagged as a false sign inversion. Skip both halves the way you skip a time window, and pool the raw values rather than tokens scraped out of formatted strings.

Same number, different units. Two real collisions on one day: a vigorous-intensity total of 52 matched a resting-heart-rate baseline of 52, and a days-to-race count of 53 matched a heart rate of 53. If your pool entries don’t carry units, your detector will keep finding these and calling them hallucinations. That’s why Known has a unit field and flag() filters on it before measuring anything.

A dense pool flags everything. Before the fix, roughly a fifth to a quarter of every integer between 1 and 200 fell inside a flagging band, purely because the pool was big enough that some entry was always nearby. When your pool grows, your false positive rate grows with it, silently. The four-arm run is what makes that visible; the unrelated arm climbs and the separation collapses.

The pool and the prompt drift apart. Some values were rendered into the model’s context but never added to the citable pool, so the model quoting them correctly counted as an invention. If your prompt builder and your pool builder are two functions, they will disagree eventually. Derive both from the same source.

Keep it advisory until the arms separate. This check logs a signal; it never blocks output. Give a detector the power to reject something only after you’ve measured its false positive rate on the faithful and unrelated arms, and it came back near zero on both.

Sources

Changelog

  • release: 0.39.0 — accuracy, persona, UX and speed audit (dev → main) (#162) (b28c74e)