Score your prompt against the types your code already trusts
Shipped
local-fitness pulls my Garmin data into a local SQLite database and a Claude agent writes a daily training briefing from it. Through v0.1.0 it was a personal script; this release put guardrails under it: a pytest suite over the deterministic core (the database layer, the schemas, the prompts, the baselines) behind a coverage gate, a CI workflow that runs lint, tests, and a prompt scorer on every push, and a version-driven release workflow that only cuts a GitHub Release when the version actually changes. The first CI run earned its keep immediately, failing on a test that had been silently reading my real local database. The README also got rewritten from personal notes into a shareable open source project, with badges, a license, and a privacy section up front.
The part worth teaching is the scorer. A prompt doesn’t compile, so the usual safety net for catching a broken contract isn’t there; the way around that is to grade the prompt against grounded pass/fail checks the same way you’d grade any other untested surface, what Anthropic calls an eval built from groundedness and coverage checks (Anthropic: Demystifying evals for AI agents). The briefing’s output is checked against a typed contract, and the prompt that produces that output is free-form English describing the same contract in prose. Nothing stops those two descriptions from drifting apart. Here’s how to catch it before it ships.
Setup: give your schema’s allowed values a name
Somewhere in a prompt-driven agent there’s a piece of code that validates what the model produced. In this project that’s a couple of Literal types: the only tones a briefing item can carry, and the only metrics it can chart.
# schema.py
from typing import Literal
# The only tones a takeaway is allowed to carry.
Tone = Literal["positive", "caution", "critical", "neutral"]
# The only metrics a takeaway is allowed to chart.
Metric = Literal["rhr", "sleep_seconds", "steps", "ctl", "atl", "tsb"]
A Literal type “indicate[s] to type checkers that the annotated object has a value equivalent to one of the provided literals.” That’s the property worth exploiting: the type itself is a finite, enumerable list of the only values that are allowed, and Python can hand you that list back at runtime.
Build it: put the same values in the prompt as a checkable block
The prompt tells the model the same thing the schema enforces, just in prose. Somewhere in the instructions it has to say what tones and metrics are valid, because the model can’t see schema.py.
# prompt.txt
Write today's briefing as a takeaway with a headline, a tone, and a metric.
Never fabricate a number; if a metric is missing, say so plainly.
TONE: one of: positive | caution | critical | neutral
METRIC: one of: rhr | sleep_seconds | steps | ctl | atl | tsb
The rest of the prompt can read however you want it to; the two labeled lines are the only part a scorer needs to find. Keeping them in a predictable LABEL: one of: a | b | c shape is what makes them checkable without turning the whole prompt into a spec sheet.
Build it: score the prompt against the schema
The scorer imports the schema, pulls the real allowed values out of it, pulls the advertised values out of the prompt with a regex, and asserts the second is a subset of the first.
# score_prompt.py
from __future__ import annotations
import re
import sys
from pathlib import Path
from typing import get_args
from schema import Metric, Tone
PROMPT_PATH = Path(__file__).parent / "prompt.txt"
def advertised(label: str, prompt: str) -> set[str]:
"""Pull the values after 'LABEL: one of: a | b | c' out of the prompt."""
match = re.search(rf"^{label}:\s*one of:\s*(.+)$", prompt, re.MULTILINE)
if not match:
return set()
return {v.strip() for v in match.group(1).split("|") if v.strip()}
def score(prompt: str) -> list[str]:
"""Return a list of failures. Empty list means the prompt is in sync."""
failures: list[str] = []
checks = [
("TONE", set(get_args(Tone))),
("METRIC", set(get_args(Metric))),
]
for label, allowed in checks:
drift = advertised(label, prompt) - allowed
if drift:
failures.append(
f"prompt advertises non-schema {label.lower()}: {sorted(drift)}"
)
return failures
def main() -> int:
prompt = PROMPT_PATH.read_text()
failures = score(prompt)
for f in failures:
print(f"FAIL: {f}", file=sys.stderr)
if failures:
return 1
print("PASS: prompt matches the schema.")
return 0
if __name__ == "__main__":
sys.exit(main())
typing.get_args() is the piece doing the real work: called on a Literal, it hands back a tuple of the values you defined, so get_args(Tone) is ("positive", "caution", "critical", "neutral") without you ever re-typing that list. The scorer never re-defines the allowed values; it always asks the type. And the check runs one direction only: the prompt’s values must be a subset of the schema’s, not the reverse. The schema is allowed to know about a value the prompt hasn’t started using yet; the prompt is never allowed to promise something the schema doesn’t recognize.
Wire it into CI
Run the scorer next to lint and tests, so a drifted prompt fails the build the same way a broken test does.
# .github/workflows/ci.yml
name: CI
on: [push, pull_request]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -e ".[dev]"
- name: Lint
run: ruff check .
- name: Tests
run: pytest
- name: Score the agent prompt
run: python score_prompt.py
Use it and verify it
With the prompt and schema above in sync, the scorer passes:
$ python3 score_prompt.py
PASS: prompt matches the schema.
$ echo $?
0
Now make the change that actually breaks briefings in production: rename a value in the schema and leave the prompt alone, the way you would if you renamed a metric during a refactor and forgot the prompt still promises the old name.
# schema.py
Metric = Literal["rhr", "sleep_seconds", "steps", "ctl", "fatigue_load", "tsb"]
$ python3 score_prompt.py
FAIL: prompt advertises non-schema metric: ['atl']
$ echo $?
1
That’s the failure this whole check exists to move earlier. Without the scorer, the model still emits atl because the prompt still tells it to, the schema rejects it because atl no longer exists, and a real briefing request breaks at parse time in production. With the scorer, the same mismatch fails a CI job on push instead.
Gotchas
A test that only passes on your machine isn’t testing anything. CI’s first run on this project failed immediately: the security tests queried the database and blew up with no such table: daily_metrics on a fresh checkout, because they’d been silently reading a real local database file that only existed on the author’s laptop. The fix was a fixture that points the app at a freshly schema-initialized temp database for the duration of the test, so nothing reads real data and nothing depends on the machine running it. This is exactly what hermetic testing is for: a test environment that is “entirely self-contained… no external dependencies,” so a pass or fail means the same thing on every machine (Software Engineering at Google: Continuous Integration). If your CI has never failed on a clean checkout, that’s worth treating as a gap, not a good sign.
Models occasionally emit noisy whitespace inside the values you validate. If your structured output is parsed into a strict schema, watch for a model streaming a value like "r\nhr" instead of "rhr", a raw newline landing mid-token. It happens rarely enough to pass local testing and often enough to break a real user’s request. Strip whitespace from enum-like fields before validating them, not just at the edges of free text.
Sources
- Anthropic: Demystifying evals for AI agents — groundedness and coverage checks for turning fuzzy quality judgments into concrete pass/fail outcomes.
- Python docs: typing.Literal — a Literal type indicates the annotated value is equivalent to one of the provided literals.
- alexwlchan: Use typing.get_args() to get a list of typing.Literal[…] values — get_args() on a Literal returns a tuple of its allowed values at runtime.
- Software Engineering at Google: Continuous Integration — hermetic tests run against a self-contained environment with no external dependencies, which is what makes their results deterministic.