Nate Swenson

A prompt dial isn't a setting until code reads it too

Local Fitness · No. 019 · 10 min read ·v0.10.0

Shipped

My daily training brief used to have one coaching voice, baked into the prompt: supportive when things were going well, a roast when I was slipping. v0.10.0 makes the voice a profile I pick instead: supportive, neutral, hardass, and adaptive (the old blended behavior, kept as the default). Each one is a file with a prose persona and a few numeric dials, and a loader resolves which profile is active and threads it into both the system prompt and the daily-brief prompt.

The four personas were the fast part to write. The part that took real thought was the dials. A number sitting next to a paragraph of prose is easy to fake yourself out about: you write harshness: 9, the model probably leans into it, and you call it configuration. It isn’t, not yet. Here’s the technique that makes a dial like that actually mean something, and the two-layer test setup that keeps every profile honest without spending a model call on every commit.

Two tiers of dial, sorted by whether you can check them

Before touching code, sort your dials into two groups. Some of them are calibration: numbers you hand to the model in prose and trust it to interpret directionally. “Warmth: 9/10” nudges the tone; it doesn’t guarantee anything you can assert against. Others describe behavior you can name exactly: “when the user misses a goal, say so directly.” That one has a concrete trigger and a concrete text block, which means it can be a real if statement instead of a hope.

The rule that falls out of this: a dial only counts as configuration if some code path reads it and does something different. If the only reader is the model, it’s decoration, not a control, no matter how precise the number looks.

Build the profile: a file, a loader, a fallback that can’t crash

A profile is a small markdown file: frontmatter for the dials, then the persona prose underneath. Keeping it a file instead of a string in a match statement means you can read four tones side by side and edit one without touching code.

<!-- profiles/hardass.md -->
---
name: hardass
warmth: 1
push: 9
---
You are a demanding coach who is never quite satisfied. Acknowledge effort in
one line, then push for the next level. Never pad with praise.

<!-- profiles/supportive.md -->
---
name: supportive
warmth: 9
push: 2
---
You are the user's biggest believer. Lead with what went well. Frame every
gap as the next small step, never a verdict.

The loader parses that frontmatter with a few lines of string splitting, no YAML dependency required for something this flat, and falls back to an in-code default if the file is missing or empty rather than raising:

# profile.py
"""Tone profiles for an LLM prompt: data on disk, not a match statement."""
from __future__ import annotations

from dataclasses import dataclass, replace
from pathlib import Path

PROFILE_NAMES = frozenset({"supportive", "hardass"})
DEFAULT_PROFILE = "supportive"
PUSH_GATE_MIN = 6  # push >= this is the only dial that changes the prompt


@dataclass(frozen=True)
class Profile:
    name: str
    warmth: int   # 0-10, prose calibration only; the model interprets it
    push: int     # 0-10, ALSO the deterministic gate below (see includes_escalation_block)
    persona: str  # the prose body

    @property
    def includes_escalation_block(self) -> bool:
        """The one dial the code actually acts on, not just describes."""
        return self.push >= PUSH_GATE_MIN


# In-code fallback so a missing/broken profile file never breaks a caller that
# just wants *a* profile back.
_FALLBACK = Profile(
    name="supportive",
    warmth=8,
    push=3,
    persona="Lead with what's going well. Frame a miss as the next step, never a verdict.",
)


def _parse_frontmatter(text: str) -> tuple[dict, str]:
    """Split a ``---`` fenced frontmatter block from the body. Minimal parser,
    no YAML dependency: every value is a bare string until a caller coerces it."""
    if not text.startswith("---"):
        return {}, text
    parts = text.split("---", 2)
    if len(parts) < 3:
        return {}, text
    fm_raw, body = parts[1], parts[2]
    fm: dict[str, str] = {}
    for line in fm_raw.splitlines():
        line = line.strip()
        if not line or ":" not in line:
            continue
        key, _, val = line.partition(":")
        fm[key.strip()] = val.strip()
    return fm, body.strip()


def load_profile(name: str, profile_dir: Path) -> Profile:
    """Load a profile by name. Unknown name, missing file, or empty body all
    fall back to the in-code default instead of raising."""
    name = (name or "").strip().lower()
    if name not in PROFILE_NAMES:
        name = DEFAULT_PROFILE
    path = profile_dir / f"{name}.md"
    try:
        text = path.read_text(encoding="utf-8")
    except OSError:
        return replace(_FALLBACK, name=name)
    fm, body = _parse_frontmatter(text)
    if not body:
        return replace(_FALLBACK, name=name)
    return Profile(
        name=name,
        warmth=int(fm.get("warmth", _FALLBACK.warmth)),
        push=int(fm.get("push", _FALLBACK.push)),
        persona=body,
    )

The fallback matters more than it looks like it should. If anything in your program builds a default prompt at import time (a module-level constant, say), a broken profile file on disk would otherwise take down every import of that module, including your test suite. Loading defensively here means a bad file degrades to a default persona instead of bricking the app.

Wire the falsifiable dial into the prompt

warmth goes into the prompt as prose and the model does what it does with it. push, once it crosses a threshold, decides in code whether an entire block of text is even in the prompt to begin with:

# prompt.py
"""Assembles the prompt. The escalation block is a code switch, not a hope."""
from profile import Profile

ESCALATION_BLOCK = (
    "The user missed their target. Name the gap directly and push for the "
    "next action. Do not soften a miss with unrelated good news."
)


def build_prompt(base: str, profile: Profile, target_missed: bool) -> str:
    prompt = f"{base}\n\n{profile.persona}\n\nCalibrate warmth to {profile.warmth}/10."
    if target_missed and profile.includes_escalation_block:
        prompt += "\n\n" + ESCALATION_BLOCK
    return prompt

A hardass profile with a missed target gets the escalation block concatenated into its prompt string, full stop; a supportive profile never does, regardless of what the model might have inferred from “push: 2” on its own. That’s the distinction Anthropic draws between workflows and agents: a workflow orchestrates the model through predefined code paths when you want predictability, rather than leaving the decision to the model (Anthropic: Building effective agents). The if profile.includes_escalation_block line is the workflow half of an otherwise free-form prompt, and it’s the one piece of tone-selection you don’t have to trust the model to get right.

Verify it: a scorer that costs nothing to run

Because the escalation block is plain string assembly, you can test it with plain string assertions, no model call, no API key, no flakiness:

# score_profiles.py
from pathlib import Path

from profile import load_profile
from prompt import build_prompt, ESCALATION_BLOCK

PROFILE_DIR = Path(__file__).parent / "profiles"


def check(desc: str, ok: bool) -> bool:
    print(f"  [{'PASS' if ok else 'FAIL'}] {desc}")
    return ok


def main() -> None:
    hardass = load_profile("hardass", PROFILE_DIR)
    supportive = load_profile("supportive", PROFILE_DIR)

    results = [
        check(
            "hardass includes the escalation block on a missed target",
            ESCALATION_BLOCK in build_prompt("Write today's update.", hardass, target_missed=True),
        ),
        check(
            "supportive omits the escalation block on a missed target",
            ESCALATION_BLOCK not in build_prompt("Write today's update.", supportive, target_missed=True),
        ),
        check(
            "hardass omits the escalation block when nothing was missed",
            ESCALATION_BLOCK not in build_prompt("Write today's update.", hardass, target_missed=False),
        ),
        check(
            "a missing profile directory still returns a usable profile",
            load_profile("hardass", Path("/nonexistent")).persona != "",
        ),
    ]

    passed = sum(results)
    print(f"\n{passed}/{len(results)} checks passed")
    if passed != len(results):
        raise SystemExit(1)


if __name__ == "__main__":
    main()

Put profile.py, prompt.py, score_profiles.py, and a profiles/ directory holding hardass.md and supportive.md in the same folder and run it:

$ python3 score_profiles.py
  [PASS] hardass includes the escalation block on a missed target
  [PASS] supportive omits the escalation block on a missed target
  [PASS] hardass omits the escalation block when nothing was missed
  [PASS] a missing profile directory still returns a usable profile

4/4 checks passed

That’s the whole falsifiable core of the tone system, checked on every commit for free. What it deliberately does not check is whether the model’s actual words sound harsh or supportive; that’s a different, non-deterministic question, and Anthropic’s own guidance on evals is to grade agents on the properties that matter rather than expecting an exact match on output or path (Anthropic: Demystifying evals for AI agents). If you want that layer too, keep it separate: run each profile against a fixed scenario, have a second model call read the output and guess which persona wrote it, and assert a contrast between your two most opposed tones rather than an exact label for every one; two adjacent personas produce prompts that are too close to expect a judge to always name the right one. Gate that layer behind an explicit opt-in (an env var, a --live flag) so a plain test run never spends money by accident. The general version of this split, deterministic checks for what’s measurable and an LLM judge only for the genuinely subjective remainder, is the same shape Braintrust recommends for grading LLM output generally: code-based checks for format and structure because they’re cheap and predictable, judge calls reserved for tone and helpfulness because those need language understanding (Braintrust: What is an LLM-as-a-judge?).

Gotchas

A dial with no code reader is unfalsifiable, and you won’t notice until you try to prove it. I originally described all the numeric dials the same way, as calibration the persona carries. Writing the test is what exposed that half of them had nothing to assert against; “hardass feels harsher than supportive” isn’t a test, it’s a vibe. The fix was splitting the dials by whether a code path reads them, then only testing the ones that do.

Claiming two renderings are “identical” is a trap the moment you add a shared line to both. Early in this build I wanted the default profile’s prompt to render byte-for-byte the same as the pre-change prompt, as a safety net. That stopped being true the moment a “here’s your calibration” line got added to every profile, including the default, since that line is new text the old prompt never had. The fix wasn’t to chase byte-equality: it was to assert the specific substrings and structure the deterministic scorer actually depends on, and call that equivalence what it is, rather than a claim that can’t survive the next intentional change.

A prompt built at import time will read the same file every time your program starts, so a missing or malformed file is not a runtime bug, it’s an import-time outage. If a module builds its default prompt as a top-level constant, that constant loads a profile off disk the moment the module is imported, before your code has any chance to catch a bad file. The fallback in load_profile above exists specifically so a missing or empty profile file degrades to a default persona instead of raising during import, which would otherwise take down anything that imports the module, tests included.

Sources

Changelog

  • feat: selectable coach tone profiles for the daily brief (0.10.0) (e53b005)
  • docs: per-profile A/B + quality scorer vs expected outcomes (mandatory, not optional) (fc17711)
  • docs: revise coach-tone-profiles design per quality-gate (2 rounds + look-harder, 4->0) (d9199fb)
  • docs: design for selectable coach tone profiles (supportive/neutral/hardass) (f15df80)