The setting that fixed a slow AI brief was hiding behind one that silently did nothing
Shipped
v0.6.0 cut a Claude-generated daily fitness brief from about four minutes to roughly 80-95 seconds, with quality judged equal or better, not worse. The fix that got there wasn’t the one I designed first. I’d planned a parallel fan-out that would split the brief into sections and compose them concurrently, and it cleared a design review before I built any of it. A measurement killed it. The real fix was flipping one setting, and finding that setting meant discovering that a different, more obvious-looking setting had been silently doing nothing the whole time. Here’s how to profile a slow LLM agent, gate a rewrite behind a number you commit to in advance, and tell a real lever from a decorative one.
Prerequisites
The examples below use Anthropic’s Claude Agent SDK, which drives the Claude Code CLI as a subprocess and works against either an API key or a Claude subscription login:
npm install -g @anthropic-ai/claude-code
pip install claude-agent-sdk
claude login # or export ANTHROPIC_API_KEY=...
Everything past this point assumes that’s in place. The render_table / fix_table_row_breaks section near the end has no external dependency and is safe to run as-is.
Profile before you build the parallel version
The instinct when an LLM call is slow is to make it concurrent: split the work, run the pieces at once, assemble the result. Before writing any of that, wrap a single real call in a phase timer and read the actual split:
import time
from contextlib import contextmanager
from claude_agent_sdk import ClaudeAgentOptions, query
class PhaseTimer:
"""Accumulate wall-clock per named phase so you can see the real split."""
def __init__(self) -> None:
self.phases: dict[str, float] = {}
@contextmanager
def phase(self, name: str):
start = time.perf_counter()
try:
yield
finally:
self.phases[name] = self.phases.get(name, 0.0) + (time.perf_counter() - start)
def report(self) -> None:
total = sum(self.phases.values()) or 1.0
for name, secs in sorted(self.phases.items(), key=lambda kv: -kv[1]):
print(f"{name:14} {secs:7.1f}s {secs / total:5.1%}")
async def run_task(prompt: str, **opts) -> tuple[str, dict | None]:
"""Drive one query() to completion. Returns the visible text and the
usage payload the SDK reports on the trailing result message. Thinking
tokens are billed as output tokens even though you never see them."""
options = ClaudeAgentOptions(
model="sonnet", permission_mode="bypassPermissions", max_turns=1, **opts
)
chunks: list[str] = []
usage: dict | None = None
async for message in query(prompt=prompt, options=options):
u = getattr(message, "usage", None)
if u is not None:
usage = dict(u) if isinstance(u, dict) else getattr(u, "__dict__", {})
for block in getattr(message, "content", []) or []:
if getattr(block, "type", None) == "text":
chunks.append(block.text)
return "".join(chunks), usage
Run one generation-heavy prompt through PhaseTimer and you should see almost the entire wall-clock land inside a single generation phase, with output_tokens running far higher than the visible text would explain. That gap is thinking the model did but never showed you. In the brief this technique is drawn from, one call accounted for effectively the whole runtime, and roughly 93% of its output tokens were hidden reasoning, not the ~15 seconds of database tool calls I’d assumed were the bottleneck.
That result reframes the fan-out idea. If the cost is one call’s reasoning, splitting the work into more calls doesn’t obviously help, and it adds real complexity: a planner step, a reduce step, per-section validation. The standard advice here is to find the simplest thing that works and add complexity only when it demonstrably earns its keep (Anthropic: Building effective agents). So before writing the fan-out, gate it behind a number:
import asyncio
KILL_CRITERION = 1.7 # decide this before you measure, not after
async def concurrency_speedup(prompt_template: str, n: int = 3) -> float:
"""Compare n calls run serially vs. the same n run concurrently.
>= 1.0 means concurrency helped at all; below KILL_CRITERION means
the fan-out isn't worth the extra moving parts."""
prompts = [prompt_template.format(i=i) for i in range(n)]
t0 = time.perf_counter()
for p in prompts:
await run_task(p)
serial = time.perf_counter() - t0
t0 = time.perf_counter()
await asyncio.gather(*(run_task(p) for p in prompts))
concurrent = time.perf_counter() - t0
return serial / concurrent if concurrent else 0.0
asyncio.run(concurrency_speedup("Write one short paragraph about the number {i}.")) measures transport concurrency cheaply, with small fixed-shape prompts rather than real briefs. The pre-registered bar matters here: it’s Donald Knuth’s old warning in a newer setting, profile first, because premature optimization spends effort where the time isn’t actually going (Program optimization). The measurement this is modeled on came back at 1.44x speedup running three calls concurrently, under a 1.7x bar set in advance. The fan-out was killed before a line of production code was written for it.
Find out which knob is real
Killing the fan-out still left the actual question open: if the cost is thinking, what controls how much of it happens? The obvious answer is a thinking token budget. It’s also, on some transports, a setting that does nothing. Sweep it against the alternative before trusting either one:
_THINKING_CONFIGS = {
"adaptive": {"type": "adaptive"},
"capped-low": {"type": "enabled", "budget_tokens": 1024},
"capped-high": {"type": "enabled", "budget_tokens": 12000},
}
async def sweep_thinking_knobs(prompt: str) -> None:
"""Compare output_tokens across thinking configs AND effort levels on
the SAME prompt. Flat output_tokens across `thinking` configs but real
movement across `effort` levels means `thinking` isn't the lever here,
`effort` is."""
print(f"{'config':<16}{'output_tokens':>14}")
for name, cfg in _THINKING_CONFIGS.items():
_, usage = await run_task(prompt, thinking=cfg)
print(f"{name:<16}{str((usage or {}).get('output_tokens')):>14}")
for effort in ("low", "high"):
_, usage = await run_task(prompt, effort=effort)
print(f"{'effort='+effort:<16}{str((usage or {}).get('output_tokens')):>14}")
On the run this generalizes, budget_tokens=1024 and budget_tokens=12000 produced the same output token count on the same prompt, run through the Claude Agent SDK’s Claude Code CLI / subscription transport. The cap wasn’t rejected and didn’t error, it was just ignored. effort, by contrast, moved the number clearly. Anthropic’s own docs now say the same thing directly: on the models and transports where adaptive thinking applies, manual thinking budgets are deprecated or unsupported in favor of effort, which is described as the primary control for trading off thoroughness against token spend (Claude Agent SDK: effort) and is exposed the same way on ClaudeAgentOptions (Claude Agent SDK for Python). The lesson isn’t “always use effort instead of thinking.” It’s that a setting compiling and running without an error is not evidence it’s doing anything; the sweep is what tells you.
Ship the lever, and make the defect it introduces impossible in code
effort="low" is the fix: fewer thinking tokens spent on an output-bound task, wired straight into the same call:
async def generate_brief(snapshot: str, effort: str = "low") -> str:
prose, _usage = await run_task(f"Write today's brief:\n{snapshot}", effort=effort)
return prose
Dropping effort has one real side effect: at lower effort the model more often drops the backslash on a \n row break inside a markdown table, gluing the separator row to the first data row into one unreadable line. Re-prompting for a better sample is not a fix, because the failure is intermittent. The fix is a table renderer the model never touches, plus a narrow repair for the one corruption pattern that’s actually been observed:
import re
_SEPARATOR_RE = re.compile(r"\|\s*:?-{2,}:?\s*\|")
_COLLAPSED_ROW_RE = re.compile(r"\|n\|")
def render_table(headers: list[str], rows: list[list[str]]) -> str:
def _cell(v: object) -> str:
return str(v).replace("|", r"\|").strip()
head = "| " + " | ".join(_cell(h) for h in headers) + " |"
sep = "| " + " | ".join("---" for _ in headers) + " |"
lines = [head, sep]
for r in rows:
lines.append("| " + " | ".join(_cell(c) for c in r) + " |")
return "\n".join(lines)
def fix_table_row_breaks(text: str) -> str:
"""No-op unless `text` actually contains a markdown table (a separator
row present), so ordinary prose is never touched."""
if not text or "|" not in text:
return text
if not _SEPARATOR_RE.search(text):
return text
return _COLLAPSED_ROW_RE.sub("|\n|", text)
That’s runnable as-is, no API access needed. Paste it into a file and run it:
table = render_table(
["Metric", "Today", "7-day avg"],
[["Resting HR", "52 bpm", "54 bpm"], ["Sleep", "7h 40m", "7h 12m"]],
)
print(table)
broken = "Recovery read:\n| Metric | Value |\n|--------|-------|n| RHR | 52 bpm |"
print(fix_table_row_breaks(broken))
| Metric | Today | 7-day avg |
| --- | --- | --- |
| Resting HR | 52 bpm | 54 bpm |
| Sleep | 7h 40m | 7h 12m |
Recovery read:
| Metric | Value |
|--------|-------|
| RHR | 52 bpm |
The metrics table renders clean every time because code owns it. The repair only fires when the text already contains a real table (a separator row), so ordinary prose with a stray | is never touched, and running it twice on already-fixed text is a no-op.
Verify the trade with a blind judge
Faith in a setting isn’t evidence. A separate judge call, in a fresh context, sees only the finished text and never learns which effort level produced it:
import json
JUDGE_PROMPT = (
'Score this daily briefing 1-5 on specificity and on how well it reads '
'like a real coach, not a template. Return JSON: {"specificity": N, "voice": N}.'
)
async def score(text: str) -> dict:
raw, _usage = await run_task(f"{JUDGE_PROMPT}\n\n{text}", effort="low")
return json.loads(raw)
async def blind_ab(snapshot: str, efforts: tuple[str, ...] = ("low", "high")) -> dict[str, int]:
wins = {e: 0 for e in efforts}
briefs = {e: await generate_brief(snapshot, effort=e) for e in efforts}
scored = {e: sum((await score(briefs[e])).values()) for e in efforts}
best = max(scored.values())
winner = min([e for e, s in scored.items() if s == best], key=list(efforts).index)
wins[winner] += 1
return wins
Run this across enough snapshots and count wins per effort level; wins["low"] >= wins["high"] is the bar a lower-effort setting has to clear before it ships. On the brief this generalizes, low-effort briefs scored at least as well as the prior default on every dimension the judge was asked about.
Trust that result directionally, not absolutely. A same-family judge scoring same-family output carries a documented self-preference risk, and LLM judges independently skew toward longer responses regardless of whether the extra length is earned (Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge). Drop any pair where either arm got truncated before it finished, since a cut-off response will lose on length alone and that measures truncation, not quality. And treat a same-model judge as one input, not a verdict; a second, differently-sourced judge is the way to firm that up before leaning on it hard.
Gotchas
A thinking token budget that compiles and runs without an error can still be inert. On the Claude Agent SDK’s Claude Code CLI transport, thinking={"type": "enabled", "budget_tokens": N} at 1024 and at 12000 produced identical output token counts on the same prompt, no exception, no warning. The only way to catch that is the sweep in the second section above, comparing against a control you know moves the number.
Lower effort makes the model more likely to drop the backslash on a \n inside a markdown table, collapsing the separator row and the first data row into one line (|---|---|n| RHR | ...). Re-running the prompt and hoping for a cleaner sample doesn’t fix an intermittent failure; a narrow, table-only repair applied at the save boundary does, because it can’t touch prose that was never broken.
A blind A/B against your own prior default is a real check, but only if you control for truncation and are honest about who’s judging. If a same-family model judges its own family’s output, both verbosity bias and self-preference bias push toward the newer or wordier arm looking better than it is; drop truncated samples before scoring, and don’t treat a single same-family judge’s verdict as final.
Sources
- Claude Agent SDK: effort — what the effort parameter controls, its levels, and where it now replaces manual thinking budgets.
- Claude Agent SDK for Python —
ClaudeAgentOptionsfields foreffort,thinking, and streaming. - Program optimization (Knuth) — profile before you optimize; premature optimization spends effort where the time isn’t going.
- Anthropic: Building effective agents — start simple, add complexity only when it demonstrably helps.
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge — self-preference and verbosity bias in LLM-judge evaluation.