Five places agreed on the wrong default
Shipped
This release changed one default in a posting skill: a single post is now the form, and a thread is drafted only when the user asks for one. The edit that took was not a new rule. It was deleting the old rule from four other places, because five separate places in the instruction set each decided the form, and every one of them reached for a thread as soon as the material had more than one beat.
If you write instructions for an LLM agent, in a SKILL.md, a system prompt, a CLAUDE.md, or a stack of markdown files it reads together, you have this problem and probably cannot see it. This is how to look.
The effective default is a vote count
Two things make a scattered restatement beat a single edited rule.
The first is position. Liu and colleagues measured retrieval accuracy as a function of where the relevant text sits in a long input and found performance highest when it appears at the beginning or the end, degrading in the middle, a result they titled Lost in the Middle. A carefully worded rule in the middle of a long instruction file is not competing on equal footing with a restatement near an edge.
The second is stranger, and it is the one that explains why declaring precedence does not save you. One of the files in this skill states outright that it is subordinate to the voice notes, and it still kept reinstating the thread default. That matches what Control Illusion reports: system and user prompt separation does not reliably establish an instruction hierarchy, and framings like authority, expertise and consensus influenced behavior more than the declared roles did. Four files agreeing is a consensus. A line saying “this file is subordinate” is a role claim. The consensus wins.
Anthropic’s own prompting guidance points the same direction from the practical side: when a prompt mixes instructions, context and examples, wrapping each kind of content separately reduces misinterpretation. Separation is a real lever. A default restated in five voices is the opposite of separation.
Make a small instruction set that has the bug
Work on something you can see all of before pointing this at your real prompt.
mkdir -p instructions/voice
cat > instructions/SKILL.md <<'EOF'
# Posting skill
## Step 4 — pick the format
If the material has more than one beat, draft it as a thread.
## How-to shape
Put the setup in tweet 1 and the steps across tweets 2-5.
## Engagement craft
Threads earn more saves than a single post, so prefer a thread when in doubt.
EOF
cat > instructions/idea-menu.md <<'EOF'
# Idea menu
Each option previews as a thread outline so the user can see the shape.
EOF
cat > instructions/voice/notes.md <<'EOF'
# Voice notes
Default to a single tweet. Draft a thread only when the user asks for one.
EOF
Four files, one intended rule, and the rule is outnumbered. Nothing here is a bug in any single file, which is exactly why reading them one at a time does not find it.
Audit every place the behavior gets decided
The tool is small. Give it the competing options and the words that vote for each, and it tells you where the decisions live.
// decisions.mjs — find every place your instruction set decides one behavior.
import { readdirSync, readFileSync, statSync } from "node:fs";
import { join, relative } from "node:path";
/** Every markdown file under `root`, recursively. */
export function instructionFiles(root) {
const out = [];
const walk = (dir) => {
for (const name of readdirSync(dir)) {
const full = join(dir, name);
if (statSync(full).isDirectory()) walk(full);
else if (name.endsWith(".md")) out.push(full);
}
};
walk(root);
return out.sort();
}
/**
* Scan for lines that express a preference about one behavior.
*
* `options` maps an option name to the patterns that vote for it, e.g.
* { thread: [/thread/i], single: [/single (tweet|post)/i] }
* A line can vote for more than one option; that is worth seeing, not hiding.
*/
export function findDecisions(root, options) {
const hits = [];
for (const file of instructionFiles(root)) {
const lines = readFileSync(file, "utf8").split("\n");
lines.forEach((line, i) => {
for (const [option, patterns] of Object.entries(options)) {
if (patterns.some((p) => p.test(line))) {
hits.push({
file: relative(root, file),
line: i + 1,
option,
text: line.trim().slice(0, 88),
});
}
}
});
}
return hits;
}
/** How many places vote for each option. */
export function tally(hits) {
const counts = {};
for (const h of hits) counts[h.option] = (counts[h.option] ?? 0) + 1;
return counts;
}
if (import.meta.url === `file://${process.argv[1]}`) {
const root = process.argv[2] ?? "instructions";
const OPTIONS = {
thread: [/\bthread(s|ed)?\b/i, /tweets? 2\s*[-–]\s*\d/i],
single: [/\bsingle (tweet|post)\b/i, /\bone tweet\b/i],
};
const hits = findDecisions(root, OPTIONS);
const counts = tally(hits);
console.log(`decision points for "form": ${JSON.stringify(counts)}`);
for (const h of hits) {
console.log(` ${h.option.padEnd(7)} ${h.file}:${h.line} ${h.text}`);
}
}
node decisions.mjs instructions
decision points for "form": {"thread":5,"single":2}
thread SKILL.md:4 If the material has more than one beat, draft it as a thread.
thread SKILL.md:7 Put the setup in tweet 1 and the steps across tweets 2-5.
thread SKILL.md:10 Threads earn more saves than a single post, so prefer a thread when in doubt.
single SKILL.md:10 Threads earn more saves than a single post, so prefer a thread when in doubt.
thread idea-menu.md:2 Each option previews as a thread outline so the user can see the shape.
thread voice/notes.md:2 Default to a single tweet. Draft a thread only when the user asks for one.
single voice/notes.md:2 Default to a single tweet. Draft a thread only when the user asks for one.
Five votes for a thread against two for a single post, and note the two lines that vote both ways. SKILL.md:10 argues for threads while naming single posts, and the voice rule names threads while forbidding them by default. Those double-voting lines are the ones that reread as innocent, because each contains the correct instruction somewhere inside it.
Pin the count so it cannot grow back
Finding the five places once is the easy half. The reason a default drifts is that a sixth mention gets added later, in good faith, by someone documenting a related thing. So nominate exactly one owner for the behavior and make every other mention a test failure.
// decisions.test.mjs — one behavior, one place allowed to decide it.
import assert from "node:assert/strict";
import { findDecisions } from "./decisions.mjs";
const ROOT = process.argv[2] ?? "instructions";
// The behavior under audit, and the competing options it can resolve to.
const FORM = {
thread: [/\bthread(s|ed)?\b/i, /tweets? 2\s*[-–]\s*\d/i],
single: [/\bsingle (tweet|post)\b/i, /\bone tweet\b/i],
};
// The single file that is allowed to decide this. Everything else must be
// silent about it, or the effective default becomes a vote count.
const OWNER = "voice/notes.md";
const strays = findDecisions(ROOT, FORM)
.filter((h) => h.file !== OWNER)
.map((h) => `${h.file}:${h.line} (${h.option}) ${h.text}`);
assert.deepEqual(
strays,
[],
`${strays.length} place(s) outside ${OWNER} decide this behavior:\n ` + strays.join("\n "),
);
console.log(`ok only ${OWNER} decides form`);
Run it against the instruction set you just made and it goes red, naming every stray:
AssertionError [ERR_ASSERTION]: 5 place(s) outside voice/notes.md decide this behavior:
SKILL.md:4 (thread) If the material has more than one beat, draft it as a thread.
SKILL.md:7 (thread) Put the setup in tweet 1 and the steps across tweets 2-5.
SKILL.md:10 (thread) Threads earn more saves than a single post, so prefer a thread when in doubt.
SKILL.md:10 (single) Threads earn more saves than a single post, so prefer a thread when in doubt.
idea-menu.md:2 (thread) Each option previews as a thread outline so the user can see the shape.
Now rewrite the strays so they defer instead of deciding. SKILL.md’s format step points at the owner, the how-to shape stops describing a tweet range, the reach note says explicitly that it cannot select the form, and the idea menu previews a hook rather than a thread outline. Re-run:
ok only voice/notes.md decides form
That green line is the useful artifact. It is not proof the agent behaves correctly; it is proof that exactly one file gets a say, which is the precondition for the rule you wrote actually being the rule that runs.
Gotchas
Deleting the competing mentions is the fix; adding a stronger rule is not. The instinct on discovering a wrong default is to write a firmer version of the correct rule, which leaves the vote at five to three. Symptom: you tighten the wording, rerun, and the agent still does the old thing. Escape: audit first, then remove, and only reword the survivor if it is genuinely unclear.
A file declaring itself subordinate does not make it subordinate. The reach-guidance file in this skill said in its own text that the voice notes take priority, and it was still the remaining path by which the thread default came back. That is the exact effect Control Illusion measured. Escape: treat a precedence claim written in prose as documentation, not as a mechanism, and enforce ownership with a check outside the files.
The pattern list is where this check quietly rots. It matches words, so a restatement that avoids your vocabulary sails through, and a synonym you never listed reads as silence. Symptom: the test is green and the behavior is still wrong. Escape: when you find a stray the check missed, add its phrasing to the option’s patterns in the same change, so the check grows with the ways people actually write.
Double-voting lines are the ones to read twice. Two lines in the sample vote for both options because each names the losing form while arguing for the winning one. Those survive review because they contain the right words. Escape: sort your audit output by line rather than by option, so a line appearing under two options is visible instead of scattered.
Sources
- Lost in the Middle — retrieval accuracy is highest at the beginning and end of a long input and degrades in the middle
- Control Illusion: The Failure of Instruction Hierarchies in Large Language Models — declared system/user roles do not reliably establish precedence, and consensus framing outweighs them
- Anthropic prompting best practices — separating instructions, context and examples reduces misinterpretation