Score a prompt file like code, and give its glue full test coverage
Shipped
This release made the ghostwriter skill hold itself to the bar any other code gets. scripts/score_skill.py checks the skill’s SKILL.md against grounded pass/fail rules and exits non-zero if one fails. A pytest suite drives all five of the skill’s Python scripts to 100% line coverage, enforced by --cov-fail-under=100 in pyproject.toml. The one bash script gets a shellcheck lint instead, since Python’s coverage tool has nothing to say about shell. All three run in CI on every push and pull request, and the release itself is the skill’s first: a version field, a changelog entry, a tag. None of that is interesting by itself. What’s worth teaching is the two techniques underneath it: turning a markdown instruction file into something a script can grade, and getting honest coverage on automation glue that talks to a network API and an optional dependency, without either one running for real in CI.
Set up: decide what “correct” means for a markdown file
A SKILL.md is a folder’s worth of instructions with a defined shape: frontmatter that must contain name and description, plus a body Claude reads for guidance (Anthropic: Equipping agents for the real world with Agent Skills). Nothing compiles that file, so nothing tells you when an edit quietly drops the line that says “never publish without approval.” The fix is to write the required shape down as literal, checkable predicates instead of trusting a human to notice. Split the frontmatter from the body, then check each requirement as a boolean over that text.
# score_skill.py
from __future__ import annotations
import re
import sys
from pathlib import Path
def split_frontmatter(text: str) -> tuple[dict, str]:
"""Naive YAML: key: value lines between the leading --- fences."""
if not text.startswith("---"):
return {}, text
parts = text.split("---", 2)
if len(parts) < 3:
return {}, text # no closing fence; treat as unparseable
front_raw, body = parts[1], parts[2]
front = {}
for line in front_raw.splitlines():
if ":" in line and not line.lstrip().startswith("#"):
key, _, val = line.partition(":")
front[key.strip()] = val.strip()
return front, body
def build_checks(front: dict, body: str) -> list[tuple[str, bool]]:
low = body.lower()
return [
("frontmatter has a name", bool(front.get("name"))),
("frontmatter has a description", bool(front.get("description"))),
("frontmatter has a version", bool(front.get("version"))),
("states the never-publish-without-approval guardrail",
"without explicit approval" in low),
("declares its operating modes", "## mode:" in low),
]
def score(path: Path) -> int:
if not path.exists():
sys.exit(f"ERROR: {path} not found.")
front, body = split_frontmatter(path.read_text(encoding="utf-8"))
checks = build_checks(front, body)
passed = 0
for desc, ok in checks:
print(f" [{'PASS' if ok else 'FAIL'}] {desc}")
passed += ok
print(f"\nScore: {passed}/{len(checks)}")
return 0 if passed == len(checks) else 1
if __name__ == "__main__":
raise SystemExit(score(Path(sys.argv[1] if len(sys.argv) > 1 else "SKILL.md")))
Five checks here for space; the real scorer runs eight, including a length cap on the description and a check that the voice-input files are named. The shape carries: each requirement is a phrase or a key you can search for, and searching for it is a test.
Test the glue at its network boundary
The scripts around the skill are almost pure glue. One of them posts text to an API. Glue like that is what people mean when they say “you can’t unit-test this,” and the honest fix is a test double placed exactly at the boundary where your code meets the outside world, not scattered through the logic (Martin Fowler: Test Coverage makes the same point about where testing effort should go). Keep the network client stdlib-only, and the boundary is a single function: urllib.request.urlopen.
# publish.py: stdlib only, no pip install needed for the glue itself
import json
import urllib.error
import urllib.request
API = "https://api.example.com/v1/posts"
def post_update(token: str, text: str) -> str:
"""POST text to the API and return the new post id."""
body = json.dumps({"text": text}).encode("utf-8")
req = urllib.request.Request(API, data=body, method="POST")
req.add_header("Authorization", f"Bearer {token}")
req.add_header("Content-Type", "application/json")
try:
with urllib.request.urlopen(req) as resp:
data = json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as e:
raise SystemExit(f"ERROR: API returned HTTP {e.code}") from e
return data["id"]
# test_publish.py
import io
import json
import urllib.error
import pytest
import publish
class FakeResponse:
"""Stands in for the context manager urlopen() returns."""
def __init__(self, body: bytes):
self._body = body
def __enter__(self):
return self
def __exit__(self, *a):
return False
def read(self):
return self._body
def test_post_update_returns_new_id(monkeypatch):
body = json.dumps({"id": "42"}).encode()
monkeypatch.setattr(publish.urllib.request, "urlopen", lambda req: FakeResponse(body))
assert publish.post_update("tok", "hello") == "42"
def test_post_update_raises_on_http_error(monkeypatch):
def boom(req):
raise urllib.error.HTTPError("url", 401, "unauthorized", {}, io.BytesIO(b""))
monkeypatch.setattr(publish.urllib.request, "urlopen", boom)
with pytest.raises(SystemExit):
publish.post_update("bad", "hello")
Two tests, both branches, no real network call. monkeypatch.setattr(publish.urllib.request, "urlopen", ...) works here because publish.py does import urllib.request and looks up urlopen as a live attribute on that module every time it’s called, so patching the attribute on the module object reaches it no matter how it’s referenced. pytest’s own guidance is to patch the reference the code under test looks up (pytest: How to monkeypatch/mock modules and environments), and Python’s own mock docs put it more sharply: you patch where a name is looked up, not where it’s defined (Python docs: unittest.mock, “Where to patch”). That distinction barely matters for a module-level attribute like urlopen. It matters a lot for the next case.
Test the glue when the dependency is optional
The skill’s diagram feature renders HTML to a PNG with Playwright, and Playwright is heavy enough that most of the skill’s users never install it. So the import lives inside the function, not at the top of the file, and only runs when you call it.
# render.py
def render_png(html: str, out_path: str) -> None:
"""Screenshot HTML to a PNG. Playwright is imported lazily, on first use."""
try:
from playwright.sync_api import sync_playwright
except ModuleNotFoundError:
raise SystemExit("Playwright isn't installed. Run: pip install playwright")
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.set_content(html)
page.screenshot(path=out_path)
browser.close()
Test this the way you tested publish.py and it breaks. There is no render.playwright attribute to patch before render_png runs, because the module has never imported playwright at all until that line executes inside the function body. monkeypatch.setattr needs an existing attribute to replace; here there isn’t one yet. The fix is to plant the fake one level up, in sys.modules itself, so when the lazy from playwright.sync_api import sync_playwright runs, Python’s import system finds your fake already sitting there and never touches the real package. pytest’s monkeypatch fixture supports exactly this through setitem (pytest: How to monkeypatch/mock modules and environments).
# test_render.py
import sys
import types
import render
def install_fake_playwright(monkeypatch, tmp_path):
page = types.SimpleNamespace(
set_content=lambda html: None,
screenshot=lambda path: (tmp_path / "marker").write_text("rendered"),
)
browser = types.SimpleNamespace(new_page=lambda: page, close=lambda: None)
chromium = types.SimpleNamespace(launch=lambda: browser)
class FakeSyncPlaywright:
def __enter__(self):
return types.SimpleNamespace(chromium=chromium)
def __exit__(self, *a):
return False
fake = types.ModuleType("playwright.sync_api")
fake.sync_playwright = FakeSyncPlaywright
# Plant the fakes in sys.modules; the lazy import inside render_png resolves
# to these instead of touching the real (probably uninstalled) package.
monkeypatch.setitem(sys.modules, "playwright", types.ModuleType("playwright"))
monkeypatch.setitem(sys.modules, "playwright.sync_api", fake)
def test_render_png_calls_through_to_screenshot(monkeypatch, tmp_path):
install_fake_playwright(monkeypatch, tmp_path)
render.render_png("<div>hi</div>", str(tmp_path / "out.png"))
assert (tmp_path / "marker").read_text() == "rendered"
def test_render_png_missing_playwright(monkeypatch):
real_import = __import__
def blocked(name, *a, **k):
if name == "playwright.sync_api":
raise ModuleNotFoundError(name)
return real_import(name, *a, **k)
monkeypatch.setattr("builtins.__import__", blocked)
try:
render.render_png("<div>hi</div>", "out.png")
assert False, "expected SystemExit"
except SystemExit as e:
assert "isn't installed" in str(e)
The second test is the missing-dependency branch: block just that one import and Playwright never has to be uninstalled from the machine running the suite to prove the fallback works.
Use it: run the gate, then break it on purpose
Two commands, and the second one is the point: this only counts as a gate if failing it fails something.
$ pytest --cov=publish --cov=render --cov-report=term-missing --cov-fail-under=100
.... [100%]
Name Stmts Miss Cover
---------------------------------
publish.py 15 0 100%
render.py 11 0 100%
---------------------------------
TOTAL 26 0 100%
4 passed in 0.03s
$ sed -i '/without explicit approval/d' SKILL.md
$ python score_skill.py SKILL.md
[PASS] frontmatter has a name
[PASS] frontmatter has a description
[PASS] frontmatter has a version
[FAIL] states the never-publish-without-approval guardrail
[PASS] declares its operating modes
Score: 4/5
$ echo $?
1
Delete the guardrail line and the exit code flips from 0 to 1. That flip is the entire mechanism; a CI step that runs score_skill.py and checks the exit code is what stops that edit from merging.
Gotchas
A dependency imported inside a function has nothing to patch until it runs. monkeypatch.setattr needs an attribute that already exists. A top-of-file import playwright gives you one; a lazy from playwright.sync_api import sync_playwright inside the function does not, since the name doesn’t exist in any namespace until that line executes. The fix is monkeypatch.setitem(sys.modules, ...), planting the fake ahead of the import instead of trying to overwrite something that isn’t there yet. Skip this and the “missing dependency” and “dependency present” branches both quietly go untested, or your test process ends up needing the real heavy package installed after all, which defeats the reason it was lazy in the first place.
A malformed frontmatter block parses as “no frontmatter” rather than “broken frontmatter.” split_frontmatter above returns {}, text whenever the file doesn’t have two --- fences, which includes a file where someone typo’d the closing fence. Every check then fails identically, and the printed output reads like the file is missing five separate things instead of one broken line at the top. Worth a dedicated test for that exact input, so you know ahead of time what a broken file reports instead of finding out mid-incident.
100% line coverage is a floor, not a quality score. It proves every line ran under a test; it says nothing about whether that test asserted anything real. Martin Fowler’s point stands: coverage is useful for finding code with zero tests, and of little use as a number describing how good those tests are, since 100% is reachable by exercising a line without checking what it did (Martin Fowler: Test Coverage). --cov-fail-under=100 catches the first kind of gap. It does not catch a test that calls post_update and never checks what it returned.
Sources
- Anthropic: Equipping agents for the real world with Agent Skills — a skill is a folder with a
SKILL.mdwhose frontmatter must declarenameanddescription. - pytest: How to monkeypatch/mock modules and environments —
setattrandsetitemon the monkeypatch fixture, and patching the reference the code under test uses. - Python docs: unittest.mock, “Where to patch” — patch the name where it’s looked up, not where it’s defined.
- Martin Fowler: Test Coverage — coverage finds untested code but is a poor measure of test quality on its own.