Nate Swenson

Proving nothing changed, when the build never repeats itself

Press · No. 088 · 8 min read ·v0.5.0

Shipped

press 0.5.0 migrated the last hand-maintained copy of a shared brand, in a repo that generates a GitHub profile README from SVG tiles. Its header had named another repository’s stylesheet as canonical with nothing keeping the two honest, which made it the ninth copy and the last one. Every value came out identical; the only content change was a comment header becoming a generated region marker.

Verifying that was the interesting part. Those SVG tiles embed subsetted WOFF2, and font subsetting is not byte-reproducible: rebuilding the untouched repository produced different base64 payloads. A straight byte comparison would have failed every time while proving nothing at all. Parity was confirmed instead by stripping the font payloads and comparing everything else, giving 7 of 7 SVGs and the generated README identical.

That problem is not really about fonts. The Reproducible Builds project defines the property you want as: “Given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.” Most builds do not have it, and their catalogue of causes is a list of things you probably ship: embedded modification times, build-id sections, absolute paths baked into output, “factors that are hard or impossible to control like the ordering of files on a filesystem or the current time.” The usual response is to give up on byte comparison entirely and eyeball the output. There’s a better move.

Two bad options, and the one worth taking

Compare raw bytes. Fails on every run for reasons that have nothing to do with your change. You learn to ignore it, which is the same as not having it.

Delete the volatile part and compare the rest. Clean diffs forever, including for a change that genuinely broke the payload. This is the tempting one, and it converts your verification into a decorative one. If your normalizer removes a font, it will happily tell you two documents match when one of them has the wrong font.

Canonicalize. Replace the volatile bytes with a stable fingerprint of what they mean. The diff stays quiet across reruns, and still goes loud if the payload’s real content moves. The rest of this guide builds that, plus the controls that prove it works.

Everything here is Python’s standard library.

A build that can’t repeat itself

First, something to verify. This emits an SVG tile with a “subsetted font” inlined as base64, standing in for any build that embeds a compiled asset. Save it as build.py:

#!/usr/bin/env python3
"""Build one SVG tile with a subsetted font embedded as base64.

Stands in for any build that inlines a compiled asset: a font subsetter, a
bundler, an image optimizer. The thing they have in common is that the payload
carries per-run metadata, so identical inputs do not produce identical bytes.
"""
import base64
import gzip
import json
import sys
import uuid
from pathlib import Path

GLYPHS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"


def subset_font(glyphs):
    """Every real subsetter stamps a per-run tag into its output."""
    tag = uuid.uuid4().hex[:6].upper()
    payload = f"SUBSET-TAG:{tag}\nGLYPHS:{glyphs}\n".encode()
    return base64.b64encode(gzip.compress(payload)).decode()


def build(tokens, out):
    font = subset_font(GLYPHS)
    Path(out).write_text(f"""<svg xmlns="http://www.w3.org/2000/svg" width="400" height="120">
  <style>
    @font-face {{ font-family: Tile; src: url(data:font/woff2;base64,{font}); }}
    .bg {{ fill: {tokens["paper"]}; }}
    .fg {{ fill: {tokens["ink"]}; font-family: Tile; font-size: 24px; }}
    .rule {{ stroke: {tokens["accent"]}; stroke-width: 3; }}
  </style>
  <rect class="bg" width="400" height="120"/>
  <text class="fg" x="24" y="56">Nate Swenson</text>
  <line class="rule" x1="24" y1="80" x2="376" y2="80"/>
</svg>
""")


if __name__ == "__main__":
    build(json.loads(Path("tokens.json").read_text()), sys.argv[1])
    print(f"built {sys.argv[1]}")

The per-run tag is not invented for the demo. Real subsetters emit exactly this: a six-character prefix identifying the subset, which is why pdffonts reports the font name “exactly as given in the PDF file (potentially including a subset prefix)”, producing entries like WMHEWB+Helvetica-Neue-Bold.

Canonicalize the payload

The whole design is in fingerprint(). Save this as canonicalize.py:

#!/usr/bin/env python3
"""Make a non-reproducible artifact comparable, without making the check vacuous."""
import base64
import gzip
import hashlib
import re
import sys
from pathlib import Path

PAYLOAD_RE = re.compile(r"base64,([A-Za-z0-9+/=]+)\)")
# The one field the subsetter stamps per run. Everything else in the payload is
# real content and must still be compared.
VOLATILE_LINE = re.compile(rb"^SUBSET-TAG:[0-9A-F]+$", re.MULTILINE)


def fingerprint(b64):
    """A stable identity for a payload whose bytes are not stable.

    Decode it, drop ONLY the field known to vary per run, and hash what is
    left. Deleting the whole payload would also make the diff clean, and would
    make it clean for a genuinely changed font too.
    """
    raw = gzip.decompress(base64.b64decode(b64))
    stable = VOLATILE_LINE.sub(b"", raw)
    return hashlib.sha256(stable).hexdigest()[:16]


def canonicalize(text):
    return PAYLOAD_RE.sub(lambda m: f"base64,<payload sha256:{fingerprint(m.group(1))}>)", text)


if __name__ == "__main__":
    a, b = Path(sys.argv[1]), Path(sys.argv[2])
    raw_same = a.read_bytes() == b.read_bytes()
    ca, cb = canonicalize(a.read_text()), canonicalize(b.read_text())

    print(f"raw bytes identical:        {raw_same}")
    print(f"canonical form identical:   {ca == cb}")
    if ca != cb:
        for la, lb in zip(ca.splitlines(), cb.splitlines()):
            if la != lb:
                print(f"  - {la.strip()}")
                print(f"  + {lb.strip()}")
    raise SystemExit(0 if ca == cb else 1)

Note what VOLATILE_LINE matches: one named field, anchored, with a known shape. It is not .* over the payload. The narrower that pattern, the more of the artifact your comparison is still actually checking.

Run it

Build the same input twice and compare:

cat > tokens.json <<'EOF'
{ "paper": "#F5F0E6", "ink": "#181510", "accent": "#E8501F" }
EOF

python3 build.py before.svg
python3 build.py after.svg
python3 canonicalize.py before.svg after.svg; echo "exit: $?"

built before.svg
built after.svg
raw bytes identical:        False
canonical form identical:   True
exit: 0

Identical inputs, different bytes, and a canonical form that matches. That’s the whole mechanism, and on its own it is worth nothing, because a normalizer that deleted the payload outright would print exactly the same thing.

Prove the check can still fail

Two controls, and you want both. The first changes something outside the payload:

cat > tokens.json <<'EOF'
{ "paper": "#F5F0E6", "ink": "#181510", "accent": "#FF6B35" }
EOF
python3 build.py after.svg > /dev/null
python3 canonicalize.py before.svg after.svg; echo "exit: $?"

raw bytes identical:        False
canonical form identical:   False
  - .rule { stroke: #E8501F; stroke-width: 3; }
  + .rule { stroke: #FF6B35; stroke-width: 3; }
exit: 1

The second is the one people skip, and it’s the one that matters: change the payload’s own content and confirm the fingerprint notices.

cat > tokens.json <<'EOF'
{ "paper": "#F5F0E6", "ink": "#181510", "accent": "#E8501F" }
EOF
sed -i '' 's/GLYPHS = "ABCDEF/GLYPHS = "BCDEF/' build.py   # GNU sed: drop the ''
python3 build.py after.svg > /dev/null
python3 canonicalize.py before.svg after.svg; echo "exit: $?"

raw bytes identical:        False
canonical form identical:   False
  - @font-face { font-family: Tile; src: url(data:font/woff2;base64,<payload sha256:b807596b4028f1e9>); }
  + @font-face { font-family: Tile; src: url(data:font/woff2;base64,<payload sha256:f4022b0d776b5c12>); }
exit: 1

Dropping one glyph from the subset moved the fingerprint. Had the normalizer elided the payload instead of fingerprinting it, this run would have reported a clean match, and you’d have shipped a font missing a character with a green check behind you. Run this control every time you add a normalization rule, because each rule is a new place the check can go blind.

With those two passing, “canonical form identical” across a migration means something: press could then state that 7 of 7 SVGs and the generated README were unchanged, and have that be a measurement rather than a hope.

Gotchas

A normalizer is a hole in your gate until you test it. Every pattern you add to make diffs quieter also makes some real change invisible, and the failure is silent by construction: a normalizer that hides too much produces exactly the output you were hoping for. Symptom: the comparison passes on a migration that actually broke something, and nothing anywhere reports an error. The escape is a mandatory negative control per rule, as above: change the thing the rule is supposed to still be watching, and require the comparison to fail. If you can’t construct a change that makes it fail, the rule is too broad.

Match the volatile field, not the volatile region. The lazy pattern is a wildcard across the whole blob, because it’s one line and it works immediately. Symptom: the comparison keeps passing as more and more of the artifact drifts out from under it, and you find out when a rendered page looks wrong. The escape is to anchor on the specific named field with its known shape, so anything else appearing inside that payload still counts. ^SUBSET-TAG:[0-9A-F]+$ and .* both silence the reruns; only one of them still checks the glyphs.

Not every migration can be verified the same way. press verified three consumers three different ways in adjacent releases: byte-identical rendered pages for one that goes through headless Chrome, identical embedded faces measured per engine for one that goes through WeasyPrint, and this canonical comparison for the SVG tiles. Symptom: you pick one verification style, apply it everywhere, and it silently degrades to a formality on the consumers it doesn’t suit. The escape is to ask what the artifact is actually made of before deciding what “identical” should mean for it.

Sources

Changelog

  • feat(press): python-consts + font_files, and the last copy is gone (0.5.0) (#125) (78cdef9)