Nate Swenson

Zero of four consumers updated, and the job went green

Press · No. 089 · 8 min read ·v0.6.1

Shipped

press 0.6.1 fixed two failures in the job that pushes a brand change out to every consumer repository, and the second one was worse than the first.

The first: gh authenticates its own API calls from GH_TOKEN, so cloning each consumer and reading its default branch worked fine. A plain git push does not use GH_TOKEN. It has no credential helper, so it asks for a username and dies on a runner. All four consumers failed that way on the 0.6.0 release.

The second: that failure had been deliberately downgraded to a warning, on the reasoning that the release itself had already happened and the weekly run would retry. So the job reported success while propagating to nobody. This guide is about building the loop that can’t do that, and the test that keeps it honest.

The half-authenticated runner

This one is worth understanding rather than just pasting the fix, because the symptom is so lopsided. Give a job a GH_TOKEN and most of what you do with GitHub works immediately: gh repo clone, gh api, gh pr create. All of those are gh talking to the API with the token it was handed.

git is a different program. It doesn’t read GH_TOKEN, and an HTTPS remote with no credential helper falls back to prompting, which on a runner means failing with could not read Username for 'https://github.com'. The fix is one line, because the GitHub CLI ships a command whose entire job is bridging exactly this gap: gh auth setup-git “configures git to use GitHub CLI as a credential helper.”

- run: |
    gh auth setup-git      # without this, every push fails and everything else works
    ./fanout.sh
  env:
    GH_TOKEN: ${{ secrets.PROPAGATE_TOKEN }}

A stand-in for the push

To build the loop without a real runner, here’s a script that fails the same way. Save it as push_to.sh and chmod +x it:

#!/usr/bin/env bash
# Stands in for `git push` inside a fan-out. It fails exactly the way a real
# one does on a runner where no credential helper has been configured.
repo="$1"
if [ -z "${GIT_CREDENTIAL_HELPER:-}" ]; then
  echo "fatal: could not read Username for 'https://github.com': No such device or address" >&2
  exit 128
fi
echo "pushed brand update to $repo"

The fan-out loop

A fan-out has a requirement that fights the usual shell advice: it must visit every consumer, so it cannot use set -e to abort on the first failure. That’s reasonable, and it’s also the exact structure that lets failures pile up unnoticed, because once you’re catching each error you have to remember to do something with them. Save this as fanout.sh:

#!/usr/bin/env bash
# NOT `set -e`: a fan-out has to visit every consumer, so each failure is
# captured and counted rather than aborting the run at the first one. That is
# also exactly how a fan-out ends up reporting success while doing nothing.
set -uo pipefail

CONSUMERS="site report profile docs"
mkdir -p logs
: > results.txt
failures=0
total=0

for repo in $CONSUMERS; do
  total=$((total + 1))
  if ./push_to.sh "$repo" > "logs/$repo.log" 2>&1; then
    printf '%-9s ok\n' "$repo" >> results.txt
  else
    printf '%-9s FAILED   %s\n' "$repo" "$(tail -n1 "logs/$repo.log")" >> results.txt
    failures=$((failures + 1))
  fi
done

cat results.txt
echo
echo "$((total - failures))/$total propagated"

# An `::error::` annotation is not a failure; GitHub decides by exit code.
# So the count has to actually become one.
if [ "$failures" -ne 0 ]; then
  echo "::error::$failures of $total consumer(s) could not be propagated"
  exit 1
fi
echo "All consumers propagated."

Two details carry the weight. The n/total propagated line means a human skimming the log sees the denominator, so “0/4” is legible at a glance in a way that four separate error lines are not. And the exit 1 is not decoration: GitHub’s workflow commands describe ::error:: as something that “creates an error message and prints the message to the log”, and note that core.setFailed is “used as a shortcut for ::error and exit 1”. The annotation and the failure are two separate things, and only the second one stops a release.

Run it with no credential helper in the environment:

./fanout.sh; echo "exit: $?"

site      FAILED   fatal: could not read Username for 'https://github.com': No such device or address
report    FAILED   fatal: could not read Username for 'https://github.com': No such device or address
profile   FAILED   fatal: could not read Username for 'https://github.com': No such device or address
docs      FAILED   fatal: could not read Username for 'https://github.com': No such device or address

0/4 propagated
::error::4 of 4 consumer(s) could not be propagated
exit: 1

Then with one configured, which is what gh auth setup-git accomplishes on a real runner:

GIT_CREDENTIAL_HELPER=gh ./fanout.sh; echo "exit: $?"

site      ok
report    ok
profile   ok
docs      ok

4/4 propagated
All consumers propagated.
exit: 0

Make the gate prove it can fail

Here’s the part that would have caught the real bug. The dangerous version of this script differs from the correct one by two tokens: ::warning:: instead of ::error::, and exit 0 instead of exit 1. It produces identical output on a healthy run. So test both directions. Save this as test_fanout.sh:

#!/usr/bin/env bash
# Two-sided on purpose. Asserting only that a healthy fan-out passes lets the
# gate rot the day someone turns the failure back into a warning.
TARGET="${1:-./fanout.sh}"
fail=0

if GIT_CREDENTIAL_HELPER=gh "$TARGET" > /dev/null 2>&1; then
  echo "ok    healthy fan-out exits 0"
else
  echo "FAIL  healthy fan-out should exit 0"; fail=1
fi

if "$TARGET" > /dev/null 2>&1; then
  echo "FAIL  broken fan-out exited 0; this gate cannot fail"; fail=1
else
  echo "ok    broken fan-out exits non-zero"
fi

exit $fail

Now build the bad version deliberately and point the test at both:

chmod +x test_fanout.sh
sed 's/::error::/::warning::/; s/^  exit 1$/  exit 0/' fanout.sh > fanout_warn.sh
chmod +x fanout_warn.sh

echo "--- fixed ---"
./test_fanout.sh ./fanout.sh; echo "exit: $?"
echo "--- warns only ---"
./test_fanout.sh ./fanout_warn.sh; echo "exit: $?"

--- fixed ---
ok    healthy fan-out exits 0
ok    broken fan-out exits non-zero
exit: 0
--- warns only ---
ok    healthy fan-out exits 0
FAIL  broken fan-out exited 0; this gate cannot fail
exit: 1

The warning-only version passes the first assertion and fails the second, which is precisely the shape of the regression that shipped. A test that only asserted the happy path would have been green for both.

Gotchas

A token that authenticates your CLI does not authenticate git. GH_TOKEN makes gh work, and the natural inference is that the job is authenticated. Symptom: the confusing kind, where cloning succeeds, listing pull requests succeeds, creating a pull request succeeds, and only git push fails, so it reads as a permissions problem with that one operation rather than a missing credential helper. press hit this on the 0.6.0 release, where all four consumers failed at exactly that step. The escape is gh auth setup-git before the first push, and the diagnostic is that the error mentions a username prompt rather than a 403.

A warning is not a failure, and neither is an error annotation. Downgrading a failed step to a warning always has a plausible story attached, and press’s was reasonable-sounding: the release itself already succeeded, this is only the fan-out, and the weekly run will retry. What that reasoning misses is that the green tick then asserts something false to everyone who reads it later. Symptom: nobody investigates, because there is nothing that looks like a problem, and the brand sits unpropagated until someone notices by eye. The escape is to decide what the job claims when it’s green, and make anything that falsifies that claim exit non-zero.

set -e will not save a loop you deliberately made non-fatal. The moment a fan-out wraps each step in if or || true so it can continue, every shell safety net stops applying to those steps, and this is documented rather than surprising: the Bash manual lists the exemptions, including a command that is “part of the test in an if statement” and one that is “part of any command executed in a && or || list except the command following the final && or ||”. Which is to say the exact two shapes a fan-out is built from. The only thing then standing between a total failure and a green run is an integer you remembered to check at the end. Symptom: a script that looks defensive at the top, with set -euo pipefail right there on line two, and still cannot fail where it matters. The escape is to treat the failure counter as the real exit condition and test it directly, rather than trusting the top of the file.

Sources

Changelog

  • fix(press): the fan-out could not push, and said it succeeded (0.6.1) (#131) (68de5ab)