The empty string is a header value | Nate Swenson Dev Log

Shipped

This release came out of the first end-to-end run that published a post with an image attached. Two shipped features turned out to be broken, and the one worth writing up is image upload, which had failed for every card since the first release. The first two posts were text only, so nothing surfaced it.

The upload targets a presigned S3 URL. Every PUT came back 403 SignatureDoesNotMatch, and both obvious fixes produced the same 403.

Why a presigned PUT rejects a correct-looking request

A presigned URL is a signature over a specific request, not a general permission to write to a key. AWS describes it as granting the permissions of whoever generated it, for a limited time, which is why the request you send has to be the request that was signed. Their guidance on signature mismatch errors puts it plainly: the headers used to generate the signature must match the headers the client sends. If a Content-Type was part of the signature, your request has to carry the same value.

The word “same” is doing more work than it appears to. The signer builds a StringToSign that includes the content type it expects. When the service that generated your URL signed it with an empty content type, the value you must send is the empty string. Sending image/png is not a more correct version of nothing; it is a different string, so it produces a different StringToSign and a 403.

That gives you three cases, and only one of them works:

what you send	StringToSign gets	result
`Content-Type: image/png`	`image/png`	403 SignatureDoesNotMatch
no `Content-Type` header at all	a client-chosen default	403 SignatureDoesNotMatch
`Content-Type:` (empty)	empty, as signed	200

The middle row is the trap, and it is specific to your HTTP client.

The library fills in a header you did not set

In Python, omitting the header does not send no header. The urllib docs state it directly: “If this header has not been provided and data is not None, Content-Type: application/x-www-form-urlencoded will be added as a default.”

So the intuitive fix, deleting the line that sets the content type, swaps one wrong value for another wrong value. The 403 does not change, which makes it look like the content type was never the problem.

You can see it without touching S3 at all:

# show_default.py — what urllib actually sends
import urllib.request

def header_for(explicit):
    req = urllib.request.Request("https://example.invalid/upload", data=b"bytes", method="PUT")
    if explicit is not None:
        req.add_header("Content-Type", explicit)
    # urllib fills in its default when the handler prepares the request,
    # so ask the Request what it will send rather than reading .headers directly.
    return req.get_header("Content-type", "<absent, urllib will default it>")

print("explicit image/png :", header_for("image/png"))
print("not set            :", header_for(None))
print("explicit empty     :", repr(header_for("")))

explicit image/png : image/png
not set            : <absent, urllib will default it>
explicit empty     : ''

The second line is the one that matters: nothing is set on the request, so the handler supplies application/x-www-form-urlencoded on the way out. The third line shows the fix is representable; an empty string is a real header value, distinct from absence.

The upload

# upload.py
import urllib.error
import urllib.request
from pathlib import Path

def put_presigned(upload_url: str, path: str, content_type: str = "") -> int:
    """PUT bytes to a presigned URL.

    `content_type` must equal whatever the URL was signed with. Pass "" when it
    was signed with an empty content type, and set it explicitly either way so
    urllib cannot substitute its own default.
    """
    blob = Path(path).read_bytes()
    req = urllib.request.Request(upload_url, data=blob, method="PUT")
    req.add_header("Content-Type", content_type)
    try:
        with urllib.request.urlopen(req) as resp:
            return resp.status
    except urllib.error.HTTPError as e:
        if e.code == 403:
            raise SystemExit(
                "403 from the presigned PUT. The Content-Type you sent does not match "
                f"the one the URL was signed with (you sent {content_type!r})."
            ) from e
        raise

Passing the value in rather than hardcoding "" matters, because the correct value is a property of whoever signed the URL. Some services sign with a real type and reject the empty string just as firmly.

Verify it before you need it

Run the header probe first, since it needs no network and no credentials:

python3 show_default.py

Then confirm against your real signer with curl, which is useful because curl also has opinions about default headers. -H 'Content-Type:' with nothing after the colon removes the header entirely, while -H 'Content-Type: ' sends it empty:

curl -sS -X PUT --upload-file ./card.png \
  -H 'Content-Type: ' \
  -w '%{http_code}\n' -o /dev/null \
  "$PRESIGNED_URL"

A 200 means the signature matched. A 403 means you are still sending something other than what was signed, and the next thing to check is whether your client added a header you did not ask for.

Gotchas

Coverage will tell you this line is tested when it is not. The test suite recorded the URL and the body of the presigned PUT and never the headers, so the broken line was fully covered and the feature was dead in production at the same time. Coverage measures execution, not assertion. If a header carries meaning, assert on the header: record it in your fake and check it, then confirm the check fails when you revert the fix.

A bug that only fires on a code path you rarely take can ship for several releases. This one survived from the first release because the first posts had no image. Before calling a feature shipped, run the path that actually exercises it once, end to end, with the real service.

The 403 body is not always the error you are debugging. Signature mismatch, an expired URL, and a clock skew between your machine and the signer all surface the same way. Check expiry and system time before you start permuting headers, because those are cheap to rule out and produce identical symptoms.

An empty header value is not portable across clients. urllib sends the empty string, curl needs -H 'Content-Type: ' with the trailing space, and some HTTP clients drop empty headers entirely on the way out. Verify what your client puts on the wire rather than what your code says it set.

Sources

Troubleshoot signature mismatch errors with S3 presigned URLs — the headers used to generate the signature must match the headers the client sends
Uploading objects with presigned URLs — a presigned URL grants a specific request, with the permissions of whoever generated it
urllib.request — the documented default that fills in application/x-www-form-urlencoded when data is present and no content type was set

Changelog

feat(ghostwriter-x): 0.2.0 — improvement pass from the first image-bearing run (#91) (d1fd8ee)