Nate Swenson

A URL is always 23 characters, no matter how long it is

Ghostwriter X · No. 071 · 6 min read ·v0.1.0

Shipped

I shipped ghostwriter-x, an X (Twitter) sibling to my LinkedIn ghostwriter skill: same voice-first draft-approve-publish flow, retargeted at X’s format. X’s own API dropped its free tier, so it publishes through Typefully instead. One of the core engines underneath it is x_len.py: a from-scratch implementation of X’s weighted character-counting rules, the thing that decides whether a draft actually fits in 280 characters before it ever reaches the API. The rules turned out to be more interesting, and more of a trap, than “count the characters.”

What “280 characters” actually means

X doesn’t count characters the way len() does. Per X’s own documentation on counting characters, most characters (Latin letters, punctuation, common symbols) cost 1 point; CJK characters and emoji cost 2; a URL costs a flat 23 no matter how long it actually is. The reference implementation for this is Twitter’s open-source twitter-text library. Its config README explains the shape: a scale divisor, a defaultWeight applied to every code point unless a listed range overrides it, and its weighted-length config spells out the exact numbers: a scale of 100, a defaultWeight of 200, and a short list of Unicode code point ranges that get a weight of 100 instead. Divide by the scale and you get the effective weights: 1 for anything in a listed range, 2 for everything else. transformedURLLength is 23, matching the t.co wrapper every URL gets rewritten through.

The listed ranges are narrow: code points 0–4351 (Latin, Cyrillic, Greek, Hebrew, Arabic, general punctuation), 8192–8205 (a small punctuation/space block that includes the zero-width joiner), 8208–8223, and 8242–8247. Everything outside those four ranges, including every emoji and every CJK character, costs 2.

Building the weight function

Start with the per-code-point weight, straight from the ranges above:

_LIGHT_RANGES = (
    (0x0000, 0x10FF),
    (0x2000, 0x200D),
    (0x2010, 0x201F),
    (0x2032, 0x2037),
)

def _codepoint_weight(cp: int) -> int:
    for lo, hi in _LIGHT_RANGES:
        if lo <= cp <= hi:
            return 1
    return 2

def _text_weight(text: str) -> int:
    return sum(_codepoint_weight(ord(ch)) for ch in text)

Before weighing anything, normalize the input. X’s docs specify Unicode Normalization Form C: per the Unicode Standard Annex #15 definition, NFC is canonical decomposition followed by canonical composition, and it matters here because a base letter plus a combining accent (two code points) and the single precomposed character look identical on screen but are different strings to ord():

import unicodedata

combining = "e" + "́"  # e + combining acute accent: 2 code points
precomposed = "é"      # the single precomposed code point for é
print(len(combining), len(precomposed))
print(unicodedata.normalize("NFC", combining) == precomposed)

That prints 2 1 and then True: two visually identical strings, two different code point counts, until NFC folds them together. Every real weight function has to normalize before it counts, or the same character costs a different amount depending on which keyboard or app produced it.

Charging URLs a flat 23

This is the part that actually breaks if you write it naively. A tweet can contain a proper https:// URL, a bare www. domain, and a bare domain with no scheme at all (example.com), and X linkifies all three. The bare-domain case is the trap: a regex match for example.com/some/path might fully contain a real https:// URL inside its own path segment (foo.com/redirect?u=https://bar.io/page), and if you charge both matches you double-count.

The fix is to collect every match as a (start, end, weight) span, find schemed URLs first, then bare domains only where they don’t already overlap a schemed match, and walk the spans in order:

import re

_SCHEMED_URL_RE = re.compile(r"(?:https?://|www\.)[^\s<>]+", re.IGNORECASE)
_BARE_DOMAIN_RE = re.compile(
    r"\b[a-z0-9][a-z0-9-]*(?:\.[a-z0-9][a-z0-9-]*)*"
    r"\.(?:com|net|org|io|dev|ai|co|app|me|sh)"
    r"(?:/[^\s<>]*)?",
    re.IGNORECASE,
)
URL_WEIGHT = 23

def weighted_length(text: str) -> int:
    text = unicodedata.normalize("NFC", text)

    spans: list[tuple[int, int, int]] = []
    for m in _SCHEMED_URL_RE.finditer(text):
        spans.append((m.start(), m.end(), URL_WEIGHT))
    for m in _BARE_DOMAIN_RE.finditer(text):
        if any(s <= m.start() < e for s, e, _ in spans):
            continue
        spans.append((m.start(), m.end(), max(URL_WEIGHT, _text_weight(m.group()))))

    total = 0
    cursor = 0
    for start, end, weight in sorted(spans):
        if start < cursor:
            continue
        total += _text_weight(text[cursor:start]) + weight
        cursor = end
    total += _text_weight(text[cursor:])
    return total

The max(URL_WEIGHT, _text_weight(m.group())) on the bare-domain branch matters too: a short bare domain like example.com (11 characters) still costs the full 23, because X still linkifies it. A long one, like a bare domain with a long path, costs whichever is bigger.

Using it, and verifying the numbers

Run it against a handful of real inputs:

samples = [
    "hello",
    "日本語",
    "🔥",
    "Check out https://example.com/a/very/long/path?x=1 today",
    "go.example.com/path",
    "foo.com/redirect?u=https://bar.io/page",
]
for s in samples:
    print(f"{s!r:60} -> {weighted_length(s)}")

Running that produces:

'hello'                                                      -> 5
'日本語'                                                        -> 6
'🔥'                                                          -> 2
'Check out https://example.com/a/very/long/path?x=1 today'   -> 39
'go.example.com/path'                                        -> 23
'foo.com/redirect?u=https://bar.io/page'                     -> 38

hello costs 5 (1 per character, plain ASCII). 日本語 costs 6, 2 per character. 🔥 costs 2. The URL sentence costs 39: “Check out " (10) plus the flat 23 for the URL plus " today” (6). go.example.com/path costs 23 flat, the short-domain case. The last one, a bare domain whose path embeds a real https:// URL, costs 38, its own text weight, not 23 plus another 23 for the URL hiding inside it.

Gotchas

Emoji sequences joined by a zero-width joiner over-count, on purpose. A combined emoji like 👩‍💻 is three code points (woman, ZWJ, laptop) under the hood. twitter-text treats the whole joined sequence as a single emoji, weight 2. Counting each code point separately gives 5 instead. That’s a real gap between this implementation and the exact spec, and it’s deliberate: every place this counter diverges from the real rules, it over-counts, never under-counts. A tweet that passes this checker is guaranteed to fit on X; the rare heavily-emoji’d draft might get flagged as over when it would have actually fit. Getting that wrong in the other direction, a counter that says a draft fits when X will actually reject it, is the failure mode that matters.

A bare-domain match can swallow a real URL whole, and charging both is the bug. The foo.com/redirect?u=https://bar.io/page case above isn’t a contrived edge case, it’s the shape of any link-shortener or redirect URL with the destination in a query string. Drop the if any(s <= m.start() < e for s, e, _ in spans): continue guard and sum every matched span’s weight independently instead, and that same string comes out to 61: 23 for the schemed URL plus 38 for the bare-domain match that separately covers the whole string, including the URL inside it. The fix isn’t “detect redirects”, it’s the boring one: never add a bare-domain span whose start already falls inside a span you already have.

The regex for “does this look like a domain” needs a real TLD allowlist, not \.\w+ at the end of a word. Matching any trailing dot-plus-letters as a domain flags things like “v2.1” or “step.1” as URLs and charges them 23. Anchoring the bare-domain pattern to a fixed list of common TLDs (.com, .io, .dev, and so on) cuts that down; it will never be exhaustive, but it’s the same conservative trade-off as everything else here: false negatives (missing a real bare domain) just mean a slightly stricter count, never a draft that silently ships over the limit.

Sources

  • Counting characters — X’s own weighted-counting rules: 1/2-point character weights, the flat 23-point URL cost, NFC normalization
  • twitter-text config README — the scale/defaultWeight/ranges structure behind the weighted-length algorithm
  • twitter-text v3.json — the exact code point ranges and weights that back this implementation
  • Unicode Standard Annex #15 — the definition of Normalization Form C (canonical decomposition, then canonical composition)

Changelog

  • feat: ghostwriter-x — X (Twitter) ghostwriter skill v0.1.0 (#88) (c8357b8)