Twelve Questions, Five Checks

The second part of this series ended with nine items: nine things that are different under the two designs for an HTTP client that retries, or does not. Finding them was most of the work. What remains is asking each one the right question, and the right question has a particular shape.

The shape comes from the definition in part one. A reason is a fact that, if it flipped, would flip the choice. So the question to ask an item is: what fact about you, if it were different, would change which option wins? There are only so many kinds of fact that do this. I have found twelve, over a lot of decisions. Others may exist; the five checks later in this part decide whether a candidate counts. I call them axes, because each one is a line the decision can slide along, and somewhere on the line there is a point where the answer changes.

The twelve

Each axis is asked about a particular kind of item. That matters: the kind is the index. You do not run all twelve on every decision. You look at each item, note what kind of thing it is, and read off the one or two axes that apply. A set of nine items can validate against six axes and rules out the other six without reading them.

axis	asked about	the question	a flip
Rate of change	a dependency: a protocol, a spec, a set of consumers, a rule	Which side changes more often, and for what kind of reason?	The set of retryable status codes was stable for years; the server team now adds one a quarter. The rule moves behind one interface instead of living in every caller.
Locus of knowledge	a decision the code must make	Where does the information needed to make the decision actually exist?	Whether a request is safe to send twice is known where the request was built, not where it is sent. The sender stops deciding and asks.
Binding time	a value, or a check	When is it fixed: at compile time, in config, at startup, per connection, per call?	Maximum attempts was a constructor argument that every caller set to three. Make it a constant. Or it must vary per endpoint, and it becomes a per-call parameter.
Cardinality	a piece of data and its producers, readers, copies	One to one, one to many, many to many?	One instance of the client becomes four hundred. Independent backoff becomes a synchronized wave.
Scarce resource	what each option spends, and how often	Latency, throughput, memory, CPU, syscalls, bandwidth? Average or tail?	Buffering every request body for replay was free at a kilobyte. At fifty-megabyte uploads it is the memory budget.
Failure domain	an item imagined wrong, missing, or late	What fails together, who detects it, who can recover?	The server goes down; four hundred clients each try three times; the retry traffic is what keeps it down.
Ownership and lifetime	a piece of state and its holder	Who creates it, changes it, destroys it? Which lifetime encloses which?	The handler that started the retry loop has already answered its own caller with a timeout. The loop is now retrying on behalf of nobody.
Who drives	a flow between two components	Push or pull? Paced by the sender or the receiver?	Backoff paced by the client’s clock, versus paced by the server’s `Retry-After`.
Consistency	a pair of values that must agree	What must match, and how stale may the match be?	The client’s belief that the request failed and the server’s record that it was applied disagree exactly when a retry is most tempting.
Trust boundary	an input from something you do not control	Is it adversarial, buggy, or merely slow?	`Retry-After` is set by a server you do not own. An unchecked value parks your caller for an hour.
Dependency direction	an edge between modules: who must name a type	Does naming it add an edge, reverse one, or leave the graph alone?	If the client retries, the client must name every error class the decision needs. The caller’s domain errors cannot flow down into it.
Reversibility	the call sites and shipped artifacts that change if you are wrong	How many, who owns them, when can they next change?	Three services you own use the client: pick and move. It ships as an SDK to customers: the retry behavior is now a contract.

Choosing which to ask

Three shortcuts, from the most reliable down.

The kind of item. Look at the “asked about” column. The caller’s deadline is a piece of state with a holder: Ownership and lifetime. The other instances are copies of a piece of data: Cardinality. The rule for retryable failures is a dependency on the server’s behavior: Rate of change. The request body is something one option spends memory on: Scarce resource.

The words you wrote in the blank. When you filled in “under A, the ___ is ___” in part two, the verbs you reached for name the axis. “Unknown to the client” is Locus of knowledge. “Keeps going after it has passed” is Ownership and lifetime. “Written in every caller” is Cardinality, or Rate of change. “Must be buffered” is Scarce resource. “Set by a server you do not own” is Trust boundary.

The shape of the question. Before you have any items at all, the form of the question narrows the field. “Who should do X” calls Locus of knowledge and Who drives. “Where should X live” calls Ownership, Dependency direction, and Locus. “One, or one per thing” calls Cardinality and Binding time. “Events, or current state” calls Ownership and Cardinality.

For retries, the nine items call seven axes: Locus of knowledge, Ownership and lifetime, Cardinality, Rate of change, Scarce resource, Trust boundary, and Who drives. Five are skipped: nothing in the set is a pair of values that must agree, nothing is a module edge, and so on. Skipping is not laziness. An axis asked about an item that is not of its kind produces a mood, not an answer.

Five checks

An axis is only a question. What makes its answer a reason is passing five checks, and I want to be clear that the checks are the authority, not the table. An axis that is not in the table counts if it passes all five. One that is in the table does not count if it fails one.

The item is in the set. You found it in part two, by expanding a verb or walking the perimeter. An axis with no item under it is the verdict from part one with a new name.
The answer is checkable, and you can say where. Sources, in order of how much authority they carry: a specification, arithmetic, prior art, a decision your team already wrote down, and the existing code. If you cannot name a source, what you have is an opinion. It might be a good one. It goes in a different column.
The two options land differently. If both options give the same answer to the question, the axis does not separate them, however interesting it is.
The inverse is a point on the same axis. State the condition under which the other option wins. It must be a value of the same fact. “A wins because the caller cannot pass the deadline in; B wins if it can” is on-axis. “A wins because of the deadline; B wins because it is simpler” is two different conversations.
The fact and the decision are about the same layer. A property of the network is not a property of the operation that runs over it. “The network is unreliable” is a fact about the link. “This request can be applied twice” is a fact about the application. A reason built on the first, used to decide something that depends on the second, is standing one floor away from the thing it is talking about.

And one rule about counting: one fact, one axis. Some facts answer two questions. “The retry loop is inside the client” answers who holds it (Ownership) and when it was fixed (Binding time). Do not claim both. Pick the axis whose kind matches the item, and call the other a consequence. Two axes with the same fact under them is one reason wearing two hats.

Running it

Here are the seven, each with the fact, the source, and the inverse.

Locus of knowledge, about whether the request is safe to repeat. The fact: the client cannot know. HTTP defines which methods are idempotent (RFC 9110, section 9.2.2: PUT, DELETE, and the safe methods such as GET; not POST), and it says a user agent should not automatically retry a non-idempotent request unless it has some other way of knowing the request is safe. Whether a particular POST is safe is a property of the application that built it. Source: the spec, and the caller’s code. Inverse: if every request this client will ever send is idempotent by construction, a read-only client for instance, the client knows enough and can retry on its own.

Ownership and lifetime, about the caller’s deadline. The fact: under A, the loop’s lifetime is not enclosed by the caller’s deadline. A handler with a two-second budget that calls a client configured for three attempts with one-second backoff can spend five seconds inside it, and will, on the day the server is slow. Source: arithmetic. Inverse: if the deadline is passed into the client per call, the loop is enclosed again and can live inside.

Cardinality, about the other instances. The fact: four hundred clients that each retry three times turn one failed request per second into twelve hundred, and no single instance can see that happening. Source: arithmetic, and prior art. The Google SRE book’s chapter on cascading failures describes exactly this and prescribes a per-client retry budget: retry only while retries are a small fraction of your own recent traffic. The AWS SDKs implement a retry quota for the same reason, and gRPC’s service configuration has retry throttling built in. A budget is a rule, and a rule can only be enforced in a place that sees all the requests. Under B, that place does not exist. Inverse: one instance, or a server that can absorb the amplification.

Rate of change, about the rule for retryable failures. The fact: which failures are transient is a property of the server and the transport, and it changes when they do. Under A it is one file. Under B it is one file per caller. Source: count the callers. Inverse: one caller.

Scarce resource, about the request body. The fact: buffering for replay costs memory proportional to body size times concurrency. Source: arithmetic, once you know the sizes. Today it is nothing. The inverse is the upload endpoint, and it is a number, not an opinion: when body size times in-flight requests approaches the memory budget, replay must become opt-in, which is what Go’s HTTP client does by only retrying a request whose body can be re-created.

Trust boundary, about Retry-After. The fact: it comes from a server you do not control. Honor it, capped. Source: the spec defines the header (RFC 9110, section 10.2.3) and says nothing about bounding it; prior art caps it. This axis separates the options only in where the cap lives: one place under A, everywhere under B.

Who drives, about the backoff. The fact: a client paced by its own clock and a server that says when to come back are two pacing sources, and they need to agree on who wins. This is a consequence of the trust-boundary fact, not a separate reason. One fact, one axis. It goes in the consequences column.

Six reasons survive. And the thing to notice is that they do not all point the same way. Two of them, safe-to-repeat and the deadline, are facts the caller has and the client cannot see. Three of them, the budget, the rule, and the cap, are facts that only make sense in one place, and the client is the only place that sees enough to enforce them. The body is waiting for a number.

What the answer looks like

When the reasons split like this, the question was wrong. “Should the client retry, inside or outside” assumed the decision was one thing that had to live in one place. It is two things. There is a mechanism: classifying transport failures, computing backoff, enforcing a budget, honoring the server’s signal. Every fact that mechanism needs is visible from inside the client, and putting it anywhere else means writing it four hundred times. There is also a policy: is this request safe to repeat, and how long is anyone willing to wait for it. Every fact the policy needs lives in the caller, and a client that guesses at them is guessing.

So the client owns the loop and the caller supplies the two facts the loop cannot see. Per call, not per client, because they vary per call: a flag or a key that says this request is safe to repeat, and a deadline. A request without the flag is sent once. A request whose deadline has passed is not sent again. Inside those two constraints the client does everything else. The caller does not think about the network, which is what the first engineer wanted, and the client never decides something it cannot know, which is what the second engineer wanted. Neither of them was wrong. They were each holding half the list.

This is also what the field converged on, which is the prior-art check passing. Go’s standard library retries a request only if its body can be re-sent and either its method is one HTTP defines as safe or it carries an idempotency-key header. Stripe’s API asks clients to send an idempotency key so that a retry is safe by construction. gRPC puts retry policy in per-method configuration with a throttle on top. None of them chose “inside” or “outside.” They all chose “inside, given the two facts from outside.” When several groups who never spoke to each other arrive at the same shape, it is usually because the shape was determined by the facts, and the facts are what we just walked through.

The inverses are worth writing down too, because they are what tells the next person when this answer stops being right. If every request the client sends is idempotent and no caller has a deadline, plain option A is correct and the flags are ceremony. If there is exactly one caller and it already holds every fact, plain option B is correct and the client is a wrapper around a socket. Both of those are real systems. The design above is for the system described in part one: many callers, many instances, one server, and requests that mean different things.

Three more habits

The twelve axes and five checks are the framework. Three habits make it work in practice, and I will state them briefly because each one is a post of its own.

Look before you reason. Most design questions are lookups wearing the costume of judgment. Before arguing, check whether a spec, a calculation, a library that already solved this, or a note your own team wrote last year has already answered it. The retry question had an RFC and three reference implementations sitting on top of it the whole time. Arguing about something that has been settled is how a team spends an afternoon feeling productive.

Force a failure. For each option, describe the day it goes wrong: who is involved, at what time, and what they see on the screen. An option whose failure you cannot describe is one you have not understood. Two options whose failures come out identical mean you are on the wrong axis. The retry-storm story, with four hundred pods and a graph an operator cannot read, did more to decide this question than any principle.

Count edits, not futures. “Will we need to change this?” is unknowable. “How many places do we edit if we are wrong?” is countable now. One to three means the decision is cheap: pick one and move, and buy the answer for two call sites instead of an hour of speculation. Thirty, or places you do not own, means the forecast is all you have and it deserves the time. Reversibility is the last axis in the table because it decides how much the other eleven are worth.

And one more, which is less a habit than a permission. Some decisions are ties. When you have built the set and run the axes and nothing separates the options, they are either equivalent for this problem, or you do not understand one of them yet. Say which. If it is the first, take the option with less state and fewer types, write one line saying why, and stop. Grinding on a tie is not rigor. Recognizing one is.

The framework, on one page

For the reader who wants to keep something.

Write each option in one line, and find the verb.
Expand the verb into nouns: input, output, error, waits on, holds meanwhile, must exist first, happens after. Once per participant.
For each noun: under A it is ___, under B it is ___. Different: keep. Same: strike. “Cannot see”: keep, and look hard at it.
Walk the perimeter: imports, public types, environment, readers of output. Differing edit sets are items.
Check that every stated constraint lands on an item.
For each item, ask the one or two axes of its kind.
Check each answer: in the set, checkable with a source, lands differently, inverse on the same axis, same layer. One fact, one axis.
If the reasons split, the question was two questions. If nothing separates, say whether it is a tie or a gap.

A reason for a design is a fact about one thing that you could be wrong about, and that would change your mind if you were. Everything else is a mood. The engineers in part one were not short of intelligence or experience. They were short of nouns.