Retry Semantics

Part of what I've been doing as a backend engineer is upgrading an existing monolith (v2, feature-incomplete) into a microservice architecture. As part of that split, a gateway layer got introduced to translate gRPC responses from the new auth service and others into HTTP for the frontend, something like Backend for Frontend - it didn't exist before, since the monolith never needed a translation layer between protocols.

Everything looked fine in isolation. It wasn't until the frontend team started integration testing against staging that a pattern showed up worth digging into.

What the logs showed

Nothing on any dashboard flagged it - error rate, latency, request volume all looked normal from the admin side. The failure pattern was too rare and too low-volume to move any aggregate metric out of normal range. I noticed it by chance, scrolling through gateway logs for something unrelated during the integration testing window, and saw the same failing request repeating in a loop.

The gateway log showed a request failing, then getting retried - nothing unusual on its own. Cross-referencing with the auth service's own log, the underlying error was gRPC code 9 (FAILED_PRECONDITION), and the message made clear this was a deterministic rejection tied to the request itself, not something a retry would ever fix.

At first this looked contradictory: gateway log says "failed, retrying," auth log says "rejected, and correctly so." But the two logs didn't actually disagree, both were right in their own perspective. The gap was that the gateway never read the semantics of code 9, only whether something failed.

The auth service was correct to reject it every single time - the OTP had already been consumed on a prior successful verification. The gateway just never found that out.

Why this is close to invisible statistically

This isn't a retry storm from mass timeouts - that kind is easy to spot, high volume, all at once. This was a rare domain error, scattered, not enough volume to cross any alert threshold. Every aggregate metric stayed within normal range the whole time this bug existed. The only way to catch it was reading raw logs directly, by chance - no systematic process caught it beforehand.

Root cause

What the two log streams looked like side by side (sanitized):

# Gateway log — one user MFA attempt
[WARN]  bff: upstream error, scheduling retry attempt=1 upstream_status=500
[WARN]  bff: upstream error, scheduling retry attempt=2 upstream_status=500
[WARN]  bff: upstream error, scheduling retry attempt=3 upstream_status=500
[WARN]  bff: upstream error, scheduling retry attempt=4 upstream_status=500
[WARN]  bff: upstream error, scheduling retry attempt=5 upstream_status=500
[ERROR] bff: max retries exhausted, returning error to client attempts=6

# Auth service log — same 800ms window
[INFO]  auth: mfa.verify requested               token_hash=7f3a9c...
[ERROR] auth: otp already consumed, rejecting    grpc_code=FAILED_PRECONDITION
[INFO]  auth: mfa.verify requested               token_hash=7f3a9c...  ← same hash
[ERROR] auth: otp already consumed, rejecting    grpc_code=FAILED_PRECONDITION
[INFO]  auth: mfa.verify requested               token_hash=7f3a9c...
[ERROR] auth: otp already consumed, rejecting    grpc_code=FAILED_PRECONDITION
# ... repeats 6 times total

The gateway has a translation layer mapping gRPC status to HTTP status. Code 9 wasn't explicitly mapped, so it fell into the default catch-all returning 500. The retry middleware sitting above it, following a standard policy of retrying any 5xx up to 5 times, saw a 500 and retried - with no way to distinguish a deterministic domain error from a transient infra failure, since both arrived as the same status code.

Each request of this kind got attempted up to 6 times instead of 1, not because of load, but because the logic was wrong from the first request.

The takeaway

There's one question that decides everything here, and it isn't "did this fail." It's "could the identical request succeed if I sent it again?"

The top two are about the world - timing, load, a network blip. The bottom two are about the request - and nothing about resending an unchanged request changes an unchanged fact. Retrying those isn't a slow path, it's a guaranteed-wrong path executed five extra times.

Two changes came out of this.

Explicit gRPC to HTTP mapping, so no error falls through to the default catch-all:

export const GRPC_TO_HTTP_MAP: Record<number, HttpStatus> = {
  [status.FAILED_PRECONDITION]: HttpStatus.UNPROCESSABLE_ENTITY, // do not retry
  [status.ABORTED]: HttpStatus.CONFLICT,                          // do not retry
  [status.UNAVAILABLE]: HttpStatus.SERVICE_UNAVAILABLE,           // retryable
  [status.DEADLINE_EXCEEDED]: HttpStatus.GATEWAY_TIMEOUT,         // retryable
};

Switched the retry policy from a denylist to an allowlist. The old policy, retry everything except X and Y, means any new or unhandled code defaults to retryable. That default is exactly what caused this:

const RETRYABLE_GRPC_CODES = new Set([
  status.UNAVAILABLE,
  status.DEADLINE_EXCEEDED,
]);

export function isRetryable(grpcCode: number): boolean {
  return RETRYABLE_GRPC_CODES.has(grpcCode);
}

Anything not explicitly on the list fails fast on the first attempt, no retry.

Runnable Reproduction

A sanitized, standalone reproduction comparing both gateway behaviors side by side is available in the lab repository:

• engineering-notebook-labs/retry-semantics

Next

Now that retryable is separated from non-retryable, the next question is what correct retry logic actually looks like once the operation isn't idempotent - a real side effect, not just a read. How do you make sure that side effect doesn't run twice. That's the next post: idempotency keys.