One Retry Key for a Team’s Long-Running Generations

01 · Step 1: Treat a timeout as an unknown outcome

A team running sustained generation workloads cannot treat every broken connection as a failed generation. The caller may stop waiting while the upstream request continues processing. Immediately submitting the same prompt again can create duplicate work and additional token charges.

Consider a worker generating a large engineering review with Claude Sonnet 5.5. If its connection closes before the response arrives, the correct state is outcome unknown, not automatically failed. Idempotent retries start by preserving that distinction.

02 · Step 2: Assign one key to each logical generation

Create a durable operation identifier before sending the first request. Reuse it when the same business operation is retried. A new user-requested generation receives a new identifier, even if its prompt happens to match an earlier request.

Store a fingerprint of the effective request alongside that identifier. Include the model, messages, generation settings, output constraints, and relevant input versions. Reject attempts to reuse the identifier with a different fingerprint. Otherwise, a retry can silently return an answer generated from obsolete instructions.

An application identifier is not automatically an upstream idempotency guarantee. Do not assume that adding an arbitrary header makes a relay or model endpoint deduplicate requests. Verify support and semantics before relying on them.

03 · Step 3: Persist state before dispatching work

Use a shared database rather than an individual worker’s memory. Multiple workers must agree on whether an operation already exists, who owns it, and whether a result is available.

• › Pending: the operation is recorded but has not been dispatched.
• › Running: a worker has claimed the operation and started dispatch.
• › Succeeded: the complete result is durably stored.
• › Unknown: dispatch may have succeeded, but completion cannot yet be confirmed.
• › Failed: the operation has a known terminal failure.

Enforce uniqueness on the operation identifier and claim work atomically. A lease can help recover abandoned workers, but an expired lease does not prove that an upstream generation stopped.

04 · Step 4: Separate caller retries from generation retries

When a caller repeats an operation, first inspect its stored state. Return a saved result for a succeeded operation. For running or unknown work, return the existing operation status rather than launching another generation. A polling interface lets callers reconnect without tying business progress to one long-lived connection.

If the endpoint provides documented request lookup or idempotent replay, use it to reconcile uncertainty. Without those capabilities, there is an unavoidable gap between dispatch and recording the response. Application-side deduplication alone cannot guarantee exactly-once upstream execution.

05 · Step 5: Set retry limits around uncertainty and spend

Retry confirmed transient failures with bounded exponential backoff and jitter, respecting server retry guidance where provided. Define maximum attempts, elapsed time, and a token or spending allowance per operation. Check whether the client library already retries so that nested retry loops do not multiply attempts.

Handle unknown outcomes separately. Your policy might pause for investigation or permit a replacement attempt after explicit approval. Record that decision and its duplicate-execution risk. Authentication errors and invalid requests require correction, not repeated dispatch.

06 · Step 6: Make streamed output and side effects recoverable

For streaming responses, persist chunks with sequence numbers and track whether a terminal completion signal arrived. Partial text is not a completed result, and replaying saved chunks to a reconnecting caller should not trigger another generation. Do not assume an interrupted upstream stream can resume.

Protect downstream actions independently. Publishing a report, updating a ticket, or sending a notification needs its own deduplication key. Even a single saved generation can cause duplicate side effects if a worker crashes after acting but before recording success.

07 · Step 7: Roll out with a metered retry ledger

Record logical operations and physical attempts separately, including latency, outcomes, reported token usage, and unresolved charges. Quanta is a practical fit for teams consolidating heavy model workloads under one pay-as-you-go balance. Open a Quanta account to centralize access while keeping retry ownership in your application.

Because calls are metered on real tokens, with cache writes and reads priced separately, reconcile those categories rather than assuming a repeated prompt is free. Test disconnects, worker crashes, and duplicate submissions before increasing volume. The goal is controlled recovery, not merely more retries.