Three Million Daily Calls Need More Than a FIFO Queue

At millions of model calls per day, the queue becomes a policy engine: it decides whose work runs, which budget pays, and what happens when capacity disappears. For a team planning three million daily calls, a single first-in, first-out queue is rarely enough. Build the system around durable jobs, token-aware admission, and explicit recovery rules.

01 · Step 1: Turn daily volume into a capacity envelope

Three million calls per day averages roughly 35 requests per second. That average hides deployment bursts, scheduled batches, and long-running generations. Measure arrival rates over short windows, response duration, input size, output size, and acceptable queue age for each workload.

Use concurrency equals arrival rate multiplied by average execution time as a starting estimate. At 35 requests per second and an illustrative 20-second average duration, approximately 700 calls would be in flight. This is planning arithmetic, not evidence that an upstream service permits that concurrency.

02 · Step 2: Make accepted work durable before acknowledging it

Store each job with a unique identifier, tenant, workload class, model, deadline, token estimate, output limit, and retry policy. Keep large prompts in durable storage and place references in queue messages. Acknowledge submission only after the job is durably recorded.

If accepting a job also changes application state, use a transactional outbox to avoid committing one operation without the other. Assume messages can arrive more than once. Workers need leases, lease renewal, and idempotent result commits so redelivery does not publish duplicate results.

03 · Step 3: Separate urgent calls from bulk processing

Give interactive requests, background enrichment, and large batch jobs separate scheduling classes. Reserve some capacity for latency-sensitive work, while guaranteeing background jobs a minimum share. Pure priority scheduling can otherwise starve the batch queue indefinitely.

Within each class, schedule fairly across tenants. Weight service by estimated token cost rather than job count alone: one enormous prompt should not receive the same scheduling weight as one short classification. Add aging so older eligible jobs gradually gain priority.

04 · Step 4: Admit jobs against several limits together

Before dispatch, acquire capacity for concurrent requests, requests per interval, estimated tokens per interval, and tenant spending. Keep these controls coordinated across workers; independent local counters can collectively exceed a shared limit.

Quanta fits naturally as the pay-as-you-go relay behind this dispatch layer, with one balance covering multiple models. Teams can open a Quanta account and route an initial workload through it while retaining their own queue and admission controls. For example, maintain separate capacity estimates for Claude Haiku 4.5 and Gemini 2.5 Flash rather than assuming interchangeable throughput.

05 · Step 5: Reserve token budgets and reconcile actual usage

Estimate input tokens, reserve an output allowance, and check the job against its tenant budget before execution. After completion, replace the reservation with actual metered usage. Release unused reservations, but retain an unresolved accounting state when the execution outcome is uncertain.

Quanta meters calls on real tokens, with cache writes and cache reads priced separately. Track those categories separately from ordinary input and output usage. Do not treat every cached token as free, or assume an expected cache hit will occur. Reconcile estimates against observed billing data.

06 · Step 6: Retry failures without multiplying the incident

Classify failures before retrying. Respect retry guidance when available, use exponential backoff with jitter, and enforce both attempt limits and elapsed-time limits. A job whose deadline has passed should expire rather than consume another execution slot.

A timeout does not prove the model call never ran. Without confirmed upstream idempotency support, retrying can cause duplicate execution and additional charges. Record ambiguous attempts explicitly. Use a shared retry budget and circuit breakers so an outage does not turn every queued job into repeated traffic.

07 · Step 7: Apply backpressure before queue age explodes

Set maximum queue age and backlog limits for each workload class. When projected completion exceeds a deadline, reject, defer, or pause producers instead of accepting work that cannot finish usefully. Allow model substitution only through an explicit policy validated for quality, cost, and data requirements.

08 · Step 8: Roll out with queue-age and cost guardrails

Start with a small traffic share, then increase it while watching oldest-job age, completion latency, token throughput, retry amplification, and spending by tenant. Test worker crashes, duplicate delivery, rate limiting, and balance exhaustion. A production-ready queue is not merely durable: it stays fair, bounded, and financially predictable when demand exceeds capacity.