Concurrency Limits for a 100,000-Task Overnight Run

Imagine your team has 100,000 documents to process overnight. Starting every request immediately creates a burst of connections, unpredictable spending and a retry backlog. A fixed worker count is better, but it cannot account for changing response times or document lengths. Build a controller that admits work only when concurrency, request rate, token capacity and budget all allow it.

01 · Step 1: Turn the overnight deadline into a throughput target

Divide the number of tasks by the available processing time. For this hypothetical batch, 100,000 tasks over ten hours requires about 2.8 completed tasks per second. Leave room for retries, slow responses and final reconciliation rather than treating the full window as usable processing time.

Define completion precisely: a successful API response may still fail your output validation. Track accepted results separately from attempted requests so the dashboard reflects actual progress.

02 · Step 2: Measure a representative slice before choosing workers

Run a small sample containing short, typical and unusually large documents. Record input tokens, output tokens, response duration, validation failures and any throttling responses. Capture both typical and slower request durations; one average can hide a long tail.

If you are considering Claude Sonnet 5.5 and DeepSeek V4 Pro, measure each separately. A concurrency setting that suits one model may produce different throughput and spending on another. Keep output limits and task instructions consistent enough to make the comparison useful.

03 · Step 3: Estimate concurrency, then check every capacity limit

Use this starting estimate: required concurrency equals target requests per second multiplied by average request duration in seconds. At 2.8 requests per second and an illustrative eight-second average duration, roughly 23 simultaneous requests would sustain the target under steady conditions.

That estimate is a starting point, not permission to launch 23 workers. Check the request and token limits that apply to your account and chosen route. If either limit cannot support the target, adjust the deadline, workload or available capacity.

• › Concurrency cap: limits requests currently in flight.
• › Request rate cap: limits how quickly new attempts begin.
• › Token rate cap: controls admission using estimated token demand.
• › Budget cap: prevents new work from exceeding the remaining spending allowance.

04 · Step 4: Reserve tokens and money before dispatch

Estimate each task's input tokens and reserve an output allowance before sending it. Reconcile those reservations against actual usage when the request finishes. Oversized documents should wait for sufficient capacity or enter a separate queue, so several large tasks cannot consume the entire allowance together.

Quanta fits teams managing sustained workloads across models through one pay-as-you-go balance. Calls are metered on real tokens, with cache writes and cache reads priced separately. Use those separate categories in your cost ledger, and avoid assuming that every repeated prompt receives a cache hit. Open a Quanta account to run your pilot and establish a measured spending baseline.

05 · Step 5: Make retries compete for the same capacity

Every retry must pass through the same admission controls as a new request. Otherwise, throttling can trigger extra traffic precisely when capacity is already constrained. Use bounded exponential backoff with jitter, honor retry guidance when provided and set a maximum attempt count.

Persist a task identifier and completion state so restarting a worker does not blindly repeat completed work. After an ambiguous timeout, account for the possibility that the request already incurred usage even though your application received no result.

06 · Step 6: Increase concurrency in measured stages

Start below your calculated cap. Increase concurrency in small steps only while completed throughput improves and latency, throttling and spending remain acceptable. Hold each setting long enough to observe representative traffic.

Reduce concurrency when throttling persists or latency rises without a useful throughput gain. Keep separate controls for models or routes with different limits, plus a shared budget gate across the batch.

07 · Step 7: Finish the batch with a reconciled ledger

Near the deadline, prioritize unfinished tasks and stop admitting work when the remaining budget cannot cover its reservation. Reconcile completed outputs, exhausted retries, unresolved requests and actual token charges. Save the measurements with the job configuration so the next overnight run starts from evidence.