Choosing Models for Long-Document Summarisation

For sustained document workloads, choose the model that delivers the lowest measured cost per acceptable summary—not the most impressive answer to one sample. This Quanta Briefings walkthrough turns model selection into a repeatable engineering process: define the output, test representative documents, compare processing strategies, and route exceptions deliberately.

01 · Step 1: Define an acceptable document summary

Before comparing models, write an output contract. Specify the audience, length, required sections and evidence format. For an incident report, that might mean impact, timeline, contributing factors, corrective actions and unresolved questions, with section references for substantive claims.

Build an evaluation checklist around factual accuracy, coverage, attribution and coherence. Long-document summarisation research identifies coherence errors that simple fluency checks can miss, so include human review rather than treating polished prose as proof of quality.

02 · Step 2: Assemble a workload-shaped test set

Select documents from your actual queue: routine reports, unusually long files, repeated templates, dense tables and sources with conflicting statements. Keep a separate holdout set for the final comparison. Record extraction failures separately; otherwise, you may blame a model for text it never received.

Include checks for important facts near the beginning, middle and end. Earlier long-context experiments found that evidence position affected performance; use this as a reason to test placement sensitivity, not as a performance claim about current candidates.

03 · Step 3: Benchmark two candidates without assuming a winner

For example, put Gemini 2.5 Flash and Claude Haiku 4.5 through the same evaluation. These are starting candidates, not a declared ranking. Confirm each route’s current input limits, output allowance, supported features and pricing before running the test.

Keep the source text, instructions and output contract consistent. Measure accepted summaries, unsupported claims, missing required facts, completion time and total billed usage. Review outputs without model labels where practical. Choose provisionally on your results, not a model name or an isolated demonstration.

04 · Step 4: Compare whole-document and staged processing

Test a whole-document request against a staged workflow. In the staged version, summarise sections first, then synthesise those summaries. Hierarchical merging and incremental summary updates are established approaches for documents that exceed an available context window, with different coherence and detail trade-offs.

For your staged test, split at meaningful section boundaries and retain document identifiers, headings and source locations. Require intermediate outputs to preserve dates, quantities, exceptions and unresolved disagreements. During synthesis, allow retrieval of original passages: do not require the final model to resolve contradictions from compressed notes alone.

05 · Step 5: Calculate cost per accepted summary

Use total processing spend divided by accepted summaries as your primary cost metric. Include extraction calls, section summaries, synthesis, verification, retries and escalations. Keep latency and reviewer effort alongside that figure rather than hiding them inside a token comparison.

Quanta fits this comparison naturally: its pay-as-you-go relay puts multiple model routes under one balance, with calls metered on real tokens and cache writes and reads priced separately. Use those separate billing categories in your ledger rather than assuming every input token carries the same charge.

06 · Step 6: Cache reusable material, not hypothetical savings

Where supported, place stable instructions and reusable document content before changing questions. Provider caching documentation specifies exact-prefix matching and recommends keeping changing content after the reusable prefix. Check the selected route’s implementation rather than assuming identical behaviour everywhere.

Benchmark a one-off summary separately from repeated questions about the same document. Record actual cache writes, reads and misses. Caching does not eliminate output generation, so keep generated-token charges in the comparison.

07 · Step 7: Route routine work and escalate defined failures

Deploy the lowest-cost candidate that passes your quality threshold for routine documents. Escalate explicit conditions: missing evidence, conflicting totals, failed format checks or documents outside the tested range. Verify critical claims against source passages, and retain sampled human review.

Open a Quanta account to run this comparison across model routes under one balance. Start with a bounded production slice, preserve document-level usage records, and rerun the holdout evaluation before changing models or prompts. Your default should be a measured operating choice—not a permanent allegiance.