Claude Opus 5 Eval Sets Built From Live Support Threads

01 · What a Live Support Thread Gives You That Synthetic Prompts Cannot

Evaluation datasets built from synthetic prompts tell you whether a model can follow an instruction. Datasets built from real support conversations tell you whether it can do the job your users actually open tickets about. For a team running heavy, sustained workloads, that second question is the one that decides a rollout.

A single support thread contains far more than a question and an answer. It carries the user's working context, the fragments of configuration they pasted, the corrections they made after an early reply missed, and the moment the tone changed once something finally worked. Treat each thread as a sequence of turns with a known outcome, not as one prompt to copy into a benchmark file.

Before you collect anything, decide which questions your eval set has to answer. Most teams need three: can the model resolve the common case without hand-holding, can it recognise the cases it must escalate, and can it stay consistent across a long conversation where the user keeps adding detail. Every thread you keep should serve one of those questions.

• › the user's original phrasing, including slang, abbreviations and typos
• › the context that arrived in pieces across several messages
• › the point where a first attempt failed and what changed afterwards
• › the final accepted resolution, or the explicit decision to escalate

Threads where a user repeated themselves because the first answer was generic are especially valuable. They are your hard negatives, and they are the cases a polished synthetic dataset will never contain.

02 · Redaction and Consent Rules Before a Single Line Leaves the Queue

Real conversations carry real data, so the collection rules come before the collection. Sit down with whoever owns privacy and legal review at your company and agree on what may leave the ticket system, what may be stored, and for how long. Write the agreement down as a checklist that an engineer can apply without interpretation.

Strip identifiers and anything that could act as a credential. Names, email addresses, account numbers, internal hostnames, API keys, session tokens and billing details all go. Replace them with stable placeholders so the shape of the problem survives: if two messages referenced the same account, they should still reference the same placeholder afterwards, because that continuity is part of what makes the thread realistic.

• › redact before storage, not after, so raw transcripts never become a second dataset
• › keep any placeholder mapping in a separate system with its own retention rule, or destroy it immediately
• › record the source system, a hashed ticket reference, the date range and every transformation applied
• › define how a withdrawn consent or a deletion request propagates into the eval set
• › audit a random slice on a schedule and let reviewers flag anything the rules missed

Good provenance records also protect the evaluation itself. When a grader disagrees with a label six months later, you want to know which revision of which thread produced that label, not a filename that someone renamed.

03 · Sampling the Long Tail Rather Than the Loudest Tickets

Support queues are not evenly distributed. A handful of recurring questions will dominate the volume, and a naive sample will hand you an eval set that scores well because it mostly tests the easy case. Sample by intent instead: cluster the threads by the problem they describe, then cap the number of examples per cluster so the common case cannot drown out the rest.

Reserve deliberate capacity for the rare cases that are expensive when they fail: silent data loss, permission problems that only appear in one environment, upgrade paths that break on legacy configuration. Those cases are usually a small share of tickets and a large share of escalation cost. Write the selection rules down as you go, so a colleague can reproduce the set or extend it without guessing at your judgement.

• › split by time, not at random, so your test slice reflects conversations the model has not seen
• › freeze a held-out slice and treat it as read-only once grading begins
• › keep the difficulty distribution visible in the dataset metadata
• › version every change, and keep the old version until the new one has earned trust

When a rollout goes wrong, the question your leads will ask first is which slice moved. A stratified, versioned set answers that question in minutes.

04 · Writing Graders and Gold Labels That Survive a Model Rollout

Labels are the part of an eval set that ages best when they describe observable outcomes. Prefer a rubric with pass conditions a reviewer can check: the correct endpoint was identified, the stated limitation was acknowledged, the escalation happened, no invented configuration value appeared. Avoid labels that depend on tone or elegance unless tone is genuinely the product requirement.

Grade in two stages. A fast, inexpensive first pass can screen the full set for obvious failures and flag disagreements, while ambiguous cases go to a stronger reviewer. A model such as Claude Haiku 4.5 fits the screening stage, and a model such as Claude Opus 5 is useful where the gold label itself is subtle. Keep the humans in the loop for the cases where your rubric turns out to be ambiguous, and feed those back into the rubric text rather than patching labels one by one.

• › pin the grader and model versions in the evaluation configuration
• › measure grader agreement on a sample before trusting automated scores
• › record the rubric version next to every score
• › re-run the frozen baseline after any rubric change, so old and new numbers stay comparable

05 · Metering the Eval Run: Cache Writes, Cache Reads and Token Truth

Evaluation runs at volume are a real workload, and they should be budgeted like one. Long support threads push input tokens up quickly, and re-sending the same rubric and context on every pass wastes money. Structure each run so the stable prefix, such as the rubric, the system instructions and the conversation itself, is reused across attempts. Then cache reads carry the repeated material instead of reprocessing it. Remember that cache writes are metered too, so a prefix is worth caching only when you will reuse it enough times to pay for the write.

Run the cheap sweep on the full set and the expensive grading pass on a stratified subset. That keeps the frequent signal fresh without turning every sprint into a full-cost exercise. Keep an eye on the ratio of cache writes to cache reads during the run; a run that only writes and never reads is a sign your prompt layout changes between attempts.

This is the kind of accounting a pay-as-you-go relay is built for. Quanta gives one balance across a wide range of models and works with the CLI and SDK tooling your engineers already use, so a single team can compare candidates during a rollout without provisioning anything new. Every call is metered on real tokens, with cache writes and cache reads priced separately, which is exactly the visibility an eval budget needs. Teams that send colleagues our way earn 19% of their top-ups.

Finish by freezing the set, recording the baseline scores, and re-running that baseline at every model or prompt change you plan to ship. When the numbers move, you will know which slice moved with them. Open an account when you are ready to run the next eval sweep at full volume.