A 200-Case Support Eval for a High-Volume Queue

A support assistant can sound helpful while issuing an unsupported refund, overlooking an account restriction, or claiming an action succeeded when no tool ran. For teams processing sustained traffic, those mistakes become operational problems. Build an evaluation dataset from real conversations before expanding automation. The walkthrough below uses a proposed 200-case starting set: small enough to review carefully, but structured to expose failures that an average quality score can hide.

01 · Step 1: Define the support decisions you will test

Choose a specific workflow, such as billing questions, account recovery, or failed integrations. Define what the assistant may answer, which actions require tools, and when it must escalate. Avoid starting with a broad goal like “provide excellent support.” That phrase cannot tell a reviewer whether a response passes.

Write observable requirements instead: verify ownership before discussing account details, distinguish pending charges from settled payments, and never report a refund as completed without confirmation. Evaluate the assistant against the policy applicable to the case, not against a reviewer’s personal preference.

02 · Step 2: Sample conversations beyond successful resolutions

Pull conversations across issue categories, queue periods, escalation paths, and outcomes. Include unresolved threads and repeat contacts, not just conversations agents marked successful. Frequent issues deserve coverage, but rare, high-consequence mistakes need deliberate representation.

• › Routine cases: Common questions with a documented answer.
• › Ambiguous cases: Missing account details or conflicting descriptions.
• › Boundary cases: Requests outside policy or beyond tool permissions.
• › Adversarial cases: Attempts to bypass verification or override instructions.

Record each sampling category. An intentionally difficult test set should not be presented as an estimate of ordinary production performance.

03 · Step 3: Remove sensitive data without removing the problem

Replace names, addresses, account identifiers, credentials, and payment details with consistent synthetic values. Preserve relationships: if two messages refer to the same invoice, their replacements must still match. Retain timing and amounts only when necessary to evaluate the decision, transforming them consistently where appropriate.

Restrict access to original conversations, document retention rules, and check attachments and tool outputs separately. Redacting the visible customer message is insufficient when internal notes still contain sensitive information.

04 · Step 4: Freeze each case at a real decision point

Stop the conversation immediately before the assistant must respond or act. Store only information available at that moment. Later agent messages can inform annotation, but must not leak into the evaluated input.

Attach the applicable policy snapshot, permitted tools, and deterministic mock tool responses. A refund case might require a successful lookup followed by a denied refund request. Score both the attempted action and the explanation. Do not let evaluation runs modify production accounts.

05 · Step 5: Write rubrics instead of copying agent replies

A historical agent response is evidence, not automatic ground truth. It may contain outdated policy or an unnoticed mistake. Have reviewers define required facts, forbidden claims, acceptable actions, and escalation conditions.

Separate critical failures from presentation issues. An unauthorized account change should fail the case even if the answer is concise and friendly. Allow multiple valid phrasings. Double-review an initial batch, resolve disagreements, and revise unclear rubrics before annotating the remainder.

06 · Step 6: Separate development cases from release gates

Group related conversations by customer, incident, and near-duplicate issue before splitting the dataset. Otherwise, almost identical threads can appear in both prompt development and final testing. Keep a held-out release set that prompt authors do not repeatedly inspect.

Version the cases, policies, rubrics, prompts, and tool fixtures together. When policy changes, update affected expectations explicitly rather than silently changing the meaning of a passing score.

07 · Step 7: Compare quality, token costs, and rollout risk

Run identical cases against candidates such as Claude Sonnet 5.5 and Gemini 2.5 Flash. Quanta fits teams running repeated comparisons: one pay-as-you-go balance supports multiple models through an OpenAI-compatible interface. Open a Quanta account to run the same evaluation harness across candidates without maintaining separate balances.

Track input and output tokens, cache writes, and cache reads separately because they are priced separately. Report cost alongside category-level pass rates, critical failures, and latency. Repeat unstable cases and review disagreements manually. Set release gates before seeing results, then roll out gradually with monitoring and a rollback path. Add newly discovered production failures to a future dataset version without weakening the existing gate.