A 5-Point Rubric for Grading 200 GPT-6 Astra Runs

01 · When three teams graded the same runs and disagreed

A rollout touching incident write-ups, retrieval summaries and customer replies stalled for a week on one question: is this output good enough to ship? Three teams graded the same two hundred runs and reached three different answers. Nobody was being difficult. Each team was grading against a private, unwritten standard, and those standards drifted apart within days. The fix was not another alignment meeting. It was a single shared rubric, short enough to memorise and specific enough that two graders scoring the same run land within a point of each other.

This primer is for team leads who own both a budget and a rollout date. A shared rubric does three jobs. It makes quality legible across teams that would otherwise argue in adjectives. It gives you a defensible number when a stakeholder asks why a faster tier was rejected. And it turns the vague complaint that output felt off into a change you can test on the next batch instead of debate in the next meeting.

02 · Five scoring axes that survive contact with real volume

Keep the rubric to five axes, each scored from one to five. Fewer axes and graders lose the ability to say what went wrong. More axes and scoring time balloons until people skip the rubric entirely. Define the middle score first: a three means usable with light editing, and every other anchor hangs off that.

• › Instruction fit: did the output do the task you asked for, in the format you asked for, with nothing extra appended?
• › Factual reliability: are the claims grounded in the supplied context, and does the output decline rather than guess when context is missing?
• › Structure and scope: is length, sectioning and level of detail matched to the audience, without padding or truncation?
• › Operational safety: would this output be safe to hand to a customer or merge into a repository without a specialist reviewing it first?
• › Edit distance: how much human work follows, measured in the changes you would actually make before the output is usable?

Make edit distance the axis you argue about most, because it is the one that connects grading to money. A run that scores well on instruction fit and poorly on edit distance is not cheap, it is a discount on labour you still have to pay. Track the axis scores separately from the total, and publish both. Averages hide the pattern where one axis is systematically failing across a whole prompt template.

03 · Anchor outputs: the calibration set every grader scores first

Before anyone grades live traffic, build an anchor set of twenty to thirty saved runs that you and your most trusted graders have already scored together. Include the unambiguous wins, the borderline cases and the clear failures, drawn from the prompt types your rollout actually uses. Every new grader scores the anchor set blind, then compares against the agreed scores and discusses the gaps. The discussion matters more than the numbers; it is where teams discover that one group treats a missing caveat as a safety failure and another treats it as an instruction fit problem.

Refresh the anchor set whenever the prompt template changes materially. A stale anchor set teaches graders to defend last quarter's judgement. Keep the old set versioned so you can tell whether quality moved or the standard moved.

04 · Wiring the rubric into the tooling your team already runs

A rubric that lives in a document will be ignored by week three. Put it where the work happens. Attach the rubric to the project instructions your engineers use in Claude Code, Codex and Gemini CLI, and to whatever service layer you run on an OpenAI-compatible SDK, so that reviewers see the same five axes in the same order every time.

• › Set a merge threshold per axis, not just on the total, so a single unsafe output cannot hide behind strong style.
• › Regrade a small sample of accepted runs each week, and require a second grader on any run that scored a one or a two.
• › If you use a model to pre-grade output, calibrate it against the human anchor set first, and never let it be the only judge on the operational safety axis.
• › Record grader, rubric version and prompt version with every score, so you can separate model drift from grader drift.

05 · Reading rubric scores beside metered tokens and cache pricing

Grading is also how you defend spend. When someone proposes swapping a heavier model for a lighter one to cut cost, run both through the anchor set and compare axis scores alongside real token usage. A model that trims tokens but doubles edit distance is not saving money; it is moving cost from metered input to paid human time. That comparison is easiest when your usage is metered honestly on real tokens, with cache writes and cache reads priced separately, because you can see whether a prompt template that reuses context is actually cheaper per accepted run.

Quanta is built for exactly this kind of sustained, high-volume work. It is a pay-as-you-go API relay where one balance covers a wide range of models, including GPT-6 Astra and Claude Sonnet 5.5, and it works with Claude Code, Codex, Gemini CLI and any OpenAI SDK, so your grading workflow does not change when you switch models to run the comparison. Teams that refer others earn 19% of their top-ups. If that matches your workload, open a Quanta account and point your existing tooling at it.

06 · Re-anchor the rubric each time the model tier changes

Rubrics decay quietly. The prompt template gets a new system message, the rollout moves to a different model tier, and graders keep applying last month's expectations to a changed distribution of output. Make re-anchoring a standing item at the start of every rollout: rerun the saved anchor set, compare scores against the stored baseline, and only then start grading live traffic. If scores shift on the anchors, you know the standard or the model moved, and you can decide deliberately rather than discovering it in a customer report.

Keep the rubric short, versioned and boring. The teams that stay consistent at volume are not the ones with the most elaborate evaluation documents; they are the ones who can say, in one sentence and one number, why a run is ready to ship.