Three Gates for Repository-Scale Agent Test Runs

When a team runs coding agents across hundreds of packages, generating tests becomes the easy part. The harder work is deciding which behaviors deserve protection, preventing parallel workers from contaminating results, and keeping maintenance costs below the value of the coverage. Treat test generation as a controlled production workflow with three gates: behavioral evidence, trustworthy execution, and demonstrated sensitivity to defects.

01 · Build a test inventory before dispatching agents

Start with a machine-readable inventory containing each package’s test command, fixtures, runtime dependencies, ownership, and isolation requirements. Record whether tests need a database, network access, a fixed clock, or exclusive access to shared resources. Discover commands from repository configuration and existing CI jobs; ask the agent to flag uncertainty instead of inventing commands.

Use that inventory to create bounded work units. A useful assignment names one behavior, its implementation boundary, the permitted files, and an acceptance command. “Cover retry exhaustion in the billing client using the existing fake transport” gives an agent a reviewable target. “Improve billing coverage” gives it an expanding search problem.

02 · Gate one: establish an independent behavioral oracle

An agent that reads an implementation and writes matching assertions can preserve the same mistake twice. Require every proposed test to identify its oracle: a documented contract, an approved example, a protocol requirement, or a property that should hold across inputs.

• › Contract tests: Assert observable outputs, errors, and side effects at stable interfaces.
• › Property tests: Check invariants such as round-trip preservation or idempotency where the domain guarantees them.
• › Characterization tests: Capture existing behavior explicitly, marking uncertain behavior for owner review.

Keep characterization separate from intended behavior. If documentation and implementation disagree, the agent should report the conflict rather than silently choosing an assertion. Require a brief explanation of which plausible defect each new test would catch.

03 · Gate two: make parallel execution trustworthy

Give each worker an isolated checkout and a narrow write boundary. Keep dependency manifests, shared fixtures, and global test configuration outside that boundary unless the assignment explicitly requires changes. This limits accidental overlap and makes integration failures easier to attribute.

Filesystem isolation alone is insufficient. Allocate separate database namespaces, temporary directories, ports, and queue names where needed. Seed randomness and control time through existing repository utilities. Tests that pass only when run alone should fail this gate until the shared-state dependency is understood.

Validate progressively: run the new test, its package suite, and then affected dependents selected through the repository’s dependency graph. Run the integrated branch through required CI checks. Preserve exact commands and results with the patch so reviewers can distinguish observed evidence from an agent’s explanation.

04 · Gate three: prove assertions detect meaningful defects

A passing test establishes compatibility with the current implementation. It does not establish usefulness. Apply targeted mutation testing around the changed behavior: invert a boundary condition, remove a retry limit, or alter an error mapping. Confirm that the relevant test fails for the expected reason, then restore the original implementation.

Bound this work to avoid turning validation into an uncontrolled compute job. Prefer a few behavior-specific mutations over a repository-wide sweep. Investigate surviving mutations; some are equivalent changes, while others expose weak assertions or unreachable scenarios. Reject tests whose only sensitivity comes from incidental call order or private implementation details.

05 · Budget context and maintenance as recurring workload

For sustained generation, keep a stable repository instruction prefix and attach only the task’s relevant source, contracts, and fixtures. Where caching is supported, reuse requires matching content under the provider’s rules; repeated intent alone does not guarantee a cache hit. Track input, output, cache-write, and cache-read usage separately, alongside accepted tests and review effort.

Quanta fits teams running this workflow through Codex, Claude Code, or an OpenAI SDK: one pay-as-you-go balance supports model selection across assignments, with calls metered on real tokens and cache writes and reads priced separately. Compare Claude Sonnet 5.5 and GPT-6 Astra on representative repository tasks before setting routing rules. Open a Quanta account to meter the workflow against your team’s workload.

Maintain accepted tests through dependency-aware change selection and periodic reviews of flaky, redundant, or expensive cases. Give the agent failure evidence and the original behavioral contract before asking for repairs. Require an explanation whenever a repair weakens an assertion, skips a case, or changes expected behavior; those edits deserve explicit reviewer attention.