Zhejiang
University

Enterprise software development

Can coding agents deliver enterprise PRDs?

PACE-Bench makes complete delivery measurable: from the final user request, to code evidence, to a diagnosis of what remains unmet.

PACE = PRD-Anchored Coding-agent Evaluation · PRD = Product Requirement Document

Explore the method View results Read the paper

When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories

43enterprise tasks

750audited Code Points

11model–scaffold settings

95.2%expert-consensus agreement

What PACE contributes

01 / BenchmarkComplete enterprise PRDs

Incremental tasks in existing repositories, with expert-audited acceptance points.

02 / EvaluationCited, contract-grounded verdicts

Structural checks, localized code evidence, and constrained semantic judgment.

03 / DiagnosisMore than a score

Unmet requirements linked to execution signals and implementation quality.

01 / Benchmark

Real requirements. Existing systems.

Each task asks an agent to extend an established system and fulfill a final Product Requirement Document (PRD).

Existing repository

Changes cross boundaries.

Reference changes span a median of 1,465 lines and 15 files.

Evolving PRD

The final request matters.

Documents and user clarifications define the final scope.

Complete acceptance

Every obligation needs evidence.

Every required behavior needs concrete implementation evidence.

02 / Method

From a final request to a cited verdict.

A Code Point is an atomic acceptance requirement, frozen before candidate evaluation.

The complete PACE-Bench pipeline

PACE-Bench pipeline: intent-preserving PRD reconstruction, bidirectional Code Point synthesis, contract validation, and symptom-signal attribution — The complete PACE-Bench framework. Follow one task through all four stages ↓

Grounded in production

Built from Alibaba Taobao & Tmall Group’s coding-agent platform, with separate expert reviews of requirements and verdicts.

43 tasks across application domains, PRD objectives, and language combinations. Open the figure for detail.

Covered1

All conditions fulfilled

Partial0.5

Real but incomplete fulfillment

Missing0

A required condition is unmet

Credit follows the contract.

Case score = weighted fulfillment ÷ total weight × 100. System scores average cases equally. Complete delivery requires a score of at least 95 and zero Missing Code Points.

Unresolved evidence blocks a final score. Trace signals and quality observations help explain the result.

Inside the scorer

One contract. Three evidence stages.

Conclusive violations only

A complete inspected scope can establish that an explicit acceptance constraint is violated. Otherwise evaluation continues. Name presence alone never earns behavioral credit.

Input: frozen contract + reconstructed patch → Output: Missing or defer

03 / Benchmark results

How much of the request was delivered?

11 configurations, evaluated on the same 43 tasks. Score measures fulfillment; success counts complete deliveries.

Best complete delivery14.0%

GPT-5.6-Sol, unified minimal scaffold

Highest mean score78.08

GPT-5.6-Sol out of 100

Evaluator agreement95.2%

Expert consensus; 17.2 percentage points above the strongest comparison judge

Mean functional score plus or minus standard error for all eleven model and scaffold configurations — All 11 configurations; blue denotes the unified scaffold and amber denotes native scaffolds. Tap the figure for detail; the table gives exact values.

Reported model performance

43 cases · mean ± SE

Model	Score (mean ± SE)	Success rate
Unified minimal scaffold
GPT-5.6-Sol	78.08 ± 2.71	14.0%
GLM-5.2	68.80 ± 3.72	9.3%
Qwen3.7-Max	66.70 ± 3.45	7.0%
Kimi-K2.6	63.09 ± 3.79	4.7%
Opus-4.8	61.68 ± 4.39	7.0%
Gemini-3.1-Pro-Preview	61.30 ± 3.84	2.3%
DeepSeek-V4-Pro	61.00 ± 3.65	9.3%
MiniMax-M2.7	52.13 ± 3.64	2.3%
Qwen3.8-27B	12.25 ± 3.12	0.0%
Native scaffolds
GPT-5.6-Sol (Codex)	76.78 ± 2.81	14.0%
Qwen3.7-Max (QwenCode)	65.69 ± 3.83	11.6%

Scaffold groups follow the paper’s main table.

Task difficulty

More to infer. Harder to deliver.

When a larger share of acceptance points must be inferred from the PRD, grouped mean scores across eight primary models are 75.0, 61.6, and 56.8 for the three bands.

Model scores across three bands of derived or implicit Code Point share: under 25 percent, 25 to 50 percent, and over 50 percent — Derived or implicit Code Point share: <25%, 25–50%, >50%. Open the paper figure for detail.

04 / Evidence

Beyond one score.

Attribution, code quality, and execution observability.

Symptom–signal attribution

What failed, and which trace signals accompany it?

Weighted loss across eight models, decomposed into contract, integration, test-delivery, and incomplete-implementation symptoms — Four mutually exclusive symptoms account for each model's acceptance shortfall.

72.0%

of pooled weighted loss comes from incomplete implementation and integration or registration gaps.

Different gaps, different repairs

An incomplete handler needs behavior; an unregistered handler needs a connection. Cited Code Point evidence distinguishes them.

Eight-model heatmap of case-level co-occurrence between four outcome symptoms and six process signals — All eight primary models, on a shared color scale. Click to enlarge. Co-occurrence directs investigation; it does not establish a cause.

Columns: verification gap · error-recovery failure · incomplete implementation evidence · context-retrieval failure · contract-alignment failure · final self-check gap.

Code quality

How does the implementation measure up?

Conventions, architecture, naming, and test delivery complement functional fulfillment.

Model	Convention	Architecture	Naming contract	Test delivery
Unified minimal scaffold
GPT-5.6-Sol	58.2	68.4	94.3	47.2
GLM-5.2	57.7	66.9	93.2	51.1
Qwen3.7-Max	52.5	62.8	94.0	45.4
Kimi-K2.6	57.4	65.1	95.5	43.2
Opus-4.8	54.9	65.9	89.9	49.6
Gemini-3.1-Pro-Preview	48.9	63.2	93.0	41.5
DeepSeek-V4-Pro	53.1	64.0	91.7	22.3
MiniMax-M2.7	51.1	63.6	92.4	36.2
Native scaffolds
GPT-5.6-Sol (Codex)	59.1	78.2	93.2	55.3
Qwen3.7-Max (QwenCode)	59.5	67.9	95.5	38.3

Test delivery uses the nine cases whose requirements call for tests. Syntax-check scores range from 98.7 to 99.8. Both observation tables cover the ten settings with quality and trace records.

Execution observability

Tool activity and verification after the final edit.

Model	Verification commands	Self checks	Post-edit verified	Tool uses
Unified minimal scaffold
GPT-5.6-Sol	10.4	8.4	5/43	62.0
GLM-5.2	5.9	4.0	11/43	122.8
Qwen3.7-Max	4.7	4.3	4/43	75.9
Kimi-K2.6	8.0	6.9	10/43	110.1
Opus-4.8	4.8	3.1	14/43	59.8
Gemini-3.1-Pro-Preview	4.6	3.4	3/43	73.9
DeepSeek-V4-Pro	4.4	4.3	0/43	95.8
MiniMax-M2.7	5.8	5.3	10/43	84.8
Native scaffolds
GPT-5.6-Sol (Codex)	20.3	14.2	16/43	116.7
Qwen3.7-Max (QwenCode)	1.7	0.7	17/43	97.2

Counts are per-case means, except post-edit verification: cases with a successful project-verification command after the final edit, out of 43.

05 / Further findings

Evidence behind the verdicts.

Expert agreement and control patches establish evaluation validity; repeated runs and interventions show what the resulting diagnosis can reveal.

Agreement with expert consensus

Judge	Agreement	Weighted κ	Severe dis.
Full pipeline	95.2%	0.945	0.0%
Without contract grounding	70.3%	0.595	9.7%
Without tiered judgment	87.7%	0.843	1.5%
Without checklist aggregation	76.3%	0.677	5.9%
Vanilla LLM judge	77.6%	0.686	4.3%
Vanilla LLM with contract	76.0%	0.652	8.3%
Agent-as-a-judge with contract	78.0%	0.726	1.5%

Expert agreement

The evaluator aligns with expert judgment.

The full pipeline reaches 95.2% agreement and 0.945 weighted κ over the fixed 750-point reference, with no severe Covered–Missing inversions. Each component contributes to this result.

Positive and negative controls

Patch	Score / 100
Accepted reference	98.35
Empty	0.00
Unrelated, same repository	0.00
Applicable cross-case	0.00
Identifier only	4.03

Control checks

Credit tracks fulfillment, not superficial matches.

Accepted references average 98.35; empty and irrelevant patches earn no credit. Experts checked the reference gaps against required PRD details.

Any-run and every-run Code Point coverage and case success across three Qwen3.7-Max executions — Any-run and every-run acceptance across three executions.

Mean variation across 43 cases

Case-level measure	Execution	Fixed-patch judging
Standard deviation	8.77	2.36
Range	16.34	4.46

Three runs each. Scores are measured per case before averaging, so opposing changes do not cancel out.

Stability

Execution changes more than judgment.

The table compares each case across three executions and three judgments of the same Run 1 patches. Execution varies 3.72× more by standard deviation and 3.67× more by range. The figure separately shows why any-run and every-run success differ.

Static knowledge inputs

Input	Mean ± SE	Δ
No knowledge	58.54 ± 5.11	—
Locators only	57.80 ± 4.84	−0.74
Repository-matched	59.23 ± 4.77	+0.69
Shared long context	57.03 ± 5.22	−1.51

Paired comparison on the 28 cases common to all four inputs; the control is specific to this comparison.

Knowledge

More context alone does not ensure delivery.

Static knowledge changes mean score by −1.51 to +0.69 points in the paired comparison. How context enters implementation matters more than its mere availability.

Harness

Diagnosis points to a concrete repair.

Trace-linked evidence identifies a missed entry path. Across three platform harness releases, mean score rises from 59.37 to 63.04 on the fixed suite, a 3.67-point gain from combined changes.

Platform observation

Adoption and delivery, February–August.

February is the predeployment reference month. From February to August, monthly median code adoption moves from 64.76% to 86.37%, while average delivery time moves from 30.36 to 14.03 days.

From February through August, monthly median code adoption rates are 64.76, 70.94, 72.48, 74.04, 83.33, 86.74, and 86.37 percent; average delivery times are 30.36, 23.89, 19.74, 18.55, 18.07, 21.72, and 14.03 days — Monthly platform observations; February precedes benchmark deployment. Vertical axes start at 60% and 10 days, respectively. The trend is observational.

06 / A real task, explained

Replace one service.
Preserve the whole experience.

A benefit-service migration from the released real case: follow one requirement from the original conversation to a checkable delivery contract.

The task in plain language

A customer visits a page to claim or use a benefit. Behind that page, several services supply results. The developer must replace the old benefit service across campaign, landing, promotion, and discovery pages while keeping other sources and existing page behavior working.

Intent Preserving PRD Reconstruction

Turn the conversation into a complete request.

User · initial request, translated excerpt

“Move that logic to renderNew. … Read the code and background, then clarify the requirements first.”

User · later clarification

“A strategy can query multiple sources and therefore has multiple conversion paths. If you return early to convert one result, will the other source results be ignored?”

Assistant · translated excerpt

“Only the migration targets should be replaced; renderModuleVO must always be retained.”

Output → Standardized PRD. The migration document and user clarifications establish the final scope: use the new service, isolate its results, and preserve independent source contributions.

Noise Filtering Bidirectional Synthesis

Make “keep other results” an acceptance contract.

Requirement view

The PRD requires selective replacement: remove migrated legacy results, retain independent results.

Implementation view

The reference diff supplies the change context. Reconciliation links required behavior to the query and conversion boundaries.

CP-14 · Selective legacy suppression

When unified data is present, suppress only migrated legacy additions; preserve unrelated additions without a global early return.

source: explicitweight: 1.2753 semantic criteria

Output → 17 reviewed Code Points. CP-14 is one of them. CP-12 separately requires a dedicated result field; CP-17 preserves discovery filtering, deduplication, and ordering.

Reference Grounded Contract Validation

Check the connected delivery, not just the new call.

InputFinal PRD + existing repository → coding agent → candidate patch

Unified service enabledNew query → dedicated result → separate conversion

Unified service disabledLegacy query → legacy conversion

Required in either branchAppend independent source results

Customer-facing outputCorrect claim / grant / use behavior for each page

Output → Cited point verdicts and a weighted score. For CP-14, the evaluator checks that new results take priority, independent results can still be appended, and no global early return discards them. The coding agent receives the PRD and repository; Code Points remain evaluator inputs.

Symptom–Signal Attribution

Explain exactly what an incomplete delivery would miss.

Acceptance gapThe new service works, but another source disappears.

This would violate CP-14 even if the new query succeeds.

Code evidenceInspect the merge and early-return branches.

Locate where independent results stop reaching the response.

Trajectory evidenceCheck whether the agent inspected those branches.

Execution records help distinguish an overlooked connection from an incomplete implementation.

Output → A traceable diagnosis. The requirement, implementation evidence, and agent behavior point to a concrete place to investigate. This is a review scenario, not a claimed failure or score for this task.

An anonymized real task: migration document + initial request + later clarifications. Business labels are public aliases. The released example includes the conversation, final PRD, all 17 contracts, and a runnable rewritten code context—not original business code.

07 / Cite

Citation

If you use PACE-Bench, please cite the arXiv paper:

@misc{liu2026pacebench,
  title = {When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories},
  author = {Tianqi Liu and Yandong Guan and Songze Li and Wenkai Wang and Gang Yu and Chengfei Lv},
  year = {2026},
  eprint = {2610.26394},
  archivePrefix = {arXiv},
  primaryClass = {cs.SE},
  url = {https://arxiv.org/abs/2610.26394}
}