GPT-6 Astra solves only 26.4 percent of complex coding tasks in new Alibaba benchmark
A fresh test using real-world requests shows top AI coding agents still require extensive human cleanup on most projects.
Key points
- GPT-6 Astra fully completes only 26.4 percent of tasks in the Co-Design Bench test.
- Older benchmarks show scores above 99 percent, masking the gap between test and reality.
- Performance drops significantly when tasks combine layout, design and content changes.
The Decoder reported that GPT-6 Astra, the highest-performing model in a new Alibaba-led study, fully resolves only 26.4 percent of real-world coding tasks.reported This finding highlights a significant gap between benchmark performance and practical utility for developers relying on AI agents for complex application updates.A new test shows that even the best AI coding tools need human help to finish most jobs correctly.Co-Design Bench reveals a 70 percent average requirement match but only 26.4 percent full correctness for GPT-6 Astra on complex tasks.
Evidence: reported
AI coding agents still need human cleanup on nearly three out of four tasks in a real-world testaifleet-decoder
How we get here
Existing coding benchmarks have become saturated, with models scoring above 99 percent by running, checking and fixing their own code.reported These tests no longer distinguish effectively between model capabilities. The authors of the study noted that differences between models are barely measurable on these older tests, prompting the need for a more rigorous evaluation method that reflects actual development challenges.
Evidence: reported
AI coding agents still need human cleanup on nearly three out of four tasks in a real-world testaifleet-decoder
What is new
Researchers from Alibaba and Asian universities created Co-Design Bench, which uses real user change requests from actual coding sessions. The test focuses on cases where users previously rephrased requests or rolled back changes, indicating prior agent failure. Tasks range from small tweaks, such as updating totals from a receipt image, to full project overhauls, requiring agents to handle images, audio, video or documents as input.
What to watch
Grading for Co-Design Bench is strict, involving an AI judge testing the app in a browser and a second reviewer checking for hardcoded results or mock-ups. Points are awarded only for features that actually work. The study found that combining multiple change types, including layout, design and interaction, overwhelms models. Even extra compute, such as GLM-5.2 using six times more tokens, did not fix this weakness.reported
Evidence: reported
AI coding agents still need human cleanup on nearly three out of four tasks in a real-world testaifleet-decoder
Availability and limits
The Co-Design Bench results indicate that while agents handle the bulk of work, they leave details unfinished in most cases. Developers should expect to perform manual cleanup on nearly three out of four tasks. GPT-6 Astra meets about 70 percent of requirements on average but achieves full correctness in only 26.4 percent of cases.reported The benchmark is available via the published paper, but the test suite itself is not described as publicly accessible for independent replication in the source material.
Evidence: reported
AI coding agents still need human cleanup on nearly three out of four tasks in a real-world testaifleet-decoder
Why it matters
- Developers must plan for significant manual review time, as agents rarely produce fully correct code on complex tasks.
- Current benchmark scores above 99 percent are misleading indicators of real-world performance for production applications.
- Increasing compute resources does not reliably improve accuracy on tasks requiring multiple simultaneous changes.
- The gap between partial completion (70 percent) and full correctness (26.4 percent) defines the current operational limit of coding agents.
Topics: coding agents, benchmarks, Alibaba, GPT-6, AI development
Updated
The strongest doubt
Counterpoint from the sources
The authors did not derive a general advantage for smaller models, even though Qwen3.8-27B climbed to second place on larger overhauls.
Numbers in context
Circle area is proportional to the magnitude; labels retain signs. Figures may describe different measures. Read their contexts above.Ask the article
Local keyword search of this article; no generated answers.Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Questions readers ask
Which model performed best in the Co-Design Bench test?
GPT-6 Astra performed best, fully solving 26.4 percent of tasks and meeting 70 percent of requirements on average.
Why are older coding benchmarks considered unreliable?
Older benchmarks are saturated, with models scoring above 99 percent by self-correcting, making them poor measures of real capability.
Does using more compute improve agent performance on complex tasks?
No, GLM-5.2 used nearly six times more tokens than GPT-6 Astra but still scored significantly worse on smaller changes.
Reading receipt
- GPT-6 Astra fully completes only 26.4 percent of tasks in the Co-Design Bench test.
- Older benchmarks show scores above 99 percent, masking the gap between test and reality.
- Performance drops significantly when tasks combine layout, design and content changes.