AI coding agents can write a lot of code, but finishing the job is another matter. A new benchmark built from real software-development requests has found that even the best AI model fully completed only 26.4% of the tasks it was given, leaving a human to clean up the rest in nearly three out of four cases.
The test, called Co-Design Bench, was developed by researchers from Alibaba and several Asian universities, including Fudan University, Zhejiang University, the Chinese University of Hong Kong, Shanghai Jiao Tong University and Nanyang Technological University. Their paper appeared on arXiv on October 26, with coverage spreading on November 8.
Unlike most coding benchmarks, which use tidy, self-contained problems, Co-Design Bench starts from messy reality. The team collected 37,463 deduplicated requests from 26,314 real coding sessions and narrowed them to 273 tasks that involve changing an existing web application. Each task comes with hidden acceptance criteria written by experts — 2,331 in total — and the agent never sees them.
The best model still leaves three in four jobs unfinished
Ten AI models were tested as coding agents. OpenAI's GPT-6 Astra came out on top, satisfying about 69.9% of the acceptance criteria on average. But meeting most of the requirements is not the same as finishing the task: it fully satisfied every requirement in only 26.4% of cases.
The gap was similar across the field. Claude Opus 5 fully completed 18.3% of tasks, while Grok 4.6 finished last at 6.6%. The benchmark splits into two kinds of work: smaller multimodal edits, where the best model fully finished 41.0% of tasks, and repository-wide edits that coordinate changes across a project, where it managed only 17.9%.
Every model tested struggled when a single request mixed three or four kinds of change at once, such as layout, interaction logic and content updates. The researchers also found that the agent harness around a model — the software that lets it read, write and run code — swung results by up to 18.9 points.
Why the grading is stricter than the usual tests
Co-Design Bench grades the finished application, not the code. An AI judge tests the running app in a browser, and a second AI judge then tries to refute every pass, so mock-ups and hard-coded fakes earn no credit. That stands in sharp contrast to older benchmarks, where the same models score above 99% — numbers that say little about day-to-day engineering work.
The researchers note that the benchmark deliberately favours hard cases: the tasks were chosen from sessions where users had to rephrase a request or roll back a change. The 26.4% figure is therefore a stress-test result, not the success rate on an average to-do item. Even so, it lines up with other reality checks: a METR study cited by The Decoder found maintainers accepted AI-generated code about 24 points less often than SWE-bench Verified scores suggest.
What this means for developers
The takeaway is not that coding agents are useless — far from it. A model that gets 70% of the requirements right on a real task is a powerful starting point. But the benchmark suggests the industry's marketing around autonomous coding has run ahead of the evidence. Until agents can reliably close the last few requirements of a compound request, the human developer remains the one who ships the finished product.
The paper is a preprint and has not yet been peer-reviewed, and most of its authors work for Alibaba. But the top-ranked model was not Alibaba's own — its Qwen models ranked third and fourth.
