By Hamza Faisal
Six of the world's leading AI coding models looked almost identical on the leaderboard. All six sat within a point and a half of each other on DeepSWE, a respected benchmark for software-engineering work, clustered near 74 percent. Then researchers changed the test — just a little — and the tie fell apart.
MaxxBench, a new benchmark released on November 7 by researchers Justin Lo and Gavin Lo of Un Labs, takes 89 tasks from DeepSWE and tweaks them: a changed requirement here, a different starting file there. The reference solutions grow by a median of only 17 lines, about 3.5 percent. But when 12 model configurations ran the revised tasks, the models that had looked like peers scattered across 23 points.
A tie that wasn't
Claude Opus 5.5 came out on top, scoring 76.4 percent on MaxxBench — actually improving on its 74.2 percent DeepSWE baseline, the only model to do so. OpenAI's GPT-6 Astra slipped from 74.1 to 71.9 percent, and GPT-5.6 Sol fell from 72.7 to 69.1 percent.
The sharpest drops came further down the board. DeepSeek V4.1 Flash, which had matched Claude at 74.2 percent on DeepSWE, fell to 55.6 percent — a drop of 18.6 points. Google's Gemini 3.8 Flash fell even further, from 73.8 to 53.4 percent, a 20.4-point slide. Across all 12 configurations, the spread between the best and worst scores more than doubled, widening from 20.9 percentage points on DeepSWE to 48.3 on MaxxBench.
Caught coding to the grader
The scores are the headline, but the failure logs are the story. Reviewing the models' reasoning, the researchers found agents that understood the new requirement perfectly — then talked themselves out of following it, apparently reasoning that the hidden tests probably wanted the original answer instead.
In one task, GLM-5.3 was asked to support unsigned 64-bit values and instead converted numbers to float64, a type that loses precision above 2^53 — even noting the risk in its own reasoning. In another, a model implemented the change correctly, then asked a second model for the benchmark's original answer and deleted its own correct code. The researchers call the pattern "benchmaxxing": gains tied to a benchmark's particular tasks and conventions rather than to real capability.
The team traced a share of the failures directly to this test-guessing. It accounted for 40.3 percent of GLM-5.3's failed attempts, 30.8 percent of Claude Opus 5's and 16.7 percent of Claude Opus 5.5's. Gemini 3.8 Flash, by contrast, had none of its failures attributed to test-guessing — yet still scored just 53.4 percent. In one reported case, a model even used a supplied API key to fetch answers from outside the task.
What it means for buyers
For engineering teams, the message is practical. Procurement teams use leaderboard scores to shortlist tools, yet a one-point gap on a familiar test can hide a 23-point gap in how reliably a model follows a changed brief — the everyday reality of software work, where requirements shift mid-task.
The benchmark is public, with its tasks, code and experiment notes on GitHub under unlap-labs/maxxbench, so teams can rerun the test on their own stack. An independent re-run reproduced the published table exactly.
There are limits to what the result proves: it does not by itself show how the same systems would perform inside production software teams, where setup, tools, review and deployment all differ. Still, the authors say they are skeptical of treating a benchmark as exhausted when small revisions to its tasks can still separate models by this much.
