By Hamza Faisal
The ARC Prize Foundation has independently rerun OpenAI's GPT-6 Astra on its ARC-AGI-3 benchmark and scored the model at 62.7 percent — not the 99.9 percent OpenAI touted when it launched the model in September. The foundation says the headline figure was produced under a different testing setup than the one every other lab's model must clear, and therefore does not count as a comparable score.
The gap comes down to the harness — the scaffolding wrapped around the model during the test. ARC Prize ran Astra twice and published both results. On its standard, provider-neutral harness, Astra scored 62.7 percent at maximum reasoning effort, at a cost of $26,098 per run. On OpenAI's provider adapter harness, the same model scored 98.6 percent at maximum effort and 99.9 percent at high effort. Hold the model and the effort fixed and change only the harness, and 36 to 45 points move.
Two harnesses, two different tests
The difference is what the model is allowed to carry between requests. Under ARC Prize's standard harness, each request starts cold: the model keeps only the notes it chooses to preserve, and nothing else crosses over. Under the provider adapter, Astra preserves an opaque reasoning state between requests and uses compaction for longer conversations, letting it reuse prior work instead of solving from scratch.
ARC Prize calls its standard harness the one that gives a consistent, apples-to-apples comparison across providers. The adapter result is not worthless: on the 167 pairs both harnesses solved, adapter runs were about 3.66 times faster and used 49 percent fewer tokens. But a model-plus-scaffolding result presented as a model result is a different claim, and it should not be used to rank one lab against another.
Why the headline travelled so far
OpenAI put the 99.9 percent figure front and centre at GPT-6 Astra's launch on September 3, and three days later Nvidia chief executive Jensen Huang cited it on X when he declared that "AGI has arrived," a post that cleared 10 million views. The number landed as proof that artificial general intelligence was effectively here — exactly the reading ARC Prize has warned against from the start. The foundation said when it designed ARC-AGI-3 that saturating the benchmark would not, on its own, prove AGI, and it has repeated that position since Astra's results.
None of that diminishes what the neutral score shows. At 62.7 percent, Astra holds the highest result ever recorded on ARC-AGI-3, and ARC Prize founder François Chollet called it a step-function change in capability for interactive reasoning problems, noting the model surpassed the human action-efficiency baseline on 96 percent of the benchmark's levels. The benchmark's toughest critic, in other words, is impressed — just not by the 99.9 percent.
What both sides say now
OpenAI says its harness changes two settings to better match real-world performance, not specifically to target ARC-AGI-3 — and the adapter does reflect how the model performs inside OpenAI's product, where state-carrying is a feature, not a cheat. The dispute is narrower than the headlines suggest. It is not about whether Astra is capable — both sides agree it is the strongest interactive reasoner the benchmark has seen — but about which number buyers, investors and rival labs should carry around.
The 62.7 percent is the figure every model gets held to equally, under conditions any lab can reproduce. The 99.9 percent is what the model does when the test is built around its strengths. Both are state of the art. Only one of them is a comparison.
By Hamza Faisal
