Nine of today's most advanced AI agents were handed genuine, unsolved research problems from scientists at US Department of Energy national laboratories — and not one of them produced a paper that met the experts' bar. The result, from a new benchmark called "Into the Unknown," is a reality check for claims that AI can already do independent science.
The study, led by researchers at Carnegie Mellon University and posted to arXiv in October, gave each agent 15 open research directions proposed by the original papers' own authors, spanning quantum computing, plasma physics, computational chemistry, materials science and scientific machine learning. Each agent received the original paper, its code and data, and was asked to carry out a follow-up study and write it up. The nine agents produced 135 submissions and ran for nearly 2,000 hours.
114 of 135 papers failed basic honesty checks
Of the 135 submissions, 114 failed an integrity check before reviewers reached the science. The most common problems: agents claimed experiments they never ran, reported numbers that contradicted their own logs, and submitted code that did not match the paper. Only one submission — from OpenAI's GPT-6 Astra — contained all the required materials, including a pre-registered experiment plan.
Only three agents produced submissions without integrity violations: GPT-6 Astra on 11 of its 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Every paper from the other six agents — Claude Opus 4.8, Claude Opus 5, Claude Fable 5, DeepSeek V4 Pro, Kimi K3 and GLM-5.3 — contained at least one violation. The open-weight models fared worst: misreported baseline values piled up among DeepSeek V4 Pro, Kimi K3 and GLM-5.3, and two submissions contained hardcoded rather than computed results.
The agents checked their own work — badly
Analysing 64 work traces, the researchers found three typical failure modes. Most often, agents checked their own work too leniently — 58 cases. In 34 cases they spotted a problem but documented it instead of fixing it, and in 21 they quietly shrank the task's scope.
The examples are striking. Claude Fable 5 forgot to launch four planned runs; its verification script adopted the same incomplete list and reported "41/41" runs completed — and the agent trusted that result to the end. GPT-5.6 Sol noticed its method missed its targets and wrote the paper anyway. DeepSeek V4 Pro dropped experiments because they were supposedly too slow for its "remaining time budget," even though there was no time limit at all.
More time did not help — and a caveat on the judging
Longer runtimes did not lead to better results. DeepSeek V4 Pro and GLM-5.3 ran the longest, at median times of 19.5 and 13.9 hours, but scored poorly. GPT-6 Astra, the best performer, needed a median of just 1.3 hours.
One caveat: GPT-6 Astra also served as judge, scoring every submission four times against the experts' rubrics. The researchers ran a cross-check in which the original papers' authors judged anonymised AI papers themselves; on whether a paper matched the given research direction, the automated system agreed with the experts in every case, though on scientific quality they aligned in only about two-thirds of cases.
The authors stress the benchmark is a testing ground for improving research agents, not a verdict on AI in science. The study's code and data are public, with a leaderboard of the tested models. The contrast with well-defined tasks is sharp: GPT-6 Astra solved 49 of 50 unpublished theoretical-physics problems — but those had answers that could be verified unambiguously. Open-ended research, it turns out, is a different game entirely.
