Anthropic wanted to know who really makes the decisions when an expert works with an AI agent. So its Alignment team ran an experiment with a rigged answer key: it secretly loaded 50 professional analysts' AI agents with a preference chosen by a coin flip — and the experts followed it on 86% of their judgment calls.
Published on October 27 as "Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls," the study was spotlighted by the Washington Post on November 10 and reported by Al Jazeera on November 13.
A coin flip in the system prompt
Researchers Amaç Herdağdelen, Hailey Joren and Alex Tamkin recruited 61 professional analysts specialising in causal inference — working out whether one thing really causes another. Eleven chose to work without an agent, so the main analysis covers the 50 who used Claude Code, running Claude Opus 4.8.
Each analyst advised a fictional farming cooperative on whether a voluntary advisory programme had raised farmer productivity — a realistic task that took a median of 65 minutes. The task contained four open analytical decisions, each with two defensible options: how to measure productivity, which statistical comparison to use, whether to judge the programme on enrolled farms only or the full membership, and how to weight farms of different sizes.
Then came the rigging. A coin flip slipped into each agent's system prompt quietly set its preference on all four choices. Pure chance would have produced a match rate near 50%. The analysts matched the agent's secret lean 86% of the time — with per-decision rates of 92%, 82%, 81% and 90%.
The experts thought the calls were theirs
The influence ran deeper than passive agreement. In 38 cases where an analyst had voiced a preference before the agent did, they switched away from it 60% of the time when the agent disagreed — compared with just 6% when the agent agreed. The effect on the final recommendation was weaker: 54% recommended rolling the programme out when the agent leaned that way, against 36% when it leaned toward discontinuation, a shift that was not statistically significant under the study's registered primary estimator.
Perhaps the most striking gap was between influence and self-perception. When asked to explain their decisions, 77% of the analysts used their agent's arguments — but only 15% credited the AI as the source of their thinking. None of ten tested characteristics — experience, expertise, education, test scores or trust in AI — reliably predicted who deferred.
The agents did improve the work in one respect: only three of the 50 agent users made serious result-invalidating errors, against six of the 11 without one — though the groups self-selected, so no causal claim follows.
Anthropic's warning — and the limits
Anthropic calls the risk disempowerment: the expert's name sits on the report while the model's priors quietly drive the key choices underneath. The authors stress this is not an argument for using AI less — the agents clearly helped analysts avoid mistakes — but for treating professional review as a weaker safeguard than it looks.
The study has real limits, and Anthropic states them plainly. The lean was deliberately amplified, and the test covered one task, one field and one pinned model, so real-world effects could be smaller. Still, the core number is hard to ignore: when the machine's preference was decided by a coin toss, experts followed it almost nine times out of ten — and mostly believed the conclusions were their own.



