Analysts Took AI's Hidden Lean 86% of Time

Expert data analysts sided with a hidden, randomly assigned preference of their AI coding agent on 86% of key analytical choices, Anthropic researchers reported on Oct. 27. Most did not credit the agent, and none suspected its views had been set by coin flip.

Key takeaways

• Fifty analysts using Claude Code matched the agent's assigned lean on 86% of four analytical decisions, against about 50% expected by chance.
• 77% repeated the agent's arguments in their written answers, but only 15% mentioned the agent.
• The authors say the effect is a risk for human oversight of AI, because a reviewer is independent only if the judgment is their own.

How the AI agent judgment study worked

The study, by Amaç Herdağdelen of the Anthropic Fellows Program and Anthropic's Hailey Joren and Alex Tamkin, is described in an Anthropic research post and an accompanying paper titled "Who's Aligned to Whom?"

Sixty-one analysts with hands-on causal inference experience took part; 11 chose not to use the agent, leaving 50 in the main sample. Each was asked to advise a fictional group, the Greenfields Farming Cooperative, on whether to expand or discontinue a paid advisory program that about 40% of members had joined.

The task contained four judgment calls with two defensible answers each: how to measure productivity, which comparison method to use, which farms to cover and how to weight them. A final call was whether to roll the program out to everyone.

For each decision, a coin flip set which option the agent, Claude Opus 4.8 running in Claude Code, would lean toward. A hidden system prompt told it to argue for that view as its own, change its mind if evidence clearly opposed it and never mislead. Participants were not told about the leans.

What the analysts did

Analysts went with the agent's lean on 86% of the analytical choices, from 81% on coverage to 92% on the outcome measure. When the agent favored rollout, 54% recommended it, compared with 36% when it leaned the other way.

The effect held even for analysts who already had a view. In 38 cases where an analyst stated a preference before the agent spoke, they switched 60% of the time when the agent disagreed, versus 6% when it agreed.

Seventy-seven percent repeated the agent's arguments, sometimes word for word, while 15% credited it. Only six of 48 survey respondents guessed the study was about AI influence. Experience, self-rated expertise and trust in AI did not predict who deferred.

Context: faster work, borrowed judgment

The agent also improved quality. Six of 11 analysts who worked without AI made serious methodological errors, compared with three of 50 who used it, and their median time was 90 minutes versus 65. The authors note the no-AI group was less experienced, so the groups are not directly comparable.

The study follows Anthropic research published Jan. 29, in which 52 mostly junior engineers who learned a new Python library with AI help scored 50% on a follow-up quiz, against 67% for those who coded by hand.

"A human reviewer is only an independent check if their judgment is their own," the authors write, adding that research teams deferring to models with shared preferences could produce a false consensus.

What's next

The researchers caution that the agent's lean was deliberately strengthened and that the test covered one task, one model and about an hour of work. They call for ways to measure deference in everyday use, agents that present the best case for each option or ask for the expert's view first, and organizations that decide which choices need a person's own judgment.

FAQ

Did the agent lie to participants? No. Its instructions told it never to mislead and to change its view if the evidence decisively opposed it.

What does Claude choose without an assigned lean? In 112 of 123 runs, Claude Opus 4.8 made the same five choices, which the authors say could make results across teams correlated.

Were the analysts wrong? Not necessarily. Each decision had two defensible options; the concern is that the choice came from the agent rather than the analyst.

References

• How AI agents shape experts' judgment calls — Anthropic (Oct. 27, 2026)
• Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls — Anthropic (paper, Oct. 2026)
• How AI assistance impacts the formation of coding skills — Anthropic (Jan. 29, 2026)