Ai2 StateBench: Frontier AIs Drop to 13%

The Allen Institute for AI (Ai2) on Tuesday, Nov. 3, released StateBench, an open benchmark that tests whether language models can keep track of how facts about entities change over very long documents. Frontier models including Claude Opus 5.5, Gemini 3.8 Flash and OpenAI's GPT-6 Sol and Luna fell to 13% or lower on the hardest, longest problems, according to Ai2.

Key takeaways

• StateBench grades long-context state tracking at four difficulty levels, all derived from the same sequence of variable assignments.
• Frontier models dropped to 13% or lower on the longest "general finite" problems, the level where values are passed by reference and swapped.
• In small 275-million-parameter tests, Gated DeltaNet and hybrid models generalized far past their training length, while transformers "could never learn the hardest level."

What StateBench measures

Ai2 describes the benchmark as a test of "how well language models keep track of information about entities in long documents as they evolve." Many long-context tests ask a model to find one buried fact. StateBench instead asks for the current value of a variable after a long run of updates.

The four levels rise in difficulty. R-trivial uses fixed assignments such as let x = 4. Aperiodic adds reassignment by value, soluble adds self-referential updates such as x = x + 5, and general finite adds assignment by reference and swaps, which Ai2 calls "the hardest of our four levels."

Each sample starts from one general finite trace, and the three easier versions are derived from it with the same number of variables, values and assignment points.

How the benchmark is built

The GitHub repository, released under the Apache 2.0 license, renders the same traces in several surface forms, including integer code, cube painting, spreadsheet cells, status lights and people moving between rooms. Its test sweep uses 340 source samples across 12 length buckets reaching 64,000 assignments, with filler padding from 0% to 90%.

Why transformers struggle with state tracking

To probe architecture, Ai2 trained 275-million-parameter models from scratch: two transformers, two hybrids and two Gated DeltaNet variants. They saw sequences of up to 256 assignments and were tested on up to 64,000, or 250 times the training length.

Gated DeltaNet and hybrid models "generalized almost perfectly" on the three easier levels. On general finite, a Gated DeltaNet variant with a more flexible memory update stayed near perfect out to roughly 10 times its longest training sequence. "Transformers were different—even trained directly, they could never learn the hardest level," Ai2 wrote in its announcement.

Ai2 also tested pretrained 7-billion-parameter models, Olmo 3, a transformer, and Olmo Hybrid, which adds Gated DeltaNet layers, along with smaller 275M and 2.7B models. Pretrained hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial problems but were near chance when the same problems were framed as spreadsheet edits.

Context: long context is not the same as memory

The results add to evidence that headline context windows overstate what models can reliably do over long inputs. Earlier this week, a separate agent benchmark found that Claude Opus 5.5 led the field while still finishing only a small share of the work.

What's next

Ai2 said the current framework assumes a fixed number of variables and "doesn't yet cover" settings where that number changes. It also left open where Gated DeltaNet with non-negative eigenvalues sits in its hierarchy. "We hope it serves as a North Star for researchers," the institute wrote.

FAQ

What is StateBench?
An open Ai2 benchmark that checks whether language models can track changing variable values across documents up to 64,000 assignments long.

How did frontier models perform?
Ai2 said Claude Opus 5.5, Gemini 3.8 Flash and GPT-6 Sol and Luna dropped to 13% or lower on the longest general finite problems.

Can I run StateBench myself?
Yes. The generation code is on GitHub under Apache 2.0, and the evaluation data is published on Hugging Face.

References

• StateBench: Testing how language models track state changes over long documents — Ai2 on Hugging Face (Nov. 3, 2026)
• allenai/state-bench — GitHub