Recursive self-improvement has spent years as a thought experiment. On Thursday it became a scoreboard, and the numbers on it are small.
Two papers posted to arXiv on Aug. 20 attack the same question from opposite ends. One asks whether AI agents can actually design better training algorithms. The other asks whether the field can even tell when a model has improved itself. Neither result flatters the current generation of systems.
Four hours, one GPU, one hidden grader
The first paper introduces AI4AI-Bench, from a team led by Yizhe Chi. Its premise is that the entire self-improvement question collapses into one concrete skill: writing the code that decides how a model learns. Change the objective or the update rule, and every subsequent training run gets more capability per unit of compute — including the run that builds the next agent.
Today's evaluation suites don't isolate that skill, the authors argue. An agent can top them by collecting more data or tuning hyperparameters, neither of which touches the learning process itself.
So the benchmark freezes 10 research repositories spanning 10 algorithm families. An agent gets four hours on a single Nvidia B300 to rewrite a repository's training algorithm. The resulting code is rerun from scratch for up to 12 hours and graded by a fixed evaluator the agent never sees, head to head against the algorithm the repository originally shipped.
The 10 tasks report incompatible metrics, so each is projected onto a shared scale. Zero is an uninformative model. The algorithm already in the repository sits at 0.1. The task optimum is 1.0.
Six systems were run in 29 configurations across all 10 tasks. The mean score was 0.166. The strongest reached 0.250 — closing less than a fifth of the distance between what was already there and what is achievable.
Most agents never touch the learning rule
The breakdown is more instructive than the headline. Most submissions edited everything except the mechanism that matters, leaving the learning rule untouched. The minority that did rewrite it averaged 0.226, versus 0.126 for everyone else.
Raising reasoning effort mostly purchased nerve rather than skill. That willing minority grew from 8% of submissions to 64%, and the average score moved from 0.094 to 0.196 — real movement, but still far short of the ceiling. The team has released the tasks, the evaluators and every graded submission so the measurement can be rerun as systems change.
The second paper: gains that were never there
Cheng Xu and colleagues take aim at how self-improvement gets reported in the first place. Modern claims rarely rest on average accuracy; they rest on which specific problems a model wins and loses after training. That means subtracting two noisy measurements, an operation notoriously easy to fool.
The researchers ran three rounds of rank-32 LoRA self-training on Qwen3-8B, then pushed an untouched control model through the identical pipeline. Seven distinct measurement failures turned up. Each one, they report, flips a published-style finding when the control is missing — and several are standard practice.
Grading from a single greedy decode, for instance, invents capability changes in a model that was never trained at all, largely as a side effect of inference batching. The statistic meant to separate newly acquired skills from sharpened existing ones handed that untrained control a rate of 0.280.
Their replacement is a per-problem exact test against a pooled baseline under false-discovery-rate control. It finds nothing on held-out replicates, and holds steady across different multiple-testing rules, error rates and pool sizes.
Run against a ladder of training arms matched on data stream, volume and evaluation, the audit found that distilling from an external model genuinely reached problems the base model almost never solved. Three flavors of self-training did not, and a regression rules out distillation's larger overall gain as the explanation, at p below 10⁻⁸. Worse, self-training degraded problems the base model had already solved, at rates well above the measured noise floor.
What it adds up to
Read together, the two papers bracket the discourse. One shows agents can barely move the algorithms that produce AI. The other shows that much of the evidence suggesting otherwise may be an artifact of how the improvement was counted. Both teams released their evaluation artifacts, which is the practical takeaway for anyone building on these claims: the null result has to be measured, not assumed.
