Qwen Weight Edits Gain 29 Points, Fade at 9B

Researchers at Shanghai Artificial Intelligence Laboratory, Tongji University and Fudan University reported in a preprint submitted Nov. 6 that having GPT-6 Astra edit a small Qwen model's weights directly raised its MATH-500 score by 29.45 percentage points. The same study found gains shrank sharply on a larger model, and six of 16 editing runs ended by keeping the original weights.

The paper, "Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations," tests a technique MIT CSAIL researchers introduced on Nov. 2 across more benchmarks and model sizes. It has not been peer-reviewed.

Key takeaways

• Changing 1,023 values in a single column of Qwen3.5-0.8B lifted held-out MATH-500 accuracy from 8.35% to 37.80%.
• On the stronger Qwen3.5-9B, kept edits added 1.54 to 3.03 points, and four of seven runs reverted to the original weights.
• Three separate searches on the same math task found structurally different edits, and only one reached the top gain.

What is weight-space programming?

Models are normally adapted by training, which uses gradients to adjust parameters. In weight-space programming, a reasoning model inspects the parameters and writes changes itself. In this study, GPT-6 Astra at maximum reasoning effort got 50 forward requests and up to 300 Python analyses per search, with no gradients or optimizers. It picked its final checkpoint using development data only.

The team ran 16 trajectories across seven benchmarks on two Alibaba Qwen3.5 models, with 0.8 billion and 9 billion parameters.

Qwen weight editing results by benchmark

The largest math gain came from one column of a layer-23 MLP projection, where 1,023 of 1,024 entries changed. Two reruns on the same task changed 1,037,572 and 1,981,083 values and gained less, which the authors say shows the number of changed values alone does not predict results.

Why bigger models resisted

Qwen3.5-9B already scored 97.70% on MATH-500 before editing, leaving little room. A candidate math edit looked strong in development, rising from 54.69% to 84.38%, but slipped to 97.35% on held-out questions. Astra kept the original weights.

Six runs ended that way: two on the 0.8B model and four on the 9B model. The authors note that development results were selected adaptively and not corrected for multiple comparisons.

One edit, several skills

Without further changes, the math-edited 0.8B checkpoint gained 3.03 points on GPQA Diamond, 12.18 on IFEval and 20.10 on SciBench, reaching 21.48% on SciBench. That is higher than the 18.49% from the edit aimed at SciBench itself. SciCode did not move, and the authors report that some transfers caused regressions.

How this compares with the MIT result

The MIT CSAIL post reported that Astra lifted the same 0.8B model's MATH-500 score from 7.9% to 48.1% by changing 131,072 weights, mainly by fixing reasoning loops. The Shanghai paper's baseline was 8.35%, so the figures are not an exact like-for-like comparison. Its authors note the MIT team saw limited success on a model without such a clear failure.

What's next

The authors say the mechanisms behind successful edits remain unclear and that the method has not been systematically tested on longer reasoning or complex agentic tasks. They published a project page and a GitHub repository with the paper.

FAQ

Does weight editing replace fine-tuning?
The study does not show that. Gains were large on a weak small model and modest or absent on the stronger one.

Did the edits overfit to practice questions?
Reported gains are on held-out test sets the editing model never saw, though results varied widely between runs.

References

• Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations — Hu et al., arXiv (submitted Nov. 6, 2026)
• Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations (full text) — arXiv HTML
• Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights — Takehi, Teoh, Zhang, Khattab, MIT CSAIL (Nov. 2, 2026)
• Exploring Weight-Space Programming — project page