Steering GPT-6 Astra to edit 0.016% of Qwen3.5-0.8B's weights can fix the infinite loops and raise math ability by up to ~6x and instruction following by ~3x
Researchers at MIT CSAIL have discovered a new capability called "Weight-Space Programming": without gradients or training loops, a general frontier model (GPT-6 Astra) is given a natural-language goal and directly reads or edits the numerical weights of a neural network. Early results cover "recognizing behavior," "fixing faults," and "writing small policy networks from scratch." After editing about 0.016% of Qwen3.5-0.8B's weights, MATH-500 rose from 7.9% to 48.1%, and instruction following improved as well. The authors argue that small symbolic optimizers like Adam and Muon embed coarse human priors, and the next step may be LLMs deliberately reasoning over the weights themselves.
https://rikiyat.github.io/blog/weight-space-programming/
How is this different from current practice?
People have long had LLMs edit prompts, harnesses, training code, and hyperparameters, often on the grounds that "text is hard to climb with gradients." Weights have been left to differentiable optimization or to specially trained "weight-writing" networks. This paper argues that inference-driven optimization has reached into the weights of differentiable networks, sometimes with no environment feedback at all.
Reading: guessing behavior from weights alone
· Four PPO policies with the same architecture (17,253 weights, a 10×10 grid, the same action set): Astra guessed from the raw weight text that they were Breakout / Pong / Pac-Man / Freeway, relying on cues such as which row the first-layer weights concentrate on and how the output layer divides up the actions.
· Three small diffusion models (circle / square / triangle): without running a forward pass, it identified the shapes from the weights. The outlines it drew suggest it was reading geometric structure rather than secretly running generation.
· Three 7.2M-parameter character-level language models, with the input and output layers removed so only the middle four Transformer blocks remain: it still distinguished Python / romanized Japanese / English, correctly flagged all the copying (induction) heads without mislabeling other heads, and in its reasoning arrived at something similar to the weight-composition analysis of Elhage et al. on its own.
Editing and writing
· Kinetix hill-climbing car: PPO was stuck in the local optimum of "drive straight up the hill." In one reply Astra edited about 3.5% of the weights, raising success over 1024 episodes from 0% to 100%. It added a bypass that reads position and velocity, reverses to build momentum, and then hands control back to the original policy.
· Three MinAtar games: with no data, no gradients, and no environment rollouts, it wrote the weights in one shot. Its scores beat a PPO trained for 100 million steps, and on Breakout it was an order of magnitude higher than a PPO trained for 500 million steps. The idea was to encode a "program" into the layers: for example, first compute where the ball will land, then drive the paddle there.
· A 9×9 Go network with about 1.48 million parameters: the weights cannot fit into the context, so it used a Python sandbox to read and write the tensors. With zero forward passes it beat a smaller PPO; with 50 forward passes (up to 128 practice games per batch) it beat a PPO of the same size trained on about 20 million games. The style resembles hand-writing a small program: liberties, dangerous chain-groups, captures, rescues, and forbidden eye-making moves, then revising the corresponding parts according to practice failures.
Scaling up to language models
· Sycophancy detection: for two Gemma 3 270M models fine-tuned under identical settings (embedding and output layers frozen), looking only at the weights could not tell which one was sycophantic. After allowing inspection of attention patterns on self-written text, it picked out the sycophantic model in all three trials and pointed out that middle-layer heads attend more to hints in the user's message about the expected answer.
· Fixing the thinking loops of Qwen3.5-0.8B: the model card already warns that this configuration is prone to looping. Astra worked over the weights in Python, using GSM8K/MATH practice problems for limited forward passes (10/25/50), located a direction written by the MLPs of layers 15 and 19, and subtracted it from 64 output columns each, editing 131,072 weights in total (0.016%). Before the 50 forward passes: MATH-500 7.9→48.1, GSM8K 13.8→67.1, IFEval 21.0→62.3, and the share of responses hitting the token limit fell from 93% to 33%.
· Control: on Gemma 3 270M, which loops less, failures were ordinary arithmetic mistakes and invalid algebraic steps, with no single obstacle that could be removed; small edits barely moved MATH-500. The authors point out that weight editing looks more like removing obstacles that block existing abilities, and does not yet amount to injecting capabilities the model lacks.
What they envision
Auditing hidden capabilities or backdoors without running evaluations; reducing training costs through one-shot initialization; redirecting or erasing abilities with little or no data while minimizing damage to the rest of the model; recursive self-improvement shifting from "writing code that launches training" to "editing weights directly"; and transplanting circuits across architectures, not just averaging within identical ones. They also acknowledge that context and harnesses are still clumsy; they are building benchmarks and harnesses to turn this "black magic" into reproducible engineering.