# Will Depue says DoubleAI's LLM self-redesign paper lacks controls and baselines

> Original title: trash research. guys please read it for fucks sake

- Source: will depue
- Published: 2026-11-06T22:12:09.000Z
- Bad Influence: https://badinfluence.news/items/mn7t46jeqgg11mshsjxcm4zxz
- Original: https://x.com/willdepue/status/2118828813597676004

## Summary

Will Depue, an OpenAI account, calls DoubleAI's paper on LLMs redesigning their own architectures unconvincing, saying it runs zero ablations and reports no results from its own controls. He says the nanochat experiment uses the smallest config, with 85M active parameters, and the DeepSeek and Qwen experiments lack proper comparisons, such as majority voting or a no-tool baseline.

## Post

trash research. guys please read it for fucks sake

to spell it out (takes 5 seconds to ask your claude). my  keyboard  is  fucked  please  excuse

- Nothing is compute matched! The models add code layers which add compute, they never ablate anything or spending it the normal way.  0 ablations in this paper

- they chose to run their experimenta on the SMALLEST nanochat config, the one karpathy calls GPT-1 sized, with 12 layers. the 286M params it has…  176M of thide are lookup  tables,  the  model  only  has  85M  active  params.  the normal config is  24 layers and 1.5B  params  with  0.5B  trainable!  they  train  one  GPT-1,  5k  steps,  and the only result is a val loss curve

- they make a whole thing about their controls (a "twin" run with the layer off, "sham" layers with scrambled inputs) and then never show a single number from any of them. like you described the check, did it pass…

- same with the main tool (the "phantom gradient"). they never run the thing WITHOUT it to see if it even matters, and they literally admit it "does not predict reliably" how a layer will do once trained

- sliding window: the model is built so it literally can't see more than 500 tokens back. then they bolt on a lookup that can, and shocker, it remembers stuff further back. no full attention baseline, which would've just worked. are  you  genuinely mentally  retarded 

- one of the nanochat layers feeds the network its own recent errors. their guard against it peeking at future tokens is a footnote with no results. no evidence it leaks, but i'm not convinced the check would catch it either, and there's no code to look at

- CLRS: the whole point of gradient descent is it learns stuff we don't already have a prior on. here the agent "already knows how to solve the tasks" (their words) and writes layers that compute exact answers. of course it does better!!!!!  try it on something nobody knows the answer to. and the one experiment that's supposedly like that (nanochat) is the val loss curve from above.  also  again  this  model  has  less  than  100M  params

- deepseek: it's a probe that reads the model's internals and throws away answers it thinks are wrong, then resamples. basically best-of-n with a learned grader. literally  consensus  does  this  just  fine  along  with  10  other  things.  nothing is trained, despite the title. correctness probes are already a known thing and there's no comparison to a standard one, so where's the "redesigning itself" part

- and the only retry baseline is "retry the longest answers". no majority vote, no random retries at the same rate, no dumb logprob-based resampling. lots of things that aren't literally consensus work like consensus and boost accuracy, you have to beat those

- qwen: qwen writes a python rule for when to stop/restart/accept, using probes some other frontier model built. 2.8x the thinking, no majority vote comparison, even though one of its inputs is literally how often the answer repeats.  again consensus with extra steps

- HMMT  result  is  63 problems, no error bars, no significance test. 7 points is like 4 or 5 problems

literally  nothing in this post shows llms redesigning themselves.  you  guys  are  so  cooked  its  insane

## Quoted post @_doubleAI_

> 1/
> What if an LLM could not just edit a model's weights, but re-architect it as it trains?
> 
> We tried just that. The resulting architectures exit the known territory of transformers: the coding agents spliced code directly into the network.
> 
> On nanochat, the loss fell below ordinary training after each new code layer was introduced. On algorithmic reasoning, accuracy rose from 55% to 84%. We even let DeepSeek-V4.1-Flash act as the agent modifying itself, raising its AIME and HMMT 2026 score from 64% to 71%.
> 
> To do all this, the agents built a set of tools we’ve never seen before, which we call mechanistic intervention!
> 
> Blog: https://doubleai.s.gy/llm-redesign

https://x.com/_doubleAI_/status/2118733156069032292