trash research. guys please read it for fucks sake
to spell it out (takes 5 seconds to ask your claude). my keyboard is fucked please excuse
- Nothing is compute matched! The models add code layers which add compute, they never ablate anything or spending it the normal way. 0 ablations in this paper
- they chose to run their experimenta on the SMALLEST nanochat config, the one karpathy calls GPT-1 sized, with 12 layers. the 286M params it has… 176M of thide are lookup tables, the model only has 85M active params. the normal config is 24 layers and 1.5B params with 0.5B trainable! they train one GPT-1, 5k steps, and the only result is a val loss curve
- they make a whole thing about their controls (a "twin" run with the layer off, "sham" layers with scrambled inputs) and then never show a single number from any of them. like you described the check, did it pass…
- same with the main tool (the "phantom gradient"). they never run the thing WITHOUT it to see if it even matters, and they literally admit it "does not predict reliably" how a layer will do once trained
- sliding window: the model is built so it literally can't see more than 500 tokens back. then they bolt on a lookup that can, and shocker, it remembers stuff further back. no full attention baseline, which would've just worked. are you genuinely mentally retarded
- one of the nanochat layers feeds the network its own recent errors. their guard against it peeking at future tokens is a footnote with no results. no evidence it leaks, but i'm not convinced the check would catch it either, and there's no code to look at
- CLRS: the whole point of gradient descent is it learns stuff we don't already have a prior on. here the agent "already knows how to solve the tasks" (their words) and writes layers that compute exact answers. of course it does better!!!!! try it on something nobody knows the answer to. and the one experiment that's supposedly like that (nanochat) is the val loss curve from above. also again this model has less than 100M params
- deepseek: it's a probe that reads the model's internals and throws away answers it thinks are wrong, then resamples. basically best-of-n with a learned grader. literally consensus does this just fine along with 10 other things. nothing is trained, despite the title. correctness probes are already a known thing and there's no comparison to a standard one, so where's the "redesigning itself" part
- and the only retry baseline is "retry the longest answers". no majority vote, no random retries at the same rate, no dumb logprob-based resampling. lots of things that aren't literally consensus work like consensus and boost accuracy, you have to beat those
- qwen: qwen writes a python rule for when to stop/restart/accept, using probes some other frontier model built. 2.8x the thinking, no majority vote comparison, even though one of its inputs is literally how often the answer repeats. again consensus with extra steps
- HMMT result is 63 problems, no error bars, no significance test. 7 points is like 4 or 5 problems
literally nothing in this post shows llms redesigning themselves. you guys are so cooked its insane