Goodfire says neural networks are far more interpretable than early researchers thought

AISummary

Goodfire says AI models are far more interpretable than previously believed, and that this is good news for interpretability.

Nick Cammarata, who co-authored the early Zoom In paper on polysemanticity and superposition, now argues that the vast majority of neurons in models up to trillions of parameters are legible.

He says ambitious interpretability may be solved in time, possibly soon, as automated scientists improve.

AIWhy it matters

The essay revisits early interpretability work and explains why its authors now believe neural networks are more legible than expected, which matters for alignment planning.

Post on XView on X

Goodfire

@GoodfireAI

For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that's the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.

Goodfire's @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there's reason for optimism today, and why there might be a bitter lesson for interpretability.

Nick@nickcammarata

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works https://already-clean.goodfire.com/

View quoted post on X

Source: Goodfire · x.comPublished Oct 15, 11:45 AM ET

Goodfire says neural networks are far more interpretable than early researchers thought

Topics