Anthropic finds many transcoder features copy single MLP neurons

Anthropic reports that many active transcoder features in late layers are close copies of single MLP neurons, across 13 open-source transcoders.

According to the authors, these neuron-aligned features are the only ones with a stable link to the underlying MLP. Other transcoder features vary more between training runs and do not match the MLP's Jacobian.

The authors argue that MLP neurons are a stronger starting point for interpretability than usually assumed. They say non-neuron transcoder features need better validation before they are used to explain model behavior.