We investigated this with both theory and experiments. Probes can break, but predictably: the concept usually moves rather than disappears. In our language model experiments, retraining a probe was enough to recover the signal.
Goodfire finds retrained probes recover signals that break in language models
AIGoodfire reports that probes for language models can break, but usually the concept moves rather than disappears. In its language model experiments, retraining a probe was enough to recover the signal.
Source details
Goodfire · x.comPublished · added here
Original title: We investigated this with both theory and experiments. Probes can break, but predictably: the concept usually moves rather than disappear...