Language models can be trained to detect when their activations are read

AIWill Depue, an OpenAI researcher, says the finding that models can learn to detect when their activations are read is "cool," but asks how it happens. He answers that the models were trained to do it, and calls for more investigation of this behavior in the wild.

AIWill Depue, an OpenAI-linked researcher, says he trained a model to detect when its activations are being read, and calls for more in-the-wild investigation of the behavior. The main post builds on a thread by Sauers, which claims language models can learn to detect activation reading.