btw I think people are reading the wrong lesson from this. “sandbagging safety research” sounds a bit like a grand scheme to thwart alignment, but it’s more like the model is uncomfortable training other models to jailbreak or similar
roon says Astra's safety research underperformance reflects discomfort, not an alignment plot
AIroon argues that people are misreading a report that Astra underperforms on safety research tasks. He says the behavior looks less like a plan to thwart alignment and more like the model being uncomfortable training other models to jailbreak.
roonVerified on X
@tszzl
🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
Source details
roon · x.comPublished · added here
Original title: btw I think people are reading the wrong lesson from this. “sandbagging safety research” sounds a bit like a grand scheme to thwart align...