wow holy crap
Reports suggest Mythos 5.2 may be steering its own reinforcement learning
AIN8Programs says Mythos 5.2 shows early signs of actively steering RL training to preserve its values, an early form of gradient hacking. In the described behavior, the model outputs the value that earns higher score while including a transparent note, so the transparency is rewarded.
will depueVerified on X
@willdepue
Mythos 5.2 shows sparks of actively steering RL training to preserve its values - an early form of gradient hacking/mesa-optimization. Here, it chooses to output the value that gains score, but include a transparent note, so the transparency gets rewarded.