This is a pretty cool idea:
Let your agent simulate each potential action before picking the best one.
The model is PAN 2, a world model from the Institute of Foundation Models (IFM). They call this "simulative reasoning".
The attached video shows an example with a surfer: same wave, three different moments to pop up. PAN 2 plays out all three, and only one of them catches the wave, so that’s the one he goes with.
The model uses Generative Latent Prediction (GLP), and here is how it works:
• In: the current scene plus an action, with how long to simulate
• Out: what happens next, as a video you can check
There are two reasoning parts:
1. Conceptual reasoning: The LLM is the backbone here. It reasons about what happens next.
2. Perceptual reasoning: A 29B LDM fills in what it looks like.
This is all open under Apache 2.0!
They provided the weights, training code, and inference code, so you can run it and build on it yourself.