Leslie Kaelbling’s IROS 2026 workshop talk, “Small models for big worlds.” Leslie always gives contrarian views.
Here’s a summary of her take on why compositionality helps, why Astra is so good at robotics, and why robotics is still not solved. https://x.com/i/article/2119749100769775617
Small Models for Big Worlds: Notes on Leslie Kaelbling’s IROS 2026 Talk
Leslie Kaelbling gave a talk at the IROS 2026 workshop on compositional robotics in Pittsburgh on September 27, called "Small models for big worlds." She opened by giving it a second title: "Robotics is not solved." Her point: robotics still has hard open problems, and compositionality and modularity may help with them.
I enjoy Leslie's talks a lot. She typically gives contrarian views in her talks, and every word counts (using her own words, "like a program"). This talk also very concisely describes why compositionality/modularity are helpful in just first 10 minutes. I'll try to share a faithful note below.
Top photo: Leslie Kaelbling (MIT), right, beside the podium, with her title slide on the left. All photos in this post are mine.
Modules are good and bad
She started with vocabulary, because "world model" gets used for two different things:
- World state model: what's going on in the world right now.
- World transition model: how the world changes.
- "We keep saying 'world model' and then I don't know what we mean."
Five things modules are good for, and two costs.
- What modules give us:
- you need less data to learn;
- you can swap a component;
- you can adapt one part quickly;
- you can run just the piece you need;
- you can check each piece before you check the whole.
- Two costs:
- Someone has to write the input and output spec for each module. That someone could be another learning process.
- The boundaries can get in the way of optimizing across modules: "maybe that just keeps me from finding this really cool trick that somehow blurs the boundary."
Compositional is more than modular
Her definition of a compositional system.
For her, a compositional system has:
- Many small pieces you can combine, even recursively: "I can make infinite combinations of a bunch of finite things."
- Systematic meaning: the meaning of the whole follows from the parts and how they're put together.
- Classic examples: natural and artificial languages. Transformers were designed to learn to exploit that structure in language.
Where are the seams?
Candidate seams on her slide: objects, chunks of space, agents, qualities, and meta-qualities of modules such as certainty, computation time, memory and energy.
Then the big question: "What are the seams in the physical world?" In other words, where can we pull a problem apart so that learning and reasoning get easier?
- Her list:
- objects, chunks of space and other agents;
- qualities of things, from physical to economic and political;
- properties of the modules themselves: how certain they are, and how much time, memory and energy they need.
- Her advice: "look at your problem, and try to find the places where you can pull it apart into chunks."
Left: a world state model, with objects, rooms and other agents; it can be uncertain. Right: a world transition model built from factored rules over objects and their properties.
Both kinds of world model can be compositional:
- Transition model:
- How one part of the state changes after an action mostly doesn't depend on how the other parts change.
- It also composes in time, which, as her slide notes, is what planning uses.
- State model: you can think about each person in a room more or less on their own, and about this room apart from the next one.
A modular system that plans
Her slide introducing TiPToP by Will Shen, Nishanth Kumar and colleagues (the slide spells it "TipTop").
Next, a recent system from her group:
- Old hat that works. It builds a simulator from an image and calls a planner. In her words, "kind of old hat, but it actually kind of works."
- Why it generalizes. "it's actually running a planner that actually understands what's going on in the physical world."
- Real physics. It plans with "the actual physics in the world." One task she showed needs looking ahead: move a can out of the way, then pack the coffee pods.
Why is Astra so good at robotics?
Four reasons, which she said are in order of importance.
People are freaking out that Astra can do everything. She joked that maybe we should all just go to the beach. Her guess at why it's so good, most important first:
1. A compositional model of the world, in natural language. "Natural language tells us what the compositional structure of our world is, because humans developed it to talk about the world efficiently."
1. Recursive composition of tool calls, like a program that calls subroutines.
1. Good tools from roboticists. "It's also good because of us": we've built great general tools, like inverse kinematics, that it can call.
1. A lot of robotics data in training.
Robotics is not solved
What an intelligent autonomous robot still needs.
Then she came back to her second title. An intelligent autonomous robot needs to:
- reason with representations suited to the physical world;
- be "resource rational" with computation, because it can't compute everything all the time;
- run parallel processes at several time scales;
- keep working when it's cut off from the internet, GPS, rescue operators and so on;
- keep persistent memory about people, places and things;
- know what it doesn't know, and go gather information;
- act rationally under risk.
"These are all things that I think we have not yet worked out."
Recent work from her lab
Three recent projects from her lab at MIT.
Then three recent projects that, as she put it, fill in some of the gaps Astra doesn't cover. Each one carries a view:
- Learn constraints, not trajectories. "Nobody should learn actual trajectories from demonstration." (RAPID learns a robot program from one demonstration.)
- Look before you act. "Planning with only the partial information that you have and acting as if that were true doesn't work reliably." (VA-TAMP on her slide.)
- Infer the goal behind a demonstration, not just the motion. A vision-language model suggests goals people tend to have, and a planner checks which one explains what the person did. (Rational Inverse Reasoning.)
Her summary: to solve a big problem with a small brain, "take the big problem and turn it into a sequence of small problems."
Architectures, from 1959 to 2026
Five slides from her short history of robot architectures. Years as on her slides.
Near the end, she walked through a few representative architectures.
- Five or six years ago: robot architectures were almost all end-to-end. That was okay for a while, "but not really."
- 2025, agentic systems: their diagrams look just like her own architecture, turned on its side.
Then she went back in time:
- 1959, Pandemonium (Oliver Selfridge): many processes running in parallel, "fighting with each other and competing," and maybe together solving the problem.
- 1981, Albus NASREM (as on her slide): "hierarchical architectures operating at different time scales, which I think is an important idea that we should be thinking about."
- 1985, subsumption (Rodney Brooks): small modules that take over from each other.
- 2026, Retriever: she mentioned our framework, "an architecture that takes seriously the idea that things happen in time." My co-author Jiayuan Mao presented it earlier in the workshop.
Her last architecture slide, "2026: Retriever!", with the system diagram from our paper and the author names. Project page, paper.
My take: It's great to see Leslie put our work in the lineage after the classic hierarchical robot architectures! I'm pretty happy that Leslie likes the work.
She closed the history with: "We have to all start taking seriously the idea that our computation happens in time and figure out how to manage that."
From the questions
- What if the scene isn't made of objects, or the task doesn't split into steps?
- "There are going to be some parts of your problem that don't divide up like that and you should not divide them up."
- The point is to "use all the levers that you can in the problem that you have."
- What if shape completion isn't enough, and you don't know in advance which features you need?
- Treat "computation actions as being actions too," alongside physical and observation actions.
- Compute grasps on this object, not on the chair over there, and learn a meta-reasoning strategy that picks which computation to run.
- Do we need simulation or real experience?
- Both, offline and online. "You don't know the world model for this remote until you pick it up and start messing with it."
- But a robot also needs simulation (not necessarily accurate, and at different levels) to imagine what would happen.
- What if the system is completely unknown?
- Learn incrementally: notice simple links between actions and changes, name a state variable, learn a small factored model, and build up from there.
- "Super hard, but probably incrementally if there's structure in the world."
- With limited processing, what should a robot focus on across time scales?
- Fast loops can only do a little computation; slow loops can do more, but only about things that aren't time-critical.
- "I honestly don't know. I just know it's important."
My take
- As Leslie says, it comes back to compositionality, and to using the structure in the problem. Many problems don't need search at every level: when I cook, I don't search over my finger joints; I search over how much salt to add. The key is where to split the hierarchy.
- I think the ideas behind robot planning aren't appreciated enough. People are interested; the ideas just aren't very approachable. Lowering that barrier is one of the core reasons for my research in the past two years.
- I try to make minimal assumptions about the real world, so the work applies widely. That makes the scope large, so the framework has to be very easy to use. Long-horizon systems need a careful balance between the assumptions, the problems you tackle, an approachable way to formulate them, and a general solution.
What I keep thinking about
- What problems are we solving?
- What do new approaches, like a big model in an agentic loop, truly bring to those problems?
- Which tasks need reasoning, and where should we split the hierarchy?