Hard lighting cut our expert-only π0.5 checkpoint's success roughly in half. The same lights cost the full fine-tune nothing.
New from @castorhat: the fine-tuning ablations behind our IROS demo. https://x.com/i/article/2112286738983116800
openpi / π0.5 Fine-Tuning: How-To and Tips
A walk through a number of ablation tests for a simple openpi fine-tuning run. Our data blend is proprietary so we can’t disclose any data ablations, but a number of training settings have a profound impact on checkpoint performance.
We collect a lot of data. When collecting data, it’s useful to occasionally train a model on that data to understand pitfalls in the dataset.
This is the fine-tuning work behind the π0.5 demo we ran at IROS 2026. If you missed the show, here’s our IROS 2026 recap.
Methodology
π0.5 fine-tuning on 668 episodes (about 10 hours) of YAM pick and place data, followed by large-scale evaluation in a custom sim (more on that below).
Following recent trends, we adopt a workflow mostly driven by Claude Fable 5.1, where the model synthesizes the code and runs experiments under strict human supervision. We’ve found this to substantially improve productivity with no more errors than a human-driven workflow, while also freeing human researchers to focus on analysis and next steps.
The eval was a trivial pick and place scenario, representing the best case for the checkpoint:
Performance was evaluated on a few criteria: time to first pickup, percentage of successful pickups, and percentage of successful dropoffs into a container.
Use a Simulator
For simple tasks that don’t involve deformable objects, real-to-sim testing is a surprisingly valid approach. In general, we’ve found sim deployments to score higher than real deployments, which we attribute to the real rig having non-idealities (backlash, joint friction, camera misalignment and latency, …) that the sim doesn’t capture.
We had Claude roll a custom simulator (now open source as yamulator) using a simple MuJoCo contact model, a direct port of the control law from the I2RT deployment scripts, and an accelerated pathtraced renderer with realistic area lights and physically-based materials. This simulator allows us to run thousands of tests per day to ablate lighting geometry, clutter, rig errors, and more.
The simulator also brings VLA training to parity with other AI work in terms of automation. The agent training the model can now invoke the simulator to test rollout performance and autonomously run ablations to inform data selection. Rollout performance in sim is a much stronger proxy for real performance than eval loss, which helps the agent stop training early and avoid wasting GPU hours.
Ablations 1: Training Scheme
Pi is a weird model because it has a fat backbone and a tiny action expert (the expert is 10× smaller than the backbone). Training the backbone is 4× slower than just training the expert and puts a ton of memory pressure on your machine (in particular, it doesn’t fit in a 48 GB card) so we wanted to see whether it was worth it.
The answer is a resounding “yes”:
LoRA and full fine-tuning look neck-and-neck here, which is good for small labs training on RTX 5090, because LoRA fits in 32 GB and full fine-tune doesn’t. As we’ll see later, there’s more to it.
Ablations 2: The Infamous Lighting Problem
What happens if we change the lighting? VLAs are infamously not robust to lighting changes, or are they?
We run 30 trials in sim across a variety of lighting conditions, collecting the success metrics.
This is the nice part about having a physically-based renderer — adjusting the lighting was as simple as changing the light shapes in the scene. Everything else is emergent from the light transport algorithm.
The Expert-Only Checkpoint
This checkpoint exhibits the usual collapse. Performance is constant-ish down to 30 cm circular ceiling lights which cast noticeable but still soft shadows, but hard lighting (a 10 cm disk is representative of an undiffused bulb) causes a precipitous decline in performance.
The Full Fine-Tune
A miracle has happened: the same 10 cm disk lights which cause the expert-only checkpoint to lose 50% of its performance cost the full fine-tune and LoRA checkpoints nothing. Another huge win for the full fine-tuning.
Ablations 3: Questionable Inference Code
There are a lot of ways to roll out a checkpoint and not all of them are the correct ones. While rolling out this experiment we wound up with a bug in our inference scripts that inserted a 300 ms delay between action chunks; we were curious how this affected performance.
Taking our best checkpoints and looking at just pickups across 30 rollouts:
A small signal in favor of the LoRA checkpoint, probably attributable to the LoRA forgetting less of its pretraining data.
We also had a particularly cursed checkpoint trained on leader actions instead of follower actions (this would be necessary if, say, the follower ran an impedance controller). This one degraded to 0% success with the delay, which is…unfortunate and merits further investigation.
Ablations 4: Camera Placement
No one seems to respect the cameras. YAM comes close — at least you can download an STL to print yourself a RealSense bracket — but the other Chinese suppliers seem to leave it as an exercise to the user. Compounding this, we’ve seen a number of commercial Aloha-style rigs with adjustable camera angles — possibly the worst decision, virtually guaranteeing that rollout won’t match data collection. Let’s not even talk about the Huggingface/SO-101 ecosystem, where the position of the top camera is left as a mere suggestion to the user.
A lot of people seem to think camera positioning doesn’t matter and this ablation chart explains why:
The first column — whether the robot lifted the target at all — remains relatively constant across runs. If your task is simple with strong visual features, you might not notice camera jitter as well. The second column — whether the robot dropped the target into one of the two clear containers — degrades markedly with camera position jitter. This isn’t even a hard task — the containers are 20 cm wide — but even so, the way the robot learns the container positions is such that performance falls apart if the overhead camera is in the wrong spot.
The wrist cameras follow a similar pattern, but much more pronounced, probably because the gripper tips are obscured for large displacements.
Ablations 5: The Wrong Cameras
This one matters more than you think. Robotics cameras come and go, and data and rollout rigs might not support the same cameras (for example, you might do data collection on static arms connected to a PC and USB cameras, then roll out on a mobile rig with a Jetson and GMSL cameras).
We discovered, for our singular benchmark and singular rollout condition, that the overhead camera is extraordinarily sensitive to the FOV being too wide and the wrist camera to the FOV being too narrow. I wouldn’t read much into it — the asymmetric change in performance suggests that obscuration/visibility matter more than the absolute FOV, but on the other hand, it does show that camera intrinsics matter a lot, potentially degrading performance to zero if they’re off too much.
Stay tuned for future intrinsics studies.
Ablations 6: Backgrounds
We wanted to bring this checkpoint to an event and were wondering whether we could face the wrist cameras out into the event space (which is more visually appealing and interactive) or if we had to face a wall to prevent environmental distractions. In order to learn more about this, we assembled the following very goofy and unscientific test, replacing the rear wall of the work cell with a photo taken at a conference:
Happily, the results were in our favor across 150 rollouts (95% confidence interval in parentheses):
We attribute this to the strong backbone that is openpi, which contains ample web images from VLM pretraining as well as thousands of hours of data collected in real environments during robotics pretraining. The full fine-tune is markedly weaker in the cluttered environment and the expert-only checkpoint actually grows stronger with clutter, something which seems hard to explain but is supported by the data.
Takeaways
- Always train the VLM. At least for this model, a substantial part of its intelligence comes from the VLM and fine-tuning the backbone is critical for success. We didn’t ablate joint training versus a knowledge-insulation-style phase scheme but this would be a natural extension for next time.
- Lights and environments matter less than you think — but only if you train the VLM. Gemma/openpi are strong models with plenty of mixed vision data seen during pretraining.
- Demand the rig specs from your data supplier. This includes full rig CAD, control laws used during teleoperation, and SKU-reproducible camera specs. This matters a lot for post-training, a lot less for pretraining, but having it for your pretraining dataset beats being caught unawares later (we proudly provide individually-measured rig URDFs for each rig in the fleet, globally available camera SKUs, and standardized teleoperation code).
All numbers above come from rollouts in our custom MuJoCo-based simulator with a pathtraced renderer. The simulator is open source: yamulator.