I’ve been exploring the infrastructure behind Physical AI and what it takes to teach robots to interact with the real world.
Here’s my breakdown of how @InvLambda is approaching it.
#invertedlambda https://x.com/i/article/2117793557893267456
Physical AI’s Fragmented Data Infrastructure: How Inverted Lambda Is Connecting the Pieces
A robot can recognise a cup, describe its shape and identify what it contains.
But can it pick it up?
Can it adjust its grip when it slips?
Can it perform the same task in a different environment?
These may seem like simple tasks, but they’re complex for machines.
Teaching AI to understand and interact with the physical world requires more than visual recognition. It requires experience
The next leap forward for Physical AI may not come from a bigger model or a more advanced robot. It may come down to the data used to train these machines.
Introducing Inverted Lambda.
The Problem Isn’t a Lack of Infrastructure
When discussing the challenges surrounding Physical AI, data is often the first thing mentioned.
Robots need demonstrations. They need to learn how objects behave, how environments change and what happens when they perform certain actions. This requires information from human demonstrations, robot movements, sensors and even failed attempts.
However, the conversation often stops at “we need more data.”
In reality, we already have a growing collection of datasets, research frameworks, teleoperation systems, simulations and evaluation tools. The problem is that they often operate across different platforms, hardware and formats.
A dataset may contain useful demonstrations but lack the environment needed to reproduce them. A teleoperation system may capture useful robot movements without providing an easy way to evaluate them. A simulation may be powerful but disconnected from the environment where the original demonstration took place.
The issue isn’t simply a lack of data. It’s making that data usable, transferable, reproducible and connected.
Research has already started showing what’s possible.
Useful Research
Open X-Embodiment
Open X-Embodiment is a collaboration between multiple research institutions to build a large-scale robot demonstration dataset and training framework.
It combines over 1 million real-world robot trajectories across 22 robot embodiments, bringing together 60 datasets from 34 labs, with 21 research institutions involved.
This work produced the RT-X models.
Traditionally, robot learning systems are trained on demonstrations specific to particular robots or tasks. This can make transferring learned skills between different hardware and environments difficult.
Open X-Embodiment explored whether training across different robots and tasks could improve transfer.
One particularly interesting result came from its emergent-skills benchmark.
Skills from the Bridge dataset, collected using a different robot embodiment (WidowX), were tested on Google Robot. These skills weren’t part of the original RT-2 training data.
The 55B RT-2-X model achieved a 75.8% success rate, compared with 27.3% for the 55B RT-2 model. When Bridge data was removed from the RT-2-X training mixture, its success rate dropped to 42.8%.
This suggests that demonstrations from other robot platforms helped RT-2-X acquire skills it hadn’t previously learned.
However, RT-2-X didn’t outperform RT-2 in a separate evaluation involving unseen objects, backgrounds and environments.
That distinction matters. The results demonstrate specific cross-embodiment skill transfer, not universal generalisation.
The main takeaway is that how data is collected and combined can significantly affect what a robot learns.
DROID
Another useful contribution is DROID, a large-scale open dataset containing approximately 76,000 robot trajectories and 350 hours of interaction data.
It includes synchronised camera footage, robot states and actions.
This is important because robot demonstrations aren’t just videos. The relationship between what the robot sees, how it moves and the actions it performs is essential for training.
DROID demonstrates the value of collecting real-world interaction data in a structured and consistent way.
ABC-130K
The ABC project provides another perspective on scaling bimanual manipulation data.
Its dataset contains approximately 130,000 episodes and over 3,500 hours of bimanual manipulation data, collected using robotic stations with two 6-DoF YAM arms.
Camera footage, robot states and gripper information were recorded, with data stored in formats such as MCAP.
The wider XDOF platform also explores data collection, simulation and evaluation.
Together, these projects highlight an important direction in robotics: building systems that make physical interaction data easier to collect, organise and use.
Inverted Lambda
Inverted Lambda describes its mission as building open, decentralised infrastructure for Physical AI.
According to its official website, its product roadmap includes three key components:
- Teleoperation Platform: Collects robot demonstrations and physical interaction data.
- Physical Agents SDK: Connects real-world environments to simulation and robot development through Real2Sim.
- Video Evaluation Engine: Evaluates robot recordings and simulation benchmarks.
The interesting part is how these components could work together.
Demonstrations provide information about the actions a robot performs. Reconstructed environments can help reproduce the conditions in which those actions took place. Evaluation can then help determine whether the robot completed a task successfully and identify where it failed.
Together, these components point towards a connected workflow rather than simply another source of robot data.
However, the product roadmap is a starting point, not proof that the entire workflow is already integrated or independently validated.
Questions remain about hardware support, technical specifications, evaluation methodology and independent benchmarks.
From Demonstration to Training Data
Collecting data doesn’t automatically make it useful for training.
Recordings may contain irrelevant frames, inconsistent movements, failed attempts or actions that don’t match how the target robot operates.
A useful data workflow could look like this:
Collect → Process → Retarget → Evaluate → Prepare Dataset → Train
Inverted Lambda’s products appear to address several parts of this process
The Teleoperation Platform provides a way to collect demonstrations.
Those recordings then need to be processed and prepared before they can become useful training data.
Retargeting is also important when transferring demonstrations between different robot systems. A movement recorded using one robot may need to be adapted before another robot can reproduce it.
Evaluation adds another layer.
A robot might appear to complete a task in a video but still miss an important instruction. It might move an object without placing it in the correct position, or succeed in one environment and fail in another.
This is where the Video Evaluation Engine comes in.
Its stated purpose is to evaluate robot recordings and simulation benchmarks. A useful evaluation process should identify whether a task was completed, where mistakes occurred and whether the recording meets the required criteria.
But evaluation is only as reliable as the criteria behind it.
How are tasks defined? How automated is the process? Which benchmarks are supported?
These are important details that will need to be documented and demonstrated.
The goal isn’t just to collect more demonstrations, but to create datasets that can actually help robots learn.
Physical Agents SDK
One of the more interesting aspects of Inverted Lambda’s infrastructure is the Physical Agents SDK.
A policy trained in simulation may not work as expected in the real world. Differences in geometry, camera placement, lighting, object properties, friction and physical contact can all affect the outcome.
Real2Sim attempts to address this gap.
Instead of building an entirely new environment, the idea is to use information captured from the physical world to recreate that environment in simulation.
Inverted Lambda states that its SDK can collect environmental information and reconstruct environments used for deployment
The proposed workflow includes physical observations and device assisted capture. The project has mentioned using robot cameras, sensors and LiDAR, with phones and tablets providing another collection method.
However, a visually accurate reconstruction isn’t automatically a physically accurate simulation.
A useful Real2Sim pipeline needs to account for geometry, object positions, robot configurations and physical properties so that simulated interactions behave meaningfully.
Research such as Agentic Real2Sim demonstrates how complex this process can be. It explores extracting scene geometry, object states, robot movements and physical parameters from real-world recordings to build simulations.
This isn’t an Inverted Lambda product benchmark, but it illustrates the technical challenge involved.
If environments can be recreated reliably, developers could test policies, reproduce failures and create additional training scenarios without constantly repeating expensive physical experiments.
The real question is how accurately and reliably those environments can be reconstructed.
Opportunity – Creating an Economy
Useful robot data doesn’t have to come exclusively from research laboratories.
It could come from teleoperation operators, robotics researchers, hardware owners, industrial environments and individuals recording everyday physical interactions.
But coordinating these contributions requires infrastructure.
Hardware and software need to work together. Recordings need to be verified, processed and evaluated. Data also needs to be made available in formats developers can actually use.
This is where the decentralised aspect becomes interesting.
Imagine a robotics researcher collecting manipulation demonstrations in a laboratory, a robot owner scanning warehouse environments and an individual recording robot interactions through a teleoperation system.
Rather than relying on one central organisation to collect everything, a decentralised network could bring these contributors together.
They could contribute demonstrations and environments, while the infrastructure provides ways to organise, validate and distribute the resulting data.
Inverted Lambda has also discussed incentives for robot owners and individuals contributing environments through the Physical Agents SDK.
This creates the possibility of an economy built around physical experiences, allowing people with access to different robots and environments to participate.
However, decentralisation alone doesn’t solve the underlying problems.
Data still needs standard formats, quality checks, validation, licensing and reliable access. Contributors also need transparency about how their submissions will be used and what makes a contribution valuable.
Otherwise, the result could simply be a larger collection of fragmented recordings.
Competitive Landscape
Several projects are already addressing different parts of this problem.
- Open X-Embodiment combines robot datasets and explores cross-embodiment training.
- DROID provides large-scale real-world manipulation data with synchronised recordings.
- XDOF / ABC focuses on large-scale bimanual data collection alongside simulation and evaluation.
- LeRobot provides an open-source ecosystem of datasets, tools, models and standardised workflows, with hardware integrations.
- Figure Index is a proprietary data collection system focused on Figure. The company reported over 16 million uploaded videos, more than 44,000 weekly active contributors and $15 million paid to contributors. Its workflow includes filtering, fraud checks, deduplication, rebalancing and annotation.
These projects aren’t directly interchangeable, but they demonstrate how much attention the industry is giving to organising physical interaction data.
Inverted Lambda’s potential distinction is its attempt to connect data collection, environment reconstruction and evaluation within one infrastructure.
Whether it can deliver that effectively will depend on execution and adoption.
Challenges
There are several factors that could determine how successful this approach becomes.
1. Quality over Quantity
More recordings don’t necessarily mean smarter robots.
Data needs to be relevant, correctly recorded, diverse and suitable for the training objective. Failed attempts can also be useful, provided their outcomes are properly understood.
2. Interoperability
Physical AI involves different robot embodiments, sensors and control systems.
A workflow that works with one robot may require significant adaptation for another. Supporting different systems will be important if the infrastructure is intended for wider use.
3. Realism
Recreating how an environment looks isn’t enough.
Simulation also needs to account for contact, friction, dynamics and object behaviour. Otherwise, a robot may perform well in simulation but fail in the physical world.
4. Evaluation
A video evaluation system is only as reliable as its criteria.
Tasks need clear definitions, assessments need to be consistent, and results should help developers understand why a robot failed.
5. Accessibility
Infrastructure only creates value when people actually use it.
Contributors need an accessible way to participate, while developers need straightforward access to useful datasets that fit their research and training workflows.
Conclusion
Physical AI is moving towards a more iterative approach.
Robots collect demonstrations. Those demonstrations inform training. Models are tested in simulation and on physical hardware. Failures reveal new situations that need to be captured, creating more data for future learning.
The infrastructure supporting this process may become just as important as the models themselves.
Research already shows the value of larger and more diverse datasets, cross-embodiment training and better connections between real-world recordings and simulation.
The challenge is bringing these capabilities together.
If Inverted Lambda can provide a coherent, accessible workflow for collecting demonstrations, reconstructing environments and evaluating robot behaviour, it could play an important role in this space.
But its success will depend on how well the technology works across different systems, how reliable its data is and whether developers actually adopt it.
Final Thoughts
The future of Physical AI won’t be decided by models alone.
A robot needs to learn how to interact with the world, understand how physical environments respond and recover when reality doesn’t match its expectations.
That comes down to experience.
And experience depends on infrastructure.
We already have useful datasets, research frameworks, data collection systems and simulation tools.
The challenge is connecting them.
This is why Inverted Lambda is worth keeping an eye on.
While models may be the visible part of the revolution, the systems that teach them how to operate in the real world could be what makes it possible.
@InvLambda