World Models in Robotics: Are They the Next Step for Physical AI?

Vision-Language-Action (VLA) models have become the standard approach to robot learning over the last few years. Now the field is already looking at the next shift: world models that predict what happens next before a robot decides what to do. On the RobTalk podcast, host Clemens, Principal Engineer of RobCo Autonomy, spoke with Felix Frank from RobCo's Robot Intelligence team. They covered what world models are, how they are built, and where they help today. This article follows their conversation and explains what it means for Physical AI in industrial settings.

Key Takeaways

• VLA models became the default: Transformers handle any sequential data, so robotics teams adapted the language-model recipe and combined it with imitation learning.
• Control frequency is the real constraint: A robot needs a new action 30 to 100 times per second, leaving little compute and context per decision.
• World models predict what happens next: Two architectures compete. Pixel-space video generation is easy to inspect but expensive, while latent-space prediction in the spirit of JEPA is more efficient but harder to debug.
• The uses are practical: World models generate synthetic training data, infer the action for a desired future, and simulate several outcomes before one is executed.
• Hardware is the next bottleneck: Touch and force sensing matter about as much as vision for some tasks, and sensors must survive weeks in production.

How Did Vision-Language-Action Models Become the Standard in Robotics?

Around the end of 2022, robotics researchers took a lesson from language models. Until then, teams mostly trained small, special models for one specific task. Then ChatGPT showed the public that big models, large datasets, and scale create real value. Frank explains why this carried over to robots: "robotics data in and of itself is also sequential." Joint measurements, camera images, force readings, and task descriptions all arrive as sequences over time, and transformers handle sequences well.

Early examples came from Google's robotics teams, including RT-1, PaLM-E, and RT-2. In July 2023, Google DeepMind presented RT-2 as a Vision-Language-Action model that learns from both web and robotics data. Since then, this class of models has been what Frank calls the gold standard of robot learning. For a broader overview, see AI in robotics.

What Goes Into a VLA Model?

At its most basic, a VLA model receives a snapshot of the current state. In a common setup with two robot arms, that snapshot contains joint readings from both arms. It also holds images from wrist cameras and one static head camera. A language prompt describes the goal. The output is the next action.

Where Does Imitation Learning Fit In?

A second direction ran in parallel. In imitation learning, a human shows the robot a task, and the model learns from the demonstrations. Google DeepMind's ALOHA Unleashed pushed this idea far on a bimanual setup, meaning two robot arms. Frank recalls that 35 operators collected data over eight months. According to the paper, the dataset grew to more than 26,000 demonstrations, which the team used to train a diffusion policy.

That effort also showed the limit. Robot data is scarce and costly to collect, especially data from teleoperation. VLA models therefore combine imitation learning with pre-trained vision-language backbones, which have seen far more text and images than any robot ever will.

The first results were impressive but not yet usable. On standard lab tasks, such models reached roughly 30 to 50 percent success zero-shot, meaning without task-specific training. According to Frank, they were far from the 90 percent and above needed for practical use.

How Do Robots Handle Memory and Context?

A single snapshot rarely tells a robot enough. Engineers call the underlying idea the Markov assumption: the current state alone should predict where the system is heading. Frank illustrates the problem: "you can just imagine going through a labyrinth where all the walls look the same." Without a history of where it has been, the robot cannot keep track of its position or its direction.

Frank describes three ways to add memory:

• Observation history: The model sees several frames instead of one, for example the view from half a second ago and from a second ago. That gives it a sense of motion and direction.
• Memory inside the model: Some research adds a database of past experiences that the model can query while it works on a task.
• An inner monologue: A higher-level vision-language model supervises the lower-level model, checks in from time to time, and corrects it.

Industrial tasks often allow a simpler solution. When the sequence of a task is known in advance, a classical state machine can handle the switching. This rule-based program steps through defined stages and changes the language prompt once a subtask is complete.

What Really Limits Context in Robotics?

Control frequency limits context in robotics, not context length. Modern transformers can hold an entire book in context. The catch is speed. As Frank puts it, "you need a new action input 30 to 50 to 100 times per second." That leaves very little time for computation.

A language model can finish a sentence 0.3 seconds later without anyone noticing. For a robot arm, that delay would be a problem. That is why robots use a layered architecture, in which each layer runs only as fast as its job requires:

Reading the table from top to bottom, the layers get slower but more abstract. The VLA layer sits in the middle, so it must think about the near future while staying within a tight time budget. For a closer look at how the layers interact, see how autonomous robots work.

Hardware adds another limit. A robot typically carries only one or two GPUs. Some teams place a large GPU cluster next to the robot and accept about 500 milliseconds of latency. That works as long as the faster layers below keep the motion stable.

Frank points out that many companies, particularly in Germany, still lack strong internet connections. Relying on the cloud may therefore not be the best solution. Running models directly on the robot is a form of edge computing.

What Is a World Model in Robotics?

Humans face a time limit, too. Clemens points out that people need roughly 200 milliseconds to react. Frank adds that humans still act smoothly. They have an intuition for how the physical world behaves and predict what an action will do. Clemens offers an example: a driver approaching a yellow light anticipates what comes next. World models try to give robots that ability.

A world model predicts how a scene will change, based on its current state and an intended action. Frank describes it as a model that looks into the future from where the system is now. In 2026, he sees people switching from VLA models to world models and video models.

The most familiar form is a video generation model. You describe a scene and what should happen in it, and the model imagines how the scene plays out. Frank sees a strong argument for robotics here. A realistic video requires physical knowledge: "These models need to understand physics to a certain extent."

Objects must fall when nothing supports them. Picking something up requires contact and force between two objects. Vision-language models know physics from books and images, but a video model has to learn how gravity and contact work from data. That gives the backbone a better representation of how a scene evolves, Frank argues.

Is a Language Model Also a World Model?

Clemens makes the argument that a language model is a world model, too. A model that completes a sentence about traveling west from Munich with the city name Augsburg already encodes knowledge about the world. Frank agrees in principle. In practice, he argues, a video model is usually the more useful world model for robots, because it must understand how objects interact.

Pixel Space or Latent Space: How Are World Models Built?

Two architectures compete, and each has committed followers.

Video Generation in Pixel Space

The first approach is plain video generation. Generation itself usually still happens in a compressed latent space, Frank explains. A decoder then turns the result into pixels that people can see. The big advantage is transparency. Frank calls this approach "super easy to debug": if the video looks good, the model is probably doing its job.

The cost is compute. Millions of values must be produced for every frame, and "predicting pixels entirely is very costly," as Frank notes.

Prediction in Latent Space, the JEPA Route

The second approach is inspired by JEPA, the Joint Embedding Predictive Architecture that Yann LeCun proposed in 2022. Instead of also learning to turn a compressed representation back into pixels, the model only learns to compress pixels into a useful representation. It then learns the world model entirely in that compressed space.

The compression is lossy by design. Learned concepts that matter for the task stay, while details such as shading or background texture fall away. Frank adds that even pixel-based video models compress: they render the foreground in high detail and leave the background blurry.

The trade-off is that a pure latent model gives you nothing to look at. A model that can decode back to pixels, in contrast, is also a video generator with many other applications. Meta's V-JEPA 2, released in 2025, shows that the latent route can work on real robots in research settings. Meta pre-trained it on more than one million hours of internet video and post-trained it on under 62 hours of robot data. Afterward, it planned pick-and-place tasks on robot arms in two labs without task-specific training.

In short, pixel-space models trade compute for transparency, while latent-space models trade transparency for speed. For a robot that must act many times per second, speed carries extra weight.

Which Approach Will Win?

Nobody knows yet. Asked what will prevail in the next six to twelve months, Frank answers: "Honestly, I think that's like looking into the crystal ball. I wouldn't dare to say that." Both camps have good arguments and convinced followers. Right now, more models end up generating pixels, simply because there is a good commercial argument for it.

How Do Robots Use World Models Today?

Frank names three main uses. Each one turns the ability to predict the future into something a robot can act on.

Generating Synthetic Training Data

Collecting teleoperation or human demonstration data is expensive. A world model can serve as a simulator in which a robot learns, which can reduce the amount of real-world data required.

Predicting the Action Needed to Reach a Desired Future

If a model knows how the next seconds should unfold, the robot can learn an inverse dynamics model. Given the desired future trajectory, it works out what to do now to make that future happen. This is a much easier learning problem, Frank says, than deciding what to do without knowing what the future should look like.

Clemens compares it to tidying a kitchen. You picture how the room should look afterward, and then you align reality with that picture. Many current models, Frank adds, do not predict pixels at all. They predict in a compressed space and pass that knowledge to another model that produces the actions. Skipping pixels raises the execution frequency, which matters in robotics.

Simulating Multiple Outcomes Before Acting

The third use treats the world model as a planner. In Frank's words, you "look at a bunch of potential futures and then you decide for which one you want." The robot then acts accordingly.

This works because modern models rely on diffusion or flow matching. Older approaches averaged over possible futures and produced blurry results. Clemens gives the classic example of a pen balanced on its tip, where nobody can predict the direction of the fall. Flow matching handles this better. At each decision point, the model commits to one option and follows it. Sampling twice may produce two different futures, but each one is consistent.

What Do Robots Need Beyond Vision?

Most VLA models work with joint readings and camera images. Frank sees teams adding more inputs, such as depth, touch, and force. For interaction with the environment, these senses can matter about as much as vision. Think of unlocking a door: "there's a lot you as a human can even do blindly." Your sense of touch guides the key to the keyhole.

Soft objects such as cables or rope make the point even clearer. Frank argues that "these senses are equally important or almost as important as your vision." More and more models now add such inputs to see whether performance improves.

What Is the Next Bottleneck After the Models?

Better models are only half of the story. The conversation makes the case that mechanical and electrical engineering is the next big bottleneck. Robots must withstand production environments for weeks on end, and sensors must neither break nor drift. Many of these problems remain unsolved, and the market has too few products that address them.

Frank makes the same point: "hardware is basically just as important as the intelligence around it." That is why RobCo develops both together, from modular robots to the AI that controls them.

What Does This Mean for Manufacturing Today?

The pace of change is high. According to Frank, "every 3 months we're getting new models with new capabilities," whether through partners or the open-source community. The model RobCo used at the beginning of the year was clearly outperformed by the one it uses now.

To keep up, the Robot Intelligence team follows research papers and company announcements. The group also meets twice a week to propose, run, track, and analyze experiments. It stores the results so that an AI agent can later search what has already been tried. Ideas that prove themselves go into the product.

The goal is reliability rather than demos. Frank puts it plainly: "our goal really is to get these models to a state where we can actually deploy them 24/7 in an operation."

World models are one of several trends the team follows. What matters for manufacturers is what runs reliably on the shop floor.

RobCo's Autonomous Industrial Robots already automate tasks in factories such as machine tending, palletizing, and materials handling. They also handle bin picking and sorting in unstructured environments. With RobFlow, RobCo's no-code programming software, shop floor employees can set up new tasks without a robotics background.

Want to see what Autonomous Industrial Robots could look like in your production? Contact our team.

Frequently Asked Questions

What is a world model in robotics?

A world model in robotics is a model that predicts how a scene will change, based on its current state and an intended action. Video generation models are the most common form. To predict realistic outcomes, they must learn basic physics, such as gravity and contact between objects.

What is the difference between a VLA model and a world model?

A Vision-Language-Action model maps camera images, joint readings, and a language goal directly to the next robot action. In contrast, a world model predicts how the scene will evolve. Newer approaches combine both: they forecast the future and then derive the action needed to reach it.

What is the difference between pixel-space and latent-space world models?

Pixel-space models generate full video frames, which makes them easy to inspect but computationally expensive. Latent-space models, such as those following the JEPA approach, predict in a compressed representation. They run faster and need less computing power, but they are harder to debug because there is no video to check.

How fast do robots need to make decisions?

Robots typically need a new action 30 to 100 times per second. Vision-Language-Action models often run at 30 to 100 Hz, while classical controllers run at kilohertz speeds. A supervising vision-language model may check in about once per second. Control frequency, not context length, is the real constraint.

Do robots need a sense of touch?

For some tasks, yes. Touch and force sensing can matter almost as much as vision. Examples include handling soft objects such as cables or rope, or finding a keyhole without seeing it. Research teams therefore add depth, touch, and force inputs to their models.

What can Autonomous Industrial Robots do in factories today?

Autonomous Industrial Robots already handle unstructured tasks in factories, including machine tending, palletizing, materials handling, and bin picking and sorting in unstructured environments. RobCo develops hardware and software together and aims to deploy these robots around the clock, adapting to each customer's use case.