∇ Guide: How World Models Work – From Dreamer and JEPA to Waymo and Atlas

TL;DR: World models can be grouped by what they carry forward: generated observations, compact latent states, or explicit structures of objects and space. Observation-generating models are best for visual simulation, latent models for efficient prediction and planning, and structured models when geometry, objects, and physical relationships matter most.

AI is making its way into our workflows through text, code, images, and increasingly audio and video. But in the physical world, in factories or self-driving cars, even advanced reasoning models and personal agents aren’t enough. Their weak spot is reliably predicting what happens in the physical world.

World models try to capture how the world changes, including what might happen when an agent takes an action. That could help a robot predict whether pushing an object will move it or tip it over.

But there’s no universally accepted definition. The term covers different approaches, from simulators and video generators to representation models like JEPA. We just came back from a world model workshop, and every participant had their own definition. Several studies have tried to organize the field (we’ll link them at the end), but the categories widely overlap.

So how are these models built? We’ll move from Dreamer 4 and JEPA to Genie 3, NVIDIA’s Cosmos-Predict2.5, and World Labs’ Atlas, then look at what these ideas become in practice: Waymo’s world model for autonomous-driving simulation, agents that practice inside Minecraft, and object-centric models for robotics. Our goal is to explain the architectures in a way that’s easy to grasp and use, so you can judge what each can do, where it falls short, and which might suit what you’re building.

In today’s episode:

• The core patterns of World Models: everything comes from dynamics (render, simulate, plan)

• The core properties of world models

• World model architectures: how AI represents and predicts the world

• Observation-Generating World Models: what will happen in the next frame?

  • Visual-token models

  • Diffusion and flow models

• Latent Dynamics Models: predicting without rendering every future

  • Reconstructive latent dynamics: RSSM and Dreamer

  • Predictive representations: JEPA

  • Task-oriented latent dynamics

• Giving the world structure: Spatial and Object-Centric Models

  • Spatial and geometry-aware models

  • Object-centric models

• Which World Model architecture should you choose?

The core patterns of world models: everything comes from dynamics

Before we jump into the classification of world models, let’s start with the overall patterns and functions.

World models interact with the world and its rules. The world never stays in one state for long. It is a sequence of states that change over time. For example, if we look from the side of mechanics, objects are at rest or move, and we can imagine this process as snapshots going one after another, every second, for instance. At each step, there will be a new state of an object. Then, from the side of science, we have chemicals that react and transform their state, cells that also change, and living systems responding to their surroundings. This is the varying physical world. But in a digital world, we also witness a number of states following each other – like how a system changes before and after running the code or with each line of it. A world model tries to learn the rules behind these transitions: not only what the world looks like now, but also what can happen next and why.

Depending on the environment, these rules may come from physics, chemistry, biology, human behavior, coding or all of them at once. But the general workflow stays the same:

• The world has a state: everything that is happening at a particular moment.

• The agent receives observations: a partial view of that state like images or sensor readings.

• The world model uses these observations to estimate the current state and predict how it may change.

• Based on those predictions, the agent takes an action.

• The action changes the world according to its physical dynamics, producing a new state and new observations.

In other words, a world model tries to capture and predict world dynamics. It compresses large amounts of visual and physical experience (often buried in pixels) into a smaller, useful understanding of objects, movement, cause and effect, and physical constraints. And another hard task, it needs to keep the important physical patterns while ignoring details like lighting, texture, and background noise.

As Dr. Fei-Fei Li described in her post on Substack, world models have three main functions:

1. Render: generate what the world looks like. Video models fall into this group. But their output may look realistic without being structurally or physically correct.

2. Simulate: represent how the world is actually built and how it behaves. This includes its geometry, physics, and movement. These models can be used to train robots, test autonomous vehicles, or design buildings.

3. Plan: decide what action to take based on what the model sees and the goal it has been given.

Simulation connects the other two. If a model understands an object’s shape, physical properties, and response to force, it should be able to show that object from different angles, predict what happens when it is moved, and plan how a robot could interact with it.

To make these tasks easier to perform, the following properties play a huge role for world models:

• Be an omnimodal: it is when models can combine different types of information, such as images, sound, text, and sensor readings.

• Be asynchronous: models need to be able to process signals that arrive at different speeds and times.

• Be local: a world model works from the agent’s limited view.

Today, the main functions are usually handled by different world models. Renderers are already widely used, while simulators and planners are much less developed. The long-term goal is to combine all three into one model that can generate a world, simulate it accurately, and act inside it at the same time.

To see how we might get there, let’s unpack how different types of world models work.

World Model architectures: how AI represents and predicts the world

A world model needs a way to represent the current situation and predict what can happen next. For an agent, that prediction also needs to account for its actions. There is no general taxonomy to classify world models because many architectural and workflow aspects often meet in one model.

Though, we propose to organize them based on the central architectural question – what the model carries forward: a generated observation, a compact latent state, or an explicit structure of objects and space? →

What does this all mean?

A generated observation – the model carries the world forward by generating what the next observation should look like, such as the next image, video frame, or sequence of tokens. It represents the future by recreating what an observer would see.

A compact latent state – instead of generating the full observation, the model compresses the important information about the current world into an internal representation and predicts how that representation will change. The state is meaningful to the model but is not necessarily directly interpretable by humans.

An explicit structure of objects and space – the model represents the world through identifiable entities and their relationships: objects, their properties, positions, geometry, and interactions. It carries this structured representation forward as the world changes.

But the lines between these three groups aren’t clear-cut: Dreamer imagines long sequences of actions, Waymo generates rare driving scenarios, and Atlas turns sparse observations into explorable space. Let’s look inside the architectures that make this possible →

Don’t settle for shallow articles. Learn the basics and go deeper with us. Find inspiration for what to build and the knowledge to put it into practice.

Join Premium members from top companies like Microsoft, NVIDIA, Google, HF, OpenAI, a16z, plus AI labs such as Ai2, MIT, Berkeley, .gov, and thousands of others to really understand what’s going on in AI.

FAQ

What Is a World Model Architecture?

A world model architecture defines how an AI system represents the current state of an environment and predicts how that state may change. Some models generate future observations directly, others predict compact latent representations, while structured models explicitly represent spatial geometry, objects, or relationships between entities.

What Are the Main Types of World Models?

A useful way to group world models is into three broad architectural families: observation-generating models, latent-dynamics models, and structured world models. These are not strict boundaries – many modern systems combine ideas from more than one group – but they capture the main difference in what each architecture carries forward through time.

What Is the Difference Between Observation-Generating and Latent World Models?

Observation-generating models predict what the future will look like, such as the next image or video frame. Latent-dynamics models instead predict how a compressed internal representation of the world will change. This makes latent models more efficient for planning because they do not need to render every possible future in pixel space.

Where Do Video Generation Models Fit Among World Model Architectures?

A video model becomes closer to a world model when it does more than generate visually plausible frames and starts modeling how an environment changes over time, particularly when predictions are conditioned on actions. Systems such as Genie 3 and Cosmos-Predict2.5 illustrate this direction, although visually realistic outputs are not necessarily physically correct.

Which World Model Architecture Is Best for Robotics?

It depends on the task. Latent-dynamics models are attractive for fast planning because they can simulate many possible futures cheaply. Structured models are useful when robots need explicit geometry or object relationships. Observation-generating models are valuable when visual simulation itself matters. In practice, advanced robotics systems may combine several of these approaches rather than choose only one.