World Model Training Needs Footage From Human Eyes

World Model Training Has a Camera Problem. Human Eyes Fix It

The model had watched more than a million hours of video.

It could guess what a cook would grab next in a busy kitchen clip, and it could answer questions about motion and timing. Planning a single movement for a robot arm was beyond it. Nothing in that million hours told it which movement caused which change.

The researchers behind Meta’s V JEPA 2 then gave it less than 62 hours of robot footage. That small slice taught the model what an action is. The million hours had only taught it what the world looks like.

Hold on to that ratio. It points to where the next jump in world model training is coming from, and to why the camera that matters most sits a few inches above a pair of working hands.

Here’s my take after reading the big results from the past year and a half. Plenty of world model data exists as video. The scarce part is video where you can see the action behind the change.

Robot footage carries that signal, but it’s the slowest and most expensive data you can buy. Buy more of it to fix generalization and you pay the most for the least variety.

Since early 2026, a run of studies has pointed somewhere cheaper. Egocentric human video carries much of that action signal, at a scale no robot fleet can match.

By the end of this piece you’ll be able to place your own data mix on one simple dial, and you’ll know which way to turn it.

What a World Model Has to Learn That Internet Video Never Shows

A world model answers one question. Given what the agent sees now and the action it takes next, what will the world look like a moment later? Planning depends on that answer, and so does testing a robot policy before it touches anything real.

This stopped being a lab curiosity a while ago. In April 2026, MIT Technology Review described world models moving to the centre of AI research.

Yann LeCun left Meta to start a world model company, and OpenAI shifted resources toward long term world simulation work. The money is here. The right data still isn’t.

Watching Is Not Acting

Internet video teaches a model what things look like, how objects fall and how scenes usually unfold. That’s real knowledge. A stock clip, though, rarely shows who caused a change or which movement did it.

Hands sit far away, half hidden or out of frame. It’s like learning to ride a bike from the blimp camera. You see the whole race and never the pedals.

That’s why V JEPA 2 needed its own robot slice before it could plan anything.

Robot Footage Has the Signal and Lacks the World

Teleoperated robot data shows the exact action, joint by joint. It’s also slow to gather and narrow by nature, since one team can only drive so many robots through so many rooms.

The DreamDojo authors, a group from NVIDIA, HKUST, UC Berkeley and others, put a number on that gap. Their human video corpus covered about 96 times more skills and 2,000 times more scenes than the most diverse public robot datasets.

The Distance Dial: Where Your World Model Training Camera Sits

Every clip in your dataset came from a camera standing somewhere, and that spot decides what your world model can learn from it. I find it helps to picture a dial with four stops.

The Stands

Almost every large video corpus lives here today. Breadth and physics come cheap. On cause, the footage says nothing.

The Sideline

Third person recordings show the whole body and the shape of the room. Contact is where they fall short, because the grip happens behind someone’s wrist.

The Eyes

For pretraining, the 2026 evidence favours this stop. Egocentric video keeps the hands, the object and the result in one frame. How much it’s worth depends on what the camera kept in view and how carefully someone filtered and labelled it.

The Translator

Your robot’s own camera comes last, in small doses. V JEPA 2 used under 62 hours, and DreamDojo used a small set of target robot data. Keep this stop small on purpose.

Most teams discover their footage sits mostly in the Stands. If that sounds like yours, this guide to first person video data shows what eye level footage looks like in practice.

What Happens When the Camera Moves to the Eyes

Here’s the evidence, newest first. Each result put egocentric human video up against something tougher than a hunch, and each one points the same way for world model training.

Head to Head With Robot Data

In June 2026, the HumanScale team ran a fair fight. They held the amount of pretraining data fixed and changed only the source.

Models pretrained on egocentric video showed 24% lower loss when predicting real robot actions. Compared with models pretrained on teleoperated robot data, they also reached 52.5% higher success on familiar tasks and 90% higher success on unfamiliar ones.

One caveat matters. The study is still a preprint, and the authors credit a careful filtering and labelling pipeline for the result rather than raw footage.

A World Model Built From Human Eyes

DreamDojo, presented at ICML 2026, pretrained a robot world model on 44,711 hours of egocentric human video. At the time, nobody had used a larger corpus for world model pretraining.

Most of that footage had no action labels. The team taught the model to infer hidden actions, then post trained it on a small set of target robot data. The authors report strong results on unseen objects and rooms, plus real time speed after distillation.

More Hours, Predictable Returns

EgoScale, from NVIDIA and partners in February 2026, trained on 20,854 hours of action labelled egocentric video. It found a log linear scaling law between hours of human data and validation loss, and that loss tracked real robot results.

To be fair, EgoScale trained a robot policy rather than a world model. It still shows that returns on human footage follow a curve you can plan around.

Industry Moved Too

In September 2025, Figure reported that its humanoid could walk through cluttered real homes from spoken commands after training only on egocentric human video. That skill used no robot specific data at all.

What Moving Your Data Mix Toward the Eyes Is Worth

Better generalization where it hurts. HumanScale’s biggest gain, 90%, came on unfamiliar tasks. Those are exactly the cases that break a demo in front of a customer.

Variety without a robot fleet. Human footage reaches kitchens, workshops and storerooms no robot has visited yet.

A smaller robot budget, spent better. Robot hours no longer carry bulk pretraining and move into a focused Translator slice instead.

Budgets you can forecast. With a scaling law, “how many hours do we need” stops being a guess and turns into a plan.

A world model you can rehearse in. The DreamDojo authors list live teleoperation, policy evaluation and planning inside the model as uses. Each rehearsal there saves one risky trial on real hardware.

The change feels personal for the people doing the work, too. Research leads stop begging for robot hours they can’t get. Data leads get a clear spec to hold vendors to, and engineers lose fewer late nights to bugs that were really data gaps.

How to Turn Egocentric Video Into World Model Training Data

Raw footage is not a dataset. Here’s how to get from one to the other, in the order I’d tackle it.

Frame for the Hands

Keep the critical moment of each task in frame: the grasp, the pour, the twist, the release. When footage drifts to the ceiling during the grip, your world model learns nothing about cause from it.

Spend on Rooms, Not Repeats

Your tenth kitchen teaches more than your hundredth take in the first kitchen. Set a variety budget across rooms, objects, lighting and people before you set an hours target.

Label the Act and the Object

Mark where each action starts and ends, what changed and what the person meant to do. Then label what happened to the object as well as the hands.

An October 2026 study trained world models on 30,000 hours of egocentric video. The models captured the person well, while object behaviour lagged far behind. Object state labels help close part of that gap.

Filter Before You Scale

HumanScale’s win rested on its filtering and labelling pipeline. Treat validation, layered QC and human review as part of the data itself, because they decide what reaches training.

Keep the Paper Trail

Consent, provenance and network level verification protect your model when legal or a customer asks where its knowledge came from. Weighing capture options? This piece on sourcing egocentric data at scale compares body cameras with verified crowdsourcing.

Save a Slice for the Translator

Reserve a small block of footage from your own robot for the final alignment step. Then test on rooms the model has never seen. Honestly, that’s the only score that tells you how it will do in the field.

Can a World Model Learn From the Stands Alone?

Fair question, and the honest answer comes in two parts.

First, new methods can pull third person footage closer. In May 2026, the EgoExo WM team converted ordinary third person video into egocentric views and used it to train a whole body action conditioned world model. Prediction and planning both improved.

Their reasoning stands out. Egocentric data is scarce, and it only shows part of the wearer’s body.

I’d read that as the Sideline moving closer. The Eyes still matter, because converted footage needs real egocentric video to learn what the view from the eyes should look like.

Second, the eyes have limits too. The October 2026 study above found that 100 times more egocentric data brought person modelling close to its ceiling while object dynamics trailed.

The authors concluded that closing the gap depends on how teams train models, not only on how much footage they see. Piling on hours won’t fix it by itself. You need better labels and better training as well.

The dial itself won’t change as models grow. The robots, rooms and tasks will. Policy models follow the same logic, as this guide to VLA training data shows.

Four Routes to Egocentric World Model Training Data

Here’s how the main sources of egocentric video data for world model training compare. I’ve ranked them by how much finished, trainable signal you get per hour of footage.

1. Humyn Labs

It takes the top spot because it runs the whole pipeline, from sourcing first person footage through validation, layered QC and annotation. You receive labelled egocentric data instead of raw clips.

Its live catalogue lists first person video and world model triplets next to speech data. It also takes custom requests built to your spec, and most custom collections ship in four to six weeks.

Every annotation passes peer review and centralised checks. Safety critical work gets an extra domain expert review layer.

Deliveries come with provenance documentation and inter annotator agreement metrics, so you can see how consistent the labelling was. The footage comes from a verified contributor network, with checks at the network level.

That one process covers every problem raised in this piece, from hands in frame and clean action boundaries to object labels and a paper trail.

Why it matters: you spend your time training the model instead of cleaning the footage.

2. Open Academic Egocentric Datasets

Open research datasets give you a fast, free way to start testing ideas on egocentric video.

Ego4D, released in 2021 by Meta and a group of university partners, set the standard here with thousands of hours of everyday first person video and a suite of benchmark tasks. Newer corpora keep arriving, and several studies in this piece built on public data.

Fit is the catch. Someone else picked the tasks, the rooms and the label schema to suit their own research rather than your robot.

Many research licences also limit commercial use, which means your legal team needs to read the terms before any of that footage reaches a product model.

Why it matters: you can prototype cheaply, though you’ll likely need your own data before launch.

3. An In House Capture Programme

Running your own capture gives you full control of the spec, from camera position to label schema.

Figure took a version of this route with Project Go Big, partnering with a large property owner to record people moving through real homes. You decide what gets filmed, who reviews it and what the labels look like.

The hard parts are yours too. Recruiting, consent, hardware, privacy review and QC staffing all land on your team, and then there’s the long tail of rooms you simply can’t reach.

Most programmes stay narrow because the people and places within reach run out long before the variety budget does.

Why it matters: you get control, and you pay for it in time your researchers could spend on the model.

4. Scraped or Stock Video

Scraped and stock video gives you the most hours for the least money. It also gives you the least action signal per hour.

This is the Stands. V JEPA 2 showed how much a model can absorb from over a million hours of internet video, including appearance, motion and the usual flow of a scene.

That footage carries no action labels, though. Hands rarely sit in frame at the moment of contact, and rights often stay unclear.

Use it for breadth in early pretraining if you can clear the licences. Just don’t expect it to teach your world model what its own actions do, because nothing in the clip says who did what.

Why it matters: you save money up front, then pay it back in robot hours later.

Want to see eye level footage for your own task before you commit? Ask for a sample slice at humynlabs.ai/syod.

World Model FAQs

What is a world model in AI?

A world model is an AI system that predicts how an environment changes when an agent acts. Robots and agents use it to plan, test choices and rehearse safely before they move in the real world.

What data do you need to train a world model?

Most world model training starts with large video collections for appearance and motion, then adds data that shows actions and their effects. Today that action signal comes from egocentric human video plus a small slice of robot footage.

What is egocentric video data?

Egocentric video is footage recorded from the eyes of the person doing a task, usually with a head worn camera. It keeps the hands, the objects and the result of each action in one frame.

Can human video replace robot data for a world model?

Not fully. Human video can carry most of the pretraining, and a June 2026 study found it beat robot data at equal scale. You’ll still need a small amount of robot footage to map that learning onto your machine.

How much egocentric video does a world model need?

There’s no fixed number. Published work ranges from tens of thousands of hours for pretraining down to under 62 hours of robot data for alignment. Variety of rooms, objects and people counts for more than raw hours.

Which egocentric data source is most reliable for world model training?

The most reliable source validates, labels and documents footage before you ever see it. Humyn Labs datasets meet that test, because peer review, central QC and provenance records come with every delivery. Open datasets still work well for early prototypes.

Move the Camera, Then Move the Robot

Every world model inherits the seat its camera sat in. Models trained from the stands describe the game well and play it badly.

The teams pulling ahead in 2026 did three things. They moved the camera to the eyes of the person doing the work, and they cleaned and labelled what it caught. They also kept a small slice of robot footage to translate the result.

None of that waits on a bigger model. It starts with one decision about your next dataset: where the camera sits, and who checks what it caught.

Ready to move your world model training data closer to the action? Talk to Humyn Labs about a camera and labelling spec for your next dataset.

About the author: [Author name] is [role] at Humyn Labs, where the team sources, validates and annotates first person and multimodal data for physical AI.