Mobile DVA: A Single Video-Action Policy for Manipulation and Mobility

Many useful tasks span more than one workspace. Restocking a shelf requires a robot to retrieve an item, carry it to the right location, and position itself to place it among other objects. Along the way, the robot must account for what it is holding, the space around it, and where its arms need to reach. We teach a robot to perform these tasks with a single policy that controls both its arms and mobile base, using limited additional robot experience.

Mobile manipulation brings together perception, navigation, and object handling. Modular systems address these challenges through specialized components, combining learned perception with mapping, planning, and manipulation routines [2, 3]. Extending these systems to new tasks and environments can require additional engineering to update maps and tune task-specific planning heuristics. Learned policies offer another route, predicting actions directly from observations and, in vision-language-action models, language instructions [4]. However, learning coordinated manipulation and mobility can require substantial robot experience covering both behaviors across varied scenes. This raises a practical question: can we build on general web video and existing manipulation experience to learn mobile tasks with much less additional robot data?

We present mobile DVA, an extension of our Direct Video-Action model (DVA) [1] that combines manipulation and mobility in a single policy, without maps, SLAM, or a separate navigation planner. Our approach builds on web-scale video pre-training, learning to predict how scenes evolve across diverse activities and environments. This broad source of video exposes the model to object interactions and movement through environments, providing a foundation for both manipulation and mobility. DVA’s general formulation lets us extend this visual prediction capability to an additional action modality: mobile base control. An inverse dynamics model translates a shared predicted visual future into both arm and base actions.

We first evaluate mobile DVA on a commercial workflow combining bottle sorting and crate transport. The robot sorts bottles into crates, transports filled crates, and returns empty ones for continued sorting. We investigate how experience from stationary manipulation transfers to mobile manipulation by initializing DVA with stationary manipulation training. We find that mobile DVA achieves 95.1% success using task-specific training data, only 30 minutes of which involve base movement. Removing initialization from static-manipulation training reduces success rate to 88.5%.

We then study data efficiency and robustness on a separate tray-relocation task. We find that with just 15 minutes of mobile robot data, mobile DVA achieves 88.3% success in a new environment with unseen obstacles across three novel layouts. Mobile DVA is also robust and reactive to changes during execution, adjusting its route around dynamic obstacles and its placement approach when the destination moves.

Together, these results show how we can expand our robot policies beyond stationary manipulation to combine navigation and manipulation in a single policy, enabling robots to take on a broader range of tasks.

DVA consists of two components: a video model that predicts how a scene will evolve, and an inverse dynamics model that translates those predictions into robot actions.

We first pretrain the video model on general web video, learning to predict future frames from a history of observations. This data captures object interactions, movement through environments, and changing viewpoints across diverse settings. These patterns provide a foundation for mobile manipulation, where the model must predict how both the objects being handled and the surrounding scene change as the robot moves.

Next, we learn an inverse dynamics model from robot data to translate video into executable actions, and adapt the video model to robot tasks through posttraining. This separates learning how the world changes from learning the robot commands needed to produce those changes, allowing DVA to draw on broad video experience alongside robot demonstrations.

At deployment, the two models run in a closed loop several times per second. The video model predicts a short sequence of future frames conditioned on a long history of robot observations, and the inverse dynamics model translates that predicted sequence into actions. As the robot acts, new camera observations update the context for the next prediction, keeping its behavior grounded in the changing physical scene. We use leapfrog inference [1] to overlap prediction with execution, generating the next visual future while the robot carries out the current action sequence.

DVA’s general formulation lets us extend the same prediction-and-action framework to an additional action modality: mobile base control. Mobile DVA retains causal video prediction and extends the inverse dynamics model to control base motion alongside arm motion. A shared predicted visual future therefore guides both manipulation and mobility, without maps, SLAM, or a separate navigation planner.

This visual future captures the combined effects of arm movement, object movement, and the changing viewpoint as the base moves. The inverse dynamics model translates that sequence into coordinated arm and base commands. Both are produced at each control step, allowing the robot to reposition while reaching, carry an object through the environment, and adjust its approach during placement without an explicit switch between navigation and manipulation modes.

We initialize the inverse dynamics model from stationary manipulation training and fine-tune it to control both arm and base motion. We use this initialization in all experiments, reusing existing manipulation experience when learning mobile tasks.

We first evaluate mobile DVA on a commercial workflow that combines precise manipulation with transport between workstations. The robot sorts glass bottles into a crate, carries the filled crate to another table, and brings back an empty crate for continued sorting. Completing this sequence requires precise bottle handling, coordinated arm and base movement, and tracking progress from observation history to determine when to transition between sorting and transport.

For this workflow, we train on task-specific data, only 30 minutes of which involve base movement. We investigate whether prior stationary manipulation experience improves performance by comparing two video-model initializations: one that has undergone stationary manipulation posttraining after general web-video pre-training, and one initialized directly from general web-video pre-training. Both models are trained on the same commercial-task dataset, and both use an inverse dynamics model initialized from stationary manipulation training.

Across five evaluation trials, mobile DVA achieves 95.1% success with prior stationary manipulation posttraining of the video model, compared with 88.5% without it—a gain of 6.6 percentage points. It also reduces full-task completion time by 2.2% relative to the baseline. These results show that stationary manipulation experience can improve performance on a workflow combining manipulation and mobility, even with limited task data involving base movement.

To examine how performance changes with the amount of mobile robot data, we turn to a simpler task: picking up a tray from one table, carrying it to another, and placing it down. We collect a separate set of teleoperation demonstrations for this task and train models with 15, 30, and 60 minutes of mobile robot data.

We evaluate each model under two conditions:

• New lab. We move the robot to a different part of the lab, changing the visual background, and test three obstacle layouts unseen during training. Each layout contains two to five novel obstacles with varied shapes, colors, and sizes.
• Factory. We introduce a larger visual change by surrounding the robot with four walls displaying factory backgrounds and replacing the carpets with a type unseen during training. We retain the same three obstacle layouts used in the new lab area.

Both conditions introduce greater clutter and a wider variety of obstacles than the robot encountered during training.

For each model, we run 20 trials per layout, totaling 60 trials per condition. Each trial consists of picking up the tray, transporting it along one route, and placing it on the destination table. We report the fraction of trials completed successfully.

With 15 minutes of task data, mobile DVA achieves 88.3% success in the new lab area, generalizing to unseen obstacles, greater clutter, and novel layouts. Increasing the dataset to 60 minutes raises success to 91.7%, suggesting that relatively little task data is sufficient for this evaluation.

The factory setup is more challenging, particularly with 15 minutes of training data. Success increases from 76.7% to 88.3% as the dataset grows to 60 minutes. The gap between the two evaluation conditions narrows from 11.7 to 3.3 percentage points, indicating a larger observed benefit from additional task data under the more substantial change in scene appearance.

The primary error mode involves returning the tray to the table from which it was picked up, rather than placing it on the destination table. The two tables are visually identical, which may contribute to this confusion. This failure pattern highlights a remaining challenge in maintaining the distinction between source and destination as the robot moves through the scene.

Beyond evaluating performance in unfamiliar layouts, we test whether mobile DVA can respond when the scene changes during execution. Using the same tray-relocation task, we introduce obstacles while the robot carries the tray and move the destination table, requiring it to replan its route or placement approach after the task has begun.

Mobile DVA responds to changes during execution by adjusting its route around newly introduced obstacles and its placement approach when the destination moves. These examples show that it can revise its ongoing behavior as the scene changes, continuing toward task completion even when the original route or placement location is no longer valid.

Mobile DVA brings manipulation and mobility into a single video-action policy. A shared predicted visual future guides both arm and base motion, allowing the robot to handle objects, transport them between workstations, and respond as its surroundings change. Building on general web-video pre-training, we learn mobile tasks with limited additional robot data and find that prior stationary manipulation training improves performance on a commercial workflow. These results demonstrate how we can build on existing manipulation experience to learn tasks that require movement through the environment.

This extension also illustrates the flexibility of DVA’s general formulation. Looking ahead, we aim to expand its multimodal capabilities through additional sensing and action modalities, giving robots richer information about their surroundings and more ways to interact with them. Combining these capabilities could enable increasingly complete workflows, from retrieving and transporting materials to unpacking, sorting, and replenishing supplies across a facility. Our goal is to expand both what a single robot can do and where it can work, while building on prior experience to make new capabilities easier to learn.