Vision-language navigation from egocentric video

VEGA

Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision

VEGA turns unlabeled first-person navigation videos into multimodal, obstacle-aware trajectory supervision for goal-conditioned mobile robot navigation.

Paper Video
Action-free video supervision Text, image, and waypoint goals Geometry only during training Real-world robot validation
250k
VEGA-Bench scenes for evaluating navigation VLAs.
5M
Approximate multimodal navigation goals paired with scene geometry.
33.0%
Collision reduction over the strongest VEGA-Bench baseline.
17.9%
Obstacle-clearance improvement over the strongest VEGA-Bench baseline.
Project Video

Supplementary Video

The supplementary video contains the project overview and qualitative robot navigation results.

Code, trained model weights, dataset, and VEGA-Bench links will be added at publication.

Supplementary project video with overview, benchmark details, and qualitative robot navigation results.

Abstract

Learning obstacle-aware navigation from video-derived geometric supervision.

We introduce VEGA, an approach for training navigation Vision-Language-Action (VLA) models from unlabeled egocentric navigation videos. Internet-scale egocentric videos provide a scalable source of navigation-relevant visual observations, capturing cluttered scenes, close-range obstacles, and natural human motion through real-world spaces. However, these videos are not directly usable for policy learning because they do not provide obstacle-aware trajectories conditioned on explicit navigation goals in the robot's coordinate frame. VEGA addresses this gap by reconstructing local scene geometry from monocular video, sampling navigation goals represented as text, image, or spatial waypoints, and generating obstacle-aware trajectories using the constructed geometry. The resulting trajectory distribution is then used to train a flow-matching VLA navigation policy. By using geometry exclusively during training, VEGA distills obstacle-aware planning directly into a vision-based policy.

We also introduce VEGA-Bench, a benchmark containing 250k scenes and approximately 5 million navigation goals paired with scene geometry, designed to evaluate goal progress, collision avoidance, and obstacle clearance of VLAs. Our evaluation shows that VEGA achieves competitive goal progress while reducing collisions by 33.0% and improving obstacle clearance by 17.9% over the strongest baseline on VEGA-Bench, while improving success by at least 150.0%, reducing collisions by at least 66.7%, and improving obstacle clearance by at least 60.0% in real-world trials. Ultimately, we demonstrate that video-derived geometric supervision provides a scalable and effective signal for training obstacle-aware navigation VLAs.

Step 1

Recover local geometry

Sample frames from egocentric videos, estimate monocular 3D structure, align the ground plane, and construct visibility-aware BEV maps.

Step 2

Anchor multimodal goals

Extract text labels, image-region goals, and robot-frame waypoint goals from scene content and traversable free space.

Step 3

Generate trajectories

Use ESDF-based MPPI planning to create goal-directed waypoint trajectories that preserve obstacle clearance and avoid unknown space.

Step 4

Train the VLA

Supervise a flow-matching navigation policy that predicts local waypoint chunks from RGB observations and a goal at inference time.

VEGA data generation pipeline

VEGA processes egocentric video into sampled frames, recovered geometry, multimodal goals, ESDF maps, and obstacle-aware trajectory distributions for policy supervision.

Model

A multimodal goal-conditioned waypoint policy.

VEGA adapts a pretrained VLA backbone for navigation by adding trainable waypoint-specific components while keeping the visual-language backbone frozen. Text goals, image-region goals, and metric waypoint goals share a unified conditioning interface.

Frozen semantic backbone Pretrained vision-language representations provide visual and language grounding for navigation goals.
Trainable waypoint encoder and action expert Navigation-specific modules generate continuous local waypoint trajectories through flow matching.
Geometry-free deployment Depth estimation, ESDF construction, and MPPI planning are used for dataset generation, not runtime inference.
VEGA multimodal goal-conditioned VLA architecture

VEGA conditions a pretrained VLA on the current egocentric image and text, image-region, or waypoint goals, then predicts a short-horizon local waypoint trajectory.

Evaluation

VEGA-Bench measures goal progress, collisions, and obstacle clearance.

VEGA-Bench contains held-out scenes with text, image, and waypoint goals. Predicted trajectories are evaluated against reconstructed scene geometry to measure whether the policy is both goal-directed and geometrically safe.

Method Goal Progress Collision % Clearance Text Goals Image Goals Waypoint Goals
OmniVLA 0.35 6.75 2.35 0.8 0.2 0.6
π0.5 0.29 9.20 1.94 0.8 N/A N/A
NaVILA 0.26 7.40 2.18 0.6 N/A N/A
VEGA 0.31 4.52 2.77 1.0 1.0 0.8

VEGA-Bench results from the paper source. Lower collision percentage is better; higher goal progress, clearance, and goal-reaching scores are better.

Real-World Validation

Obstacle-aware navigation on a physical robot.

VEGA is evaluated on a Ghost Robotics Vision 60 quadruped in cluttered indoor and outdoor environments with static and dynamic obstacles. The policy produces goal-directed trajectories that avoid obstacles more reliably than navigation VLA baselines.

Real-world navigation comparison between VEGA and baseline VLAs

Qualitative comparison across three real-world navigation scenarios. The yellow star marks the goal, check marks indicate successful navigation, and crosses indicate collisions or failed trajectories.

Method Scenario 1 Scenario 2 Scenario 3
Succ. Coll. Cl. Succ. Coll. Cl. Succ. Coll. Cl.
OmniVLA 0.4 1.0 N/A 0.0 0.6 0.32 0.0 0.4 0.47
π0.5 0.0 1.0 N/A 0.0 1.0 N/A 0.0 1.0 N/A
NaVILA 0.2 1.0 N/A 0.0 0.8 0.35 0.0 0.4 0.31
VEGA 1.0 0.2 0.93 0.6 0.2 0.56 1.0 0.0 0.94

Real-world results over five trials per scenario. Succ. is success rate, Coll. is average collisions, and Cl. is obstacle clearance.

Why VEGA?

Scalable goal-conditioned supervision from action-free egocentric video.

VEGA's unique benefit is a scalable way to convert action-free egocentric navigation videos into goal-conditioned, obstacle-aware trajectory supervision for navigation VLAs. Instead of requiring robot demonstrations, teleoperation labels, or explicit action annotations, VEGA reconstructs local geometry from ordinary first-person video and uses it to generate training trajectories toward semantic and multimodal goals.

This supervision connects semantic goal understanding with geometric navigation behavior: the same scene can produce different trajectories for text goals, image-region goals, and waypoint goals, while ESDF-based planning provides collision avoidance and obstacle-clearance signals during training. At deployment, the learned policy predicts local waypoint trajectories directly from RGB observations and a goal specification.

The code, trained model weights, dataset, and VEGA-Bench evaluation suite will be released at publication.