LoTIS: Learning to Localize Reference Trajectories in Image-Space for Visual Navigation

Finn Lukas Busch, Matti Vahs, Quantao Yang, Jesús Gerardo Ortega Peimbert, Yixi Cai, Jana Tumova, Olov Andersson
Division of Robotics, Perception, and Learning at KTH Royal Institute of Technology

Robotics: Science and Systems (RSS), 2026

arXiv Paper Code 🤗 Demo Experiments

TL;DR: Given a reference video recorded with, for example, a phone, LoTIS allows a robot with an onboard RGB camera to navigate to a selected point along the recorded trajectory from a visually overlapping initial view, without robot-specific training.

Abstract

We present LoTIS, a model for visual navigation that provides robot-agnostic image-space guidance by localizing a reference RGB trajectory in the robot's current view, without requiring camera calibration, poses, or robot-specific training. Instead of predicting actions tied to specific robots, we predict the image-space coordinates of the reference trajectory as they would appear in the robot's current view. This creates robot-agnostic visual guidance that easily integrates with local planning. Consequently, our model's predictions provide guidance zero-shot across diverse embodiments. By decoupling perception from action and learning to localize reference-trajectory frames rather than imitate behavioral priors, we enable a cross-trajectory training strategy that learns robust invariance to viewpoint and camera changes. We outperform state-of-the-art methods by 20-50 percentage points in success rate on forward navigation, and paired with a local planner we achieve 94-98% success rate across diverse sim and real environments. Furthermore, we achieve over 5x improvements on challenging tasks where baselines fail, such as backward traversal. The system is straightforward to use: we show how even a video from a handheld phone camera directly enables different robots to navigate to any point on the trajectory.

Method

LoTIS encodes the full reference trajectory once offline. For every query image, lightweight online inference matches the query against the encoded trajectory, while a recurrent prediction head localizes each reference-trajectory frame as image-space coordinates, normalized distance, and visibility.

The yaw controller directly uses the predicted image-space coordinates, normalized distances, and visibility. For MPPI, the predicted image locations are first projected into 3D using estimated depth.

LoTIS reference-trajectory predictions overlaid on the robot's live camera image
  1. Identify the closest visible frame k from LoTIS's distance predictions.
  2. Choose a small look-ahead k ± Δ toward the selected goal g.
  3. Steer to reduce its horizontal image error while moving forward.

Long-Range Navigation

Long reference trajectories are divided into shorter chunks and processed in batches, enabling navigation over kilometer-scale trajectories without encoding the full trajectory at once.

BibTeX

@misc{busch2026learninglocalizereferencetrajectories,
  title={Learning to Localize Reference Trajectories in Image-Space for Visual Navigation},
  author={Finn Lukas Busch and Matti Vahs and Quantao Yang and Jesús Gerardo Ortega Peimbert and Yixi Cai and Jana Tumova and Olov Andersson},
  year={2026},
  eprint={2602.18803},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2602.18803},
}

Reference Trajectory