Multi-Modal Sensor Fusion in Latent Embedding Space for Robust Autonomous Navigation

Abstract

In this paper, we present a framework for multi-modal data fusion in the latent embedding space to enhance the robustness of autonomous navigation. Autonomous vehicles often rely on cameras and LiDAR, which are susceptible to degraded optical conditions. To address this, we fuse data from LiDAR, radar, and sonar using a shared latent representation, generated via a phased training approach with transformer-based models. This fusion mitigates the limitations of individual sensors and facilitates robust generation of 2D LiDAR point clouds. Our approach demonstrates improved performance in SLAM tasks, particularly under sub-optimal optical conditions, and introduces an adaptive anomaly detection mechanism for sensor reliability.