Robust Offline Reinforcement Learning for Autonomous Vessel Navigation

Abstract

Reinforcement Learning (RL) has shown promising advances in autonomous navigation, particularly in learning adaptive control policies through interaction with dynamic environments. However, training RL agents typically require direct interaction with the real environment, which can be dangerous and costly. To mitigate this, offline RL has emerged as a promising alternative, where agents learn from pre-collected datasets. In this work, we study the application of offline RL approaches in inland shipping, comparing the performance and robustness of Behavior Cloning (BC) and Conservative Q-Learning (CQL), two offline approaches, with Soft Actor-Critic (SAC), an online RL algorithm. We design experiments to test the robustness of these algorithms within the domain of unmanned surface vehicles (USVs), using an autonomous shipping simulator based on MOOS-IvP with varied environmental conditions. Our results demonstrate that both offline approaches can achieve performance comparable to the expert online policy in nominal conditions while maintaining similar robustness characteristics in out-of-distribution scenarios. Furthermore, we investigate the impact of dataset quality on offline RL performance, showing that with sufficient data, both BC and CQL can learn policies that match the dataset quality and achieve robustness similar to online RL algorithms.