One-Shot Team Recognition and 3D Pose Estimation of Cyclists for Augmented Reality Visualization

Abstract

Advanced computer vision and machine learning technologies transform how we experience sports events. This research focuses on enhancing the viewing experience of cycling races by automatically identifying teams from helicopter footage. It employs a multi-stage pipeline that tackles challenges such as rapid motion and similar team uniforms. Initially, cyclists are detected and tracked. Team recognition is then performed using a one-shot learning approach based on Siamese neural networks, achieving a classification accuracy of 85% on a test set composed of previously unseen teams. This method reduces the need for extensive labeling. Additionally, temporal post-processing techniques, such as applying a moving average to confidence scores, further enhance classification performance. These methods ensure reliable identification of teams and track their presence throughout the race footage. Furthermore, we integrate 3D pose estimation to generate augmented reality (AR) overlays that display rider-specific information, such as names and speeds, enhancing the broadcast’s informational value. The combination of advanced computer vision and AR showcases new possibilities for improving live sports broadcasts, particularly in challenging environments like road cycling.