Toward Efficient Orchestration of GenAI Workloads in Modern Container Clouds

Abstract

As Artificial Intelligence (AI) continues to drive innovation across multiple industries and verticals, the efficient orchestration of AI workloads in modern containerized cloud environments is becoming increasingly crucial. In contrast to traditional cloud applications, modern Generative Artificial Intelligence (GenAI) models and widely used Deep Learning (DL) workloads - particularly those involving real-time inference and distributed training - have unique and highly demanding compute, storage, and network requirements. This article presents our vision for efficient orchestration of diverse AI workloads in container clouds, including a set of orchestration strategies designed to meet the dynamic and resource-intensive needs of state-of-the-art AI applications while ensuring performance and scalability. We discuss the open challenges in orchestrating AI workloads with diverse communication patterns, such as the need for low-latency communication, high-throughput data pipelines, and efficient resource allocation across highly distributed systems. This work also highlights the importance of integrating AI-specific optimizations into popular container orchestration platforms such as Kubernetes (K8s), leveraging modern technologies, including GPU scheduling mechanisms and strategies for distributed training and inference. We envision a future in which container clouds not only scale seamlessly to accommodate the growing demands of AI workloads, but also incorporate AI-driven orchestration mechanisms that intelligently adapt to workload fluctuations, predict resource requirements, and mitigate bottlenecks. This article aims to provide a foundational framework for efficient life-cycle management of AI workloads in modern cloud infrastructures, paving the way for future research in this rapidly evolving field.