Multicluster Computing Systems With Julia

Abstract

The ability to aggregate the computational resources of multiple clusters is useful for computationally challenging applications that cannot run on a single cluster available to the user, or that require special computational resources not accessible locally to execute essential components, such as specific-purpose accelerators. However, deploying and using multicluster computing platforms is quite complex for the vast majority of computational and data scientists, who are not experts in high-performance computing (HPC) systems. This article explores the construction of multicluster computing systems using the Julia programming language and packages from its ecosystem, offering contributions and insights into the feasibility of simple interfaces for multicluster computing in real-world applications.