Wafer-Scale GPU Memory Pool With In-Package Optics for Enhanced Capacity and Bandwidth

Abstract

The escalating demands of modern AI applications push the limits of conventional GPU memory systems, where high-bandwidth memory (HBM) architectures face critical challenges in scalability and bandwidth efficiency. We propose a wafer-scale system integrating photonic interconnects to decouple GPU compute from memory, addressing capacity and bandwidth limitations of conventional HBM-based architectures. We develop an analytical framework, HBM-pool, blending in-package optical I/O (OIO) parameters with GPU memory traffic for high-density nodes. The system-technology co-optimization (STCO) study shows that 300 mm wafer-scale disaggregation with OIO can achieve a 4× increase in memory capacity and reduce bandwidth bottlenecks up to 8× compared to a 2.5D electrically packaged node, enabling larger training/inference and improving latency.