Events

Generative AI use cases have led to the emergence of distributed systems (aka AI datacenters) at scales of 100,000+ accelerators. In such distributed clusters, frequent synchronization across the accelerator devices is inevitable. This makes the design and optimization of the network fabric a crucial component for overall performance and efficiency. Designing an optimized fabric for AI datacenters is an open and active challenge today – with massive co-design opportunities spanning across technology (e.g., waferscale, photonics), hardware architectures (i.e., network topologies) and software scheduling (e.g., optimal collective algorithms).

This talk will introduce our work towards a systematic co-design approach for designing and deploying scalable network fabrics for AI datacenters.  In particular, we will deep dive into two frameworks. First, we present ASTRA-sim, a distributed AI system simulator. ASTRA-sim can model and simulate the end-to-end stack of AI datacenters spanning workload, scheduling, hardware architectures and technologies. ASTRA-sim enables easy plug-ins of customized workload descriptions, collective algorithms and network fabric implementations via a suite of APIs (being standardized via MLCommons and OCP). Second, we present TACOS, a Topology-Aware Collective Algorithm Synthesizer. TACOS automates the generation of topology-aware collective algorithms for arbitrary network topologies using the time-expanded network data structure and a novel link-chunk matching algorithm. Evaluated over future systems (simulated via ASTRA-sim) and real deployments over existing systems, TACOS demonstrates speedups over the state-of-the-art libraries today (e.g., NCCL) and scalable synthesis of customized collective algorithms for tens of thousands of accelerators.