Resilient AI Supercomputer Networking using MRC and SRv6
Authors:
Joao Araujo,
Alex Chow,
Mark Handley,
Ryder Lewis,
Christoph Paasch,
Jitendra Padhye,
Michael Papamichael,
Greg Steinbrecher,
Amin Tootoonchian,
Lihua Yuan,
S. Anantharamu,
Abhishek Dosi,
Mohit Garg,
Mahdieh Ghazi,
Torsten Hoefler,
Deepal Jayasinghe,
Jithin Jose,
Abdul Kabbani,
Guohan Lu,
Yang Wang,
K. Doddapaneni,
Murali Garimella,
Vipin Jain,
Yanfang Le,
H. Nagulapalli
, et al. (25 additional authors not shown)
Abstract:
Tail latency dominates the performance of synchronous pretraining jobs when running at very large scales. We describe a three-pronged approach: (1) a new RDMA-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions (2) the use of multi-plane Clos topologies to get the benefits of high switch radix and redundancy, all…
▽ More
Tail latency dominates the performance of synchronous pretraining jobs when running at very large scales. We describe a three-pronged approach: (1) a new RDMA-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions (2) the use of multi-plane Clos topologies to get the benefits of high switch radix and redundancy, allowing training clusters well over 100K GPUs to be built as two-tier topologies while increasing physical redundancy, and (3) the use of static source-routing using SRv6 to allow MRC the freedom to bypass failures by itself. We describe our experiences running MRC and static SRv6 routing in production in OpenAI and Microsoft's largest training clusters, where it has been used to train the latest frontier models. We demonstrate how MRC allows AI training jobs to ride out many network failures that previously would have interrupted training.
△ Less
Submitted 5 May, 2026;
originally announced May 2026.
007: Democratically Finding The Cause of Packet Drops
Authors:
Behnaz Arzani,
Selim Ciraci,
Luiz Chamon,
Yibo Zhu,
Hingqiang Liu,
Jitu Padhye,
Boon Thau Loo,
Geoff Outhred
Abstract:
Network failures continue to plague datacenter operators as their symptoms may not have direct correlation with where or why they occur. We introduce 007, a lightweight, always-on diagnosis application that can find problematic links and also pinpoint problems for each TCP connection. 007 is completely contained within the end host. During its two month deployment in a tier-1 datacenter, it detect…
▽ More
Network failures continue to plague datacenter operators as their symptoms may not have direct correlation with where or why they occur. We introduce 007, a lightweight, always-on diagnosis application that can find problematic links and also pinpoint problems for each TCP connection. 007 is completely contained within the end host. During its two month deployment in a tier-1 datacenter, it detected every problem found by previously deployed monitoring tools while also finding the sources of other problems previously undetected.
△ Less
Submitted 20 February, 2018;
originally announced February 2018.