Meta introduced MetaRoCE, a clean‑sheet RDMA transport protocol intended for AI‑scale Ethernet fabrics. Unlike conventional RoCE, which presumes an in‑order, lossless network and relies on priority‑based flow control, MetaRoCE treats the underlying fabric as lossy and shifts responsibility for packet ordering, path selection, and error recovery to the NIC. This enables the creation of many fine‑grained logical paths, each exposing real‑time telemetry such as per‑path RTT, ECN state, and utilization, thereby allowing the network to operate more like a set of independent channels rather than a single ordered stream.
The protocol was validated on AMD Pensando programmable NICs using a 64‑node AMD GPU cluster running RCCL collectives. Compared directly with RoCEv2, MetaRoCE delivered higher throughput and lower flow‑completion times for all‑reduce and all‑to‑all patterns. Key results include: ~86% throughput at 1% packet loss, continued useful bandwidth even at 10% loss, linear throughput scaling with the number of planes, and stable operation with up to 4,000 concurrent connections. Meta is releasing the specification, a DPDK‑optimized software reference implementation, and a compliance test suite through the Open Compute Project, with the artifacts expected around the OCP Global Summit in October 2026; early hardware support exists on Pensando NICs and additional vendor implementations are underway.
Why this matters
The ability to maintain high throughput despite packet loss indicates that AI workloads can run on standard Ethernet without the costly over‑provisioning of lossless fabrics. By moving ordering and recovery to the NIC, MetaRoCE reduces network‑induced stall cycles in large‑scale collective operations, which, given the observed linear scaling with additional planes, suggests a path to more cost‑effective, horizontally scalable GPU clusters.
Share this article
Found this insightful? Share it with your community on Reddit, X, or copy the link.