The 400 Gb/s number stamped on an AI switch is a per-port line rate, not the bandwidth your training job will see. What a GPU-to-GPU collective gets depends on the fabric's congestion behavior, its routing, and whether the NICs can skip the CPU entirely.
That throughput difference decides the east-west choice for any serious multi-node build: InfiniBand, RDMA over Converged Ethernet (RoCE), or plain 400GbE without RDMA. Each is a real option today, and each has a shape of cluster it fits. Pick the wrong one and you either overspend on gear you cannot use or cap throughput the GPUs are rated for.
What Your East-West Fabric Carries
The east-west network moves gradients, activations and KV cache slices between GPUs that live in different nodes. Large collectives, including AllReduce, AllGather and ReduceScatter, are the dominant traffic pattern during training. They are step-synchronous: if one GPU stalls on a packet, every GPU in the job waits for it.
Three fabric properties govern what shows up on the wall clock. Bandwidth per GPU decides how fast each step moves data. Tail latency decides the stall. Congestion behavior decides what happens when flows collide on oversubscribed links.
Standard TCP/IP Ethernet handles those flows in the kernel. The CPU-mediated path adds tens of microseconds per message and bleeds cycles the GPU process could otherwise use. RDMA (Remote Direct Memory Access) cuts the kernel out and moves data straight between NIC and GPU memory. Both InfiniBand and RoCE deliver RDMA; plain 400GbE does not. That structural divide is where the rest of the choice sits. Context on how step time scales with concurrent GPUs is on the concurrent-user sizing page.
InfiniBand as the Reference Point
InfiniBand was built for HPC fabrics and treats the whole network as a flow-controlled, lossless system. Credit-based flow control on every link means a sender will not emit a packet it has no buffer for on the receiver, so drops under congestion are rare. Adaptive routing in the switch ASIC spreads flows across available paths without software intervention. In-network reductions (SHARPv4) let the switches themselves aggregate partial results, turning an AllReduce into a tree computed in the fabric rather than a ring computed by endpoints.
Current generations sit at the top of today's per-port bandwidth curve, as the chart above shows, and single-hop latency is in the nanoseconds, not microseconds.
The trade-off is vendor concentration. One supplier builds essentially the whole stack, from switch to NIC to fabric manager. There is no meaningful second source. Pricing per port tends to sit above equivalent-rate Ethernet, and the engineers who can operate it are scarcer than Ethernet engineers. If the rest of your data center is standardized on open Ethernet tooling, InfiniBand means running a separate operational discipline for one fabric. The architectural choices for a multi-node GPU cluster hinge on whether that fit is acceptable.
| Slice of the list | InfiniBand systems | InfiniBand or Ethernet RoCE |
|---|---|---|
| All 500 systems | 276 | 362 |
| 45 new systems | 31 | 35 |
RoCE: Ethernet With RDMA at Scale
RoCEv2 runs the InfiniBand verbs API over an Ethernet switch fabric. It wraps RDMA in UDP/IP so traffic routes across Layer 3 networks, which lets you build large east-west fabrics on commodity Ethernet silicon while keeping the kernel-bypass semantics training depends on.
What makes it work also makes it fragile. To behave as a lossless fabric, Ethernet needs Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) tuned per workload. Oversubscribed spine-leaf topologies produce ECMP collisions that strand flows on a single path, and when that happens RDMA retransmits are costly because they stall collectives for every job sharing the fabric.
Meta documented how far the tuning goes. The team built two 24,576-GPU H100 clusters for Llama 3, one on RoCE and one on InfiniBand, and used the RoCE cluster for the largest Llama 3 model. Getting it to the same performance class took topology changes, queue-pair-level routing and years of in-house engineering. Most IT shops do not have a network team on that scale, and that is the honest gating question for RoCE.
The direction of travel is the Ultra Ethernet Consortium spec, released as UEC 1.0 on June 11, 2025 and designed around AI and HPC traffic without reinventing switch silicon. It is new enough that no one is running a large production training job on UEC-certified gear end to end.
400GbE Without RDMA: Where It Fits
A pure 400GbE fabric, switches and NICs and all, without RDMA is still a credible build for the parts of an on-prem LLM system that are not GPU-to-GPU training. Storage back-end, north-south serving traffic, model-weight distribution and management planes all run on it without strain.
Training does not. The CPU-mediated TCP/IP path adds latency that compounds across every step of a collective, and there is no mechanism to reach the sustained throughput a tuned RDMA fabric delivers. The IEEE 802.3bs standard that defined 400 Gigabit Ethernet was approved in December 2017, and standard-conformant switches are commodity hardware now. For many enterprises that keeps 400GbE in the mix for the non-east-west parts of the build, running next to an InfiniBand or RoCE fabric that handles the training traffic.
Where 400GbE fails as a training fabric is exactly where it succeeds elsewhere. It was designed as a general-purpose Ethernet speed, not an RDMA-optimized AI fabric.
Matching Fabric to Cluster Size
Scale shifts the arithmetic. The decision looks different at single-digit nodes than at a few hundred.
- One or two nodes. NVLink or NVSwitch handles the intra-node collective. Inter-node traffic is small enough that any 200-400 Gb/s RDMA link works, and fabric cost is a rounding error against GPU cost. A cstmAI Desk or small Rack sits here.
- Four to sixteen nodes. RDMA stops being optional. Either InfiniBand or well-tuned RoCE works. The question is whether your team has the Ethernet engineering to tune PFC, ECN and ECMP for a single-tenant AI workload, or whether you want the fabric to arrive working.
- Thirty-two nodes and up. The percentage of step time spent in the fabric grows with scale. InfiniBand's in-network reductions and lossless defaults earn their premium. A RoCE build at this scale requires dedicated network engineering as a standing discipline.
Sizing the GPU count itself is upstream of fabric choice. The AI hardware estimator covers memory and GPU count, and the guidance on GPU count gives rough shapes to work from before fabric decisions are locked.
Where Your Cluster Should Land
Three conditions decide the fabric:
- Cluster size and workload. For east-west training at thirty-two nodes or more, choose InfiniBand unless you have a network team committed to RoCE tuning as an ongoing discipline. At four to sixteen nodes, either works, so let the rest of the data center decide.
- Operational fit. If your staff already runs Ethernet at scale and does not operate InfiniBand anywhere else, RoCE keeps one skill set on the floor. If you have no existing fabric team, InfiniBand arrives more opinionated and ships with vendor tooling that absorbs more of the learning curve.
- Future growth. If you plan to scale from a single rack to many over the next few years, pick the fabric that will carry the final size, not the first. Changing fabrics mid-build is expensive and disruptive, and it tends to happen while the cluster is in production.
Those three questions settle the fabric before any hardware is quoted, which is why they belong in the discovery phase of a cstmAI Cluster build rather than at the switch-selection stage.