Lossless Ethernet for AI Clusters: PFC, ECN and the Failure Modes That Bite

The first time a RoCE cluster lost most of its throughput while every drop counter stayed at zero, the team blamed the AI framework. The framework was fine. Priority Flow Control was doing exactly what it had been configured to do, on a queue that had been quietly sharing a priority with traffic it should never have shared, and the resulting pause spread across a rail until half the cluster was waiting on one slow receiver. Nothing had been dropped, so nothing looked broken, which is the worst kind of network failure to debug.

What lossless actually promises

RoCEv2 carries RDMA over UDP, conventionally to destination port 4791. It assumes a lossless or near-lossless path because RDMA retransmits at the operation level rather than the packet level: a single lost packet can cost far more goodput than the same loss would cost a TCP flow. To get that behaviour on Ethernet, three mechanisms are combined. Priority Flow Control, defined in IEEE 802.1Qbb, pauses a specific priority class on a port when its receive buffer crosses a threshold. Explicit Congestion Notification, defined in RFC 3168, marks packets before buffers overflow, giving senders an early signal. The senders respond with rate control, and the whole loop only works if the thresholds and the priority mapping are consistent across the fabric.

The word to notice is priority. Lossless is a property of one traffic class, not of the link. The same port carries lossy storage or management traffic on other priorities, and the link is only "lossless" for the queues that were configured and negotiated for it. That is why a fabric can be certified as lossless by a vendor and still drop the traffic that matters, if the queue mapping on one leaf does not match the rest.

The mechanism also changes what a buffer means. A switch cannot pause a sender instantaneously. In the time it takes a pause frame to be generated, transmitted and acted on, the upstream device keeps sending. The receiving switch must therefore reserve headroom for the frames already in flight, and if that headroom is undersized, packets are dropped before the pause can take effect. Headroom sizing is a design decision driven by port speed, cable length and whether the fabric is expected to absorb bursts, and no two platforms express it the same way.

Mechanism	What it does	How it fails
PFC	Pauses one priority class on a port before the buffer overflows	Pause storms, head-of-line blocking, cyclic deadlock across hops
ECN	Marks packets early so senders slow down	Inconsistent thresholds across the fabric cause drops or permanent throttling
DCQCN	Turns congestion marks into sender rate changes	Depends entirely on ECN being enabled end to end; one blind hop breaks it
Buffer headroom	Absorbs frames in flight while a pause takes effect	Undersized headroom means drops occur before the pause arrives
DCBx and ETS	Negotiates per-priority behaviour and bandwidth allocation	Mismatched negotiation gives asymmetric or silently absent lossless
Oversubscription	Sets the worst-case queue growth on an uplink	Higher ratios mean more pausing and more spreading under incast

The failure modes that matter

The most common failure is not a deadlock. It is head-of-line blocking that looks like a performance problem. One host stops draining its receive queue, its leaf starts pausing the spine, the spine starts pausing other leaves, and every flow that shares the lossless priority on that path slows down together. The offending host may be a single GPU server with a slow memory path or a misconfigured NIC; the blast radius is the whole priority class. The reason is structural: PFC pauses a queue, not a flow, so a shared priority means shared fate. This is the trade-off you accept for losslessness, and it is acceptable only if the lossless priority is reserved for RDMA and nothing else rides on it.

The second failure is a genuine deadlock. Pause frames stop traffic in the lossless priority when a buffer crosses a threshold, and if the topology contains a cycle, or if the buffer dependency graph contains one, then each hop can end up paused by the next with no hop able to make progress. In a well-designed spine-leaf fabric the graph is acyclic and deadlock should be impossible, but "should" is doing a lot of work when there is an accidental L2 loop, a misconfigured link aggregation, or a multi-stage design with a shared buffer between stages. PFC watchdog exists for this: when pause persists beyond a configured duration, it drops packets for a period to break the cycle. The trap is that disabling the watchdog to avoid false positives converts a recoverable stall into a hard one.

The third failure is ECN inconsistency. A fabric where some switches mark and others do not, or where thresholds differ by an order of magnitude, gives senders a signal that does not reflect the real bottleneck. The symptom is either congestion collapse, because the last hop had no ECN and dropped, or permanent under-utilisation, because one switch marked everything and the sender never recovered its rate. The reason is that DCQCN is a feedback controller: it is only as good as the feedback path, and the feedback path is every switch in the fabric.

The fourth is the unglamorous plumbing. Optics with marginal bit error rates produce CRC errors, FEC corrections and link flaps, and a flapping link removes capacity from a path that was sized for it, which produces pause waves that look like a congestion problem. Speed mismatch between a 400G leaf uplink and a 100G host port creates step-down behaviour that is easy to miss. MTU inconsistent end to end forces fragmentation of large RDMA messages and destroys the performance the fabric was built for. None of these are exotic; all of them show up as "the cluster is slow" and none of them are visible in a packet-drop counter.

How I debug a stall

Start with the counters that tell you who paused whom, in which direction, on which priority. The commands below are illustrative shapes, not vendor-verified output; every platform names these differently.

show interface priority-flow-control
show interface priority-flow-control counters
show interface counters queue
show qos queue watermarks
show interface counters errors

The order I use: first, confirm that the lossless priority is actually negotiated on both ends of every link in the path, because DCBx can silently fall back. Second, find the port that is generating pause frames and the port that is receiving them; pause transmit on a leaf toward a spine means that leaf is the one asking for room, not the one that is full. Third, look for a single host or a single port pair that appears every time. Fourth, check whether the pause correlates with a link error, a flap or a speed mismatch. Fifth, reproduce with a workload the cluster actually runs, because synthetic ping-based tests will never generate the incast pattern that exposes the problem.

The boundary: counters tell you where backpressure is applied, not why the receiver stopped draining. Often the receiver is a NIC or a host-side problem, and the network is only reporting it. An AI cluster's lossless fabric is part of a system that includes the NIC firmware, the driver, the PCIe path and the collective library, and blaming the switch is a habit rather than a diagnosis.

Designing for predictable behaviour

What I would design for, in order of impact. Keep the lossless priority exclusive to RDMA. Keep the fabric as non-oversubscribed as the budget allows, because every extra ratio becomes worst-case queue growth under incast. Keep ECN thresholds consistent across the fabric and treat an inconsistent threshold as a defect, not a tuning choice. Size headroom on the receiving side rather than trusting the pause to arrive in time. Reserve buffer for the lossless priority so that lossy traffic cannot starve it. Make PFC, ECN and queue-drop counters first-class monitoring signals, not something you look at during an incident. And test with the real collective pattern before accepting the cluster, because a fabric that passes a bandwidth test can still deadlock under all-reduce.

One honest comparison: InfiniBand is still easier to run for this job, because credit-based flow control and adaptive routing are inherent to the fabric rather than a layer of tuning on top of Ethernet. Ethernet is winning on cost, ecosystem and operational familiarity, and the industry is actively working on the gap through efforts such as the Ultra Ethernet Consortium. Neither fact makes the failure modes above disappear. They make them an engineering cost you choose to pay.

What I check before a cluster goes into service:

• Is the lossless priority explicitly mapped and reserved for RDMA only, on every switch and every host?
• Does DCBx negotiation succeed on every link, and what happens when it fails?
• Are ECN marking thresholds identical across the fabric, and is ECN enabled on every hop?
• Is buffer headroom sized for the longest cable and the highest port speed in the design?
• Is PFC watchdog enabled, and is its behaviour understood and tested?
• Are pause, ECN-marked, queue-drop and error counters alerted on, not just collected?
• Has the fabric been tested with the real collective pattern, including incast and all-to-all?
• Is the oversubscription ratio known per path, and does the workload fit inside it?
• Is MTU consistent end to end, including any overlay encapsulation?
• Is there a documented procedure for isolating one paused port without disabling losslessness for the rest?