Finding the one slow link in a 384-GPU job

A customer told us their 384-GPU run was taking longer per step than the same job a month earlier. Nothing was failing. It was just slow, and slow is harder to find than broken.

Start with the step time, not the logs

Collective operations move at the speed of the slowest link, so one weak connection drags every GPU down to its pace. The first thing we did was split the step time into compute and communication.

1. Run the job with per-rank timing on for 200 steps

2. Compare communication time across ranks

3. Look for ranks that all wait on the same neighbour

Forty-eight ranks were waiting on the same two nodes. That pointed at one switch port.

The cable

The port showed no errors. It was running at full speed but retrying a small share of packets, which our alerts treated as noise. Swapping the optical cable fixed the step time on the next run.

$ ferrite fabric check bright-otter --per-link
link        speed   retries/s   status
h7-r3:p12   400G    0           ok
h7-r3:p13   400G    1,842       degraded
h7-r4:p01   400G    0           ok

$ ferrite fabric check bright-otter --per-link
link        speed   retries/s   status
h7-r3:p12   400G    0           ok
h7-r3:p13   400G    1,842       degraded
h7-r4:p01   400G    0           ok

$ ferrite fabric check bright-otter --per-link
link        speed   retries/s   status
h7-r3:p12   400G    0           ok
h7-r3:p13   400G    1,842       degraded
h7-r4:p01   400G    0           ok

What changed

• Retries per second now page the on-call engineer above a low threshold

• Every cluster runs a per-link check during the fabric test

• The check is in the CLI, so you can run it yourself at any time

Broken hardware gets fixed in minutes. Slightly slow hardware can cost a team a week.