Timeline
All times are US Central (UTC−5).
| Time | Event |
|---|---|
| 14:02:11 | Optic in LAG member Te0/1 begins intermittently failing. Link stays up; CRC errors climb. |
| 14:02:40 | Monitoring fires on interface error rate. On-call paged. |
| 14:04:05 | Engineer confirms hashing is still distributing flows onto the degraded member. |
| 14:06:22 | Te0/1 administratively shut. Traffic rehashes onto healthy members. Loss stops. |
| 14:41:00 | Optic physically replaced. Member returned to service after a soak period. |
Root cause
The optic degraded rather than failing outright. Because the link never dropped, LACP kept the member in the bundle and the switch continued hashing flows onto it. Customers whose traffic happened to hash onto that member saw loss; everyone else saw nothing at all — which is exactly why the first reports looked inconsistent.
Our alerting watched link state closely and error rate loosely. That is the wrong way round for this failure mode.
What we changed
- Error-rate thresholds now page directly. CRC and input-error rates on any LAG member cross to a paging alert at a far lower threshold, rather than a dashboard warning.
- Automatic member isolation. A member exceeding the error threshold for 30 seconds is now removed from the bundle automatically, before a human is involved.
Both changes are live across every aggregation switch in HOU1.
What we did not change
We considered dropping LACP hashing in favour of active/standby, and decided against it — the capacity loss is real and this failure mode is now handled. We would rather fix the detection than give up the bandwidth.
If you were affected and want the interface counters for your own records, open a ticket referencing this postmortem and we will send them.
Runs the edge and the maintenance calendar in HOU1. Writes the postmortems, including the ones that are not flattering.