Starting on 2019-07-16 13:25:29 UTC until 2019-07-16 17:00:40 UTC, several customers experienced degraded audio quality for both inbound and outbound voice.
The degraded audio quality was caused by increased packet loss beginning when our Core Network team brought a new provider online, providing a new direct link to our Chicago PoP.
Our customers connect to our core network via their own ISPs. The route they take is based on how we advertise our public prefixes via our transit link providers. When we brought this new provider online, our public prefixes were advertised to the provider who, in turn, re-advertised this to their other direct-connect partners.
What this meant is that any customer whose traffic passed through those direct-connect partners to reach the Chicago PoP would use the new direct link instead of any of our existing links.
This in itself would not have been an issue except that the link was unexpectedly rate-limited far below our standard thresholds. Once the bandwidth was saturated, packets were dropped by the new provider.
The new provider link was ordered
The new provider link was brought online
Inbound and outbound calls begin to experience degraded audio quality
The Telnyx Telephony, Core Network, and NOC teams begin investigating per our internal incident response process
While our Telephony team was able to detect the degraded audio in their monitoring, the Network team’s monitoring indicated a healthy internal backbone. This suggested that the source of the packet loss was outside our network.
Core Network team starts a call with affected customers to expedite troubleshooting.
Core Network team shuts down peering with the new provider to resolve the issue
The issue is resolved