Summary

On November 5, 2018 starting at 20:19 UTC / 16:19 CST, we experienced a disruption with a server in our London PoP hosting telephony services. The disruption resulted in the disconnection of (50) active calls being handled by the server at the time of incident.

Telnyx has a number of High Availability mechanisms in place for the Telnyx Telephony Engine, including:

• Distributed nodes where calls can be routed.
• Automatic re-routing of new call attempts when node specific issues are detected, including application restarts or server crashes.
• Call recovery during application restart

Telnyx currently lacks an active call recovery mechanism when the server upon which a given set of Telnyx Telephony Engine applications crashes or otherwise becomes unresponsive.  This is what happened in the case of this incident.

Impact

Approximately 50 customer calls were disconnected.

Cause

The server experienced a hardware failure at 20:19:35 UTC.  New calls were immediately re-routed.

Action Items

• Telnyx is currently working on implementing a new High Availability mechanism for media, enabling recovery for active calls. This will require low-level changes to our Back-to-Back User Agent software. The changes will be implemented in a way such that they neither interfere with existing support features, nor affect metrics like PDD or CPT, nor require significantly more resources.