On November 5, 2018 starting at 20:19 UTC / 16:19 CST, we experienced a disruption with a server in our London PoP hosting telephony services. The disruption resulted in the disconnection of (50) active calls being handled by the server at the time of incident.
Telnyx has a number of High Availability mechanisms in place for the Telnyx Telephony Engine, including:
Telnyx currently lacks an active call recovery mechanism when the server upon which a given set of Telnyx Telephony Engine applications crashes or otherwise becomes unresponsive. This is what happened in the case of this incident.
Approximately 50 customer calls were disconnected.
The server experienced a hardware failure at 20:19:35 UTC. New calls were immediately re-routed.