Call Control Service Disruption

Incident Report for Telnyx

Postmortem

Summary

The Telnyx Call Control service experienced diminished webhook delivery from January 5 at 14:00 UTC to January 7 at 19:00 UTC.

Impact

This disruption resulted in applications not receiving call events, thereby disrupting customer call flows.

Cause

The service responsible for delivering webhooks to customer applications was unable to retrieve customer connection configurations (e.g., Webhook URLs), due to a problem with the Call Control Service’s database reconnection logic.

The prolonged duration was attributable to problems in our end-to-end test suite and alerting system.

Action Items

Improve alerting around webhook delivery and customer configuration retrieval.
Update database reconnection logic within Call Control services
Upgrade end-to-end test suite to more accurately simulate Telnyx-customer application interactions

Posted Jan 09, 2019 - 21:30 UTC

Resolved

Issues with Call Control webhook delivery were reported, and we are currently investigating. As of right now, this issue has subsided, but potentially impacted customer applications from Sunday around 07:00 UTC until Monday around 19:00 UTC. During this time, webhooks were delivered intermittently to customer applications.

Posted Jan 07, 2019 - 20:24 UTC

This incident affected: Programmable Voice - Voice API (US).