This incident has been fully resolved since 20:50 UTC.
A full post-mortem will be provided as soon as a comprehensive review of all details, along with our networking vendors details, have been fully digested.
Thank you again for continued patience.
Posted Nov 13, 2019 - 21:56 UTC
Update
Update:
Thanks again for your continued patience while our engineers continue to work on gathering more details on root cause.
Our networking engineers have enlisted the help of our vendor to determine the cause of the high CPU in our Chicago region. This is a continued and ongoing effort.
We'd like to clarify that services located in Chicago region were impacted during the times of 20:15 UTC - 20:49 UTC.
Messaging will have seen little impact as other regions picked up the re-routed traffic.
Call control was impacted during this the above time-frame and users would have seen intermittent 5XX responses from their respective API's.
Telephony calls were re routed but some users may have observed added latency or audio quality issues at 20:20 - 20:22 UTC.
The mission control portal and related services such as porting were also impacted during this time-frame.
We are continuing to monitor all services at this time but everything has been fully stable since 20:50 UTC.
Posted Nov 13, 2019 - 21:27 UTC
Update
Update:
We're observing continued stability.
We can confirm that traffic from other US regions were affected and further details will be available as soon as everything is confirmed by the networking engineers.
Thank you for your continued patience.
Posted Nov 13, 2019 - 20:55 UTC
Monitoring
Update:
Issues are currently resolved.
In the meantime, we continue to bypass traffic via Chicago and can confirm all production based services are not impacted.
Posted Nov 13, 2019 - 20:41 UTC
Identified
Update:
We're seeing high CPU usage across our Chicago region which resulted in DNS flaps.
Our networking team have identified the issue and are working on resolving it.
We're still confirming exact services which are impacted at this time.
Posted Nov 13, 2019 - 20:32 UTC
Investigating
We are currently investigating internal dns resolution issues which are causing certain services to intermittently resolve.
More information on what services are impacted shortly.
Posted Nov 13, 2019 - 20:22 UTC
This incident affected: Programmable Voice - Voice API (US) and Messaging.