On April 2, 2023 at 15:34 UTC we observed an increased rate of timeouts and errors in our services for Presence, Push, History, Subscribe, and Auth Services in our US East Point of Presence. We identified the issue was related to repair work being performed on this PoP by one of our third party vendors. After identifying the issue, we rerouted traffic from US East to US West and moved the incident to Monitoring status since improvement was observed. Once we received an update from our third-party vendor that the repair work was complete, we routed traffic back to our US East PoP, and the issue was resolved at 17:13 UTC.
To prevent a similar issue from occurring in the future we will implement more granular monitoring and alerting for early warning of such issues to the affected infrastructure.