At 22:42 UTC on August 22, 2024, we observed increased latency and errors for our Presence service. We found evidence of network issues from our own testing and monitoring, and we created a ticket with our network service provider for additional investigation. The issue was resolved as of 22:50 UTC. The root cause of this issue was a lack of monitoring and alerting around transient network issues within our network service provider's inter-VPC routing.
To prevent a similar issue from occurring in the future we have configured the proper monitoring and alerting to provide us with enough time to address this issue before it can affect the QoS of our services.