On October 30, 2023 at 17:52 UTC, we observed a high number of errors and increased latency in our Subscribe service, mostly in our US-East point of presence. This was a result of increased Subscribe traffic in the region growing since around 13:45 UTC.
Customers making subscribe API calls at the time may have experienced increased response latency, or failures with error. We increased service capacity, and memory allocation in all regions, and the issue was resolved on October 30, 2023 at 18:23 UTC.
Mitigation Steps and Recommended Future Preventative Measures
To prevent a similar issue from occurring in the future, we updated our monitoring to provide early warning of increased Subscribe traffic, giving us sufficient time to scale before issues occur.