Cause:
Increase in APNS delete devices, this caused the database to become CPU bound.
Resolution:
We backed the application off the database at a higher rate than normal and let the database work through the backlog. We will be making further application tunings to prevent the database from entering this state.
Posted May 07, 2018 - 06:24 UTC
Resolved
The application and data layers have not seen a return in this latency we are resolving this issue.
Posted May 07, 2018 - 06:17 UTC
Monitoring
The backlog has been mitigated and we are seeing the queues return to a healthier state. We will be monitoring that they remain this way for a little while.
Posted May 07, 2018 - 06:01 UTC
Identified
We have identified the issue. We are working to mitigate the backup in the queues and resolve the problem.
Posted May 07, 2018 - 05:45 UTC
Update
We are continuing to investigate this issue.
Posted May 07, 2018 - 05:45 UTC
Investigating
We are seeing push message queue's backup, we are investigating
Posted May 07, 2018 - 05:40 UTC
This incident affected: Realtime Network (Mobile Push Gateway).