Push Latency
Incident Report for PubNub
Postmortem

Cause: Node failure that automatic service discovery did not take underlying services out of rotation. A underlying service to push ended up on a unhealthy node that automatic service discovery did not take out of rotation.

Resolution: We manually removed the node from the pool and moved the service to a different node. We are also actively removing this service discovery technology and replacing it with a new one.

Posted May 17, 2018 - 21:01 UTC

Resolved
The system has remained stable. Closing this incident.
Posted May 17, 2018 - 19:37 UTC
Monitoring
The system recovered from the bad node and now we are monitoring.
Posted May 17, 2018 - 19:10 UTC
Identified
We've identified an un healthy node and routing around it and replacing it.
Posted May 17, 2018 - 19:03 UTC
Investigating
Push Messages are currently delayed. We are investigating the cause.
Posted May 17, 2018 - 18:45 UTC
This incident affected: Realtime Network (Mobile Push Gateway).