Increased latency in US East PoP

Incident Report for PubNub

Postmortem

Final RCA for this incident has been completed. Please contact PubNub Support for a copy of the RCA.

Posted Apr 27, 2020 - 04:05 UTC

Resolved

The incident has been resolved and stable for 2 hours and we are confident all services will remain operational. An RCA will be provided once we have gathered all the necessary details.
Posted Apr 24, 2020 - 02:58 UTC

Update

We are back to a normal operational state and all customers should be fully recovered at this point. We will move to a Resolved status after we are in a stable state for the next 30 minutes.
Posted Apr 24, 2020 - 00:37 UTC

Update

We are back to a normal operational state. We will continue to monitor and report any changes in status.
Posted Apr 24, 2020 - 00:19 UTC

Update

We are continuing to monitor for any further issues.
Posted Apr 23, 2020 - 23:34 UTC

Update

For the past 10 minutes, we experienced a new wave of errors and we are working to mitigate this.
Posted Apr 23, 2020 - 23:33 UTC

Update

We are continuing to monitor for any further issues.
Posted Apr 23, 2020 - 23:03 UTC

Monitoring

The actions we have taken have brought all services back to normal operation. We are continuing to actively monitor in the event we see the issues reoccur but we are confident this will not happen.
Posted Apr 23, 2020 - 23:02 UTC

Identified

We believe we are more or less operating normally since 22:07 UTC. Because of the throttling we had to implement, there could be some outlier customers who will still be impacted and we are working on full resolution there so we are staying in "Identified" mode.
Posted Apr 23, 2020 - 22:24 UTC

Update

We are no longer seeing any increased latency in our system and nor seeing any further errors. Again, we are still working to ensure resolution.
Posted Apr 23, 2020 - 21:37 UTC

Update

We are continuing to investigate this issue.
Posted Apr 23, 2020 - 21:19 UTC

Update

We are continuing to investigate this issue.
Posted Apr 23, 2020 - 21:19 UTC

Investigating

We are experiencing increased latency in our US East PoP. We are currently investigating this issue.
Posted Apr 23, 2020 - 21:07 UTC
This incident affected: Realtime Network (Publish/Subscribe Service, Storage and Playback Service, Stream Controller Service, Presence Service, Access Manager Service, Realtime Analytics Service, Mobile Push Gateway), Functions (Functions Service, Vault, Key Value store, Scheduler Service), and Points of Presence (North America Points of Presence).