Storage service is experiencing elevated latencies and reporting error in the Asia Northeast PoP.
Incident Report for PubNub
Postmortem

Problem Description, Impact, and Resolution 

At 10:31 UTC on 07 July 2021, we observed elevated latencies and errors in our Tokyo point of presence which affected History. This issue occurred because of an atypical combination of usage patterns causing CPU over-utilization in that region, ultimately resulting in the latencies and errors we observed. The incident was fully resolved at 11:07 UTC.

Mitigation Steps and Recommended Future Preventative Measures 

We added additional checks to prevent this from reoccurring in the short term. Still, in the coming weeks, we will be analyzing the causal usage patterns and working to separate the affected services with the goal of mitigating the possible recurrence of these multiple process failures.

Posted Jul 12, 2021 - 21:02 UTC

Resolved
The incident has not resurfaced since 11:07 UTC. We are resolving this issue, and we will follow up with a post-mortem soon.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service.
Posted Jul 07, 2021 - 11:48 UTC
Monitoring
The Storage service has been operating as expected since 11:07 UTC. While we are optimistic that we are stable, we will continue to monitor this closely for at least the next 30 minutes before we determine that this incident is resolved.

If you continue to experience issues that you believe are related to this incident, please report the details to PubNub Support (support@pubnub.com).
Posted Jul 07, 2021 - 11:17 UTC
Identified
At about 10:31 UTC (03:31 PST), the Storage began to experience elevated latencies and errors in the Asia Northeast PoP. PubNub Technical Staff is investigating and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).
Posted Jul 07, 2021 - 11:08 UTC
This incident affected: Realtime Network (Storage and Playback Service) and Points of Presence (Asia Pacific Points of Presence).