Storage is experiencing elevated latencies and errors

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

At 04:59 UTC on August 10, 2022, we observed an elevated error rate for History and Channel Groups services globally. We contacted our third-party vendor where the affected servers reside, and the issue was resolved at 05:57 UTC when the backlog of requests was processed. This issue occurred because of a faulty process with our vendor that unexpectedly restarted our servers while performing an update.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future we are working with our vendor to ensure this process gap is addressed and fixed.

Posted Aug 12, 2022 - 16:11 UTC

Resolved

This incident has been resolved.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you'd like to discuss the impact on your service.

Posted Aug 10, 2022 - 06:34 UTC

Monitoring

A fix has been implemented, and most of the errors have been gone since 22:24 PST. Currently, we are monitoring the results for the next 30 mins.

Posted Aug 10, 2022 - 06:04 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Aug 10, 2022 - 05:49 UTC

Investigating

At about 05:00 UTC on Aug 10 (Aug 09, 22:00 PST), the History began to experience elevated latencies and errors. PubNub Technical Staff is investigating, and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Aug 10, 2022 - 05:15 UTC

This incident affected: Points of Presence (North America Points of Presence, European Points of Presence, Asia Pacific Points of Presence, Southern Asia Points of Presence) and Realtime Network (Storage and Playback Service).