Channel Groups services are reporting errors in the EU PoP

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

At 18:42 UTC on 2022-02-16, we observed subscribe (to channel groups) errors increase due to issues with channel group registrations (add/remove channels to/from channel groups) in our EU PoP. We notified our storage provider and began rerouting storage traffic to our Mumbai PoP to mitigate the issues. At 21:56 UTC on 2022-02-16, the issue was resolved.

A new usage pattern at scale exposed some sub-optimal behavior which required us to scale our storage services on short notice to mitigate the issue.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future we are fixing the bottleneck so we are able to scale our storage service more quickly.

Posted Feb 18, 2022 - 19:19 UTC

Resolved

The incident has not resurfaced for 60 minutes. We are resolving this issue, and we will follow up with a post-mortem soon.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service.

Posted Feb 17, 2022 - 02:32 UTC

Update

Engineering is still working with our storage provider to completely resolve the underlying issue. We will continue to provide timely updates to report any changes or progress.

Posted Feb 17, 2022 - 00:40 UTC

Monitoring

The Channel Groups service has been operating as expected for the past 30 minutes. We will continue to monitor this closely for at least the next 60 minutes before we determine that this incident is resolved.

If you continue to experience issues that you believe are related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Feb 16, 2022 - 23:01 UTC

Identified

The engineering team has identified the issue and is taking necessary steps to mitigate and resolve as quickly as possible. At this point, latency should be returning to acceptible levels.

Posted Feb 16, 2022 - 22:44 UTC

Investigating

At about 21:25 UTC on Feb 16 (Feb 16, 13:25 PST)}, the Channel Groups service began to report errors in the EU PoP. PubNub Technical Staff is investigating and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Feb 16, 2022 - 21:33 UTC

This incident affected: Points of Presence (European Points of Presence) and Realtime Network (Stream Controller Service).