The Subscriber and Channel Groups services are reporting errors in the AP South PoP

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

At 03:27 UTC on 2022-16-02, we observed errors with Channel Group registrations (add/remove channels to/from channel groups) in the South AP PoP (Mumbai). We immediately escalated to our storage provider while we routed storage traffic to US West to mitigate the issue. The issue was resolved at 04:30 UTC on 2022-02-15.

This issue occurred due to a bug in our storage providers' platform. Our internal monitoring detected the errors which allowed us to take immediate actions with rerouting traffic and escalating to our provider.

Mitigation Steps and Recommended Future Preventative Measures

We plan to migrate our Mumbai PoP to Kubernetes which will allow us to more efficiently reroute traffic to another PoP when needed. Our storage provider will resolve the existing bug.

Posted Feb 18, 2022 - 18:18 UTC

Resolved

The incident has not resurfaced for 30 minutes. We are resolving this issue, and we will follow up with a post-mortem soon.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service.

Posted Feb 16, 2022 - 05:05 UTC

Monitoring

Our storage vendor has identified the issue and taken steps to mitigate it. We will provide further details once we have done a full investigation.

Posted Feb 16, 2022 - 04:32 UTC

Update

Our storage vendor has responded that they advised that the affected PoP is currently recovering. Our engineering staff continues to investigate and mitigate as possible, in parallel to our vendor's efforts.

Posted Feb 16, 2022 - 04:21 UTC

Update

Engineering is still working to resolve the underlying issue. We will continue to provide timely updates to report any changes or progress.

Posted Feb 16, 2022 - 04:09 UTC

Investigating

At about 03:25 UTC on Feb 16 (Feb 15, 19:25 PST)}, the Subscriber and Channel Groups service began to report errors in the AP South PoP. PubNub Technical Staff is investigating and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Feb 16, 2022 - 03:41 UTC

This incident affected: Realtime Network (Stream Controller Service) and Points of Presence (Southern Asia Points of Presence).