Elevated Channel Groups Subscribe latency in the US-West PoP

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

On Friday, August 11th at 23:45 UTC, we observed a delay in message delivery for subscribe requests using our Channel Groups service. After identifying the delay, we restarted the affected pods, and the issue was resolved at 01:44 UTC on Saturday, August 12th.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future, we are improving the Channel Groups service communication, as well as exploring enhanced error handling and retries to ensure improved monitoring and alerting.

Posted Aug 31, 2023 - 19:28 UTC

Resolved

This incident has been resolved. The incident was declared with an overabundance of caution, and was determined to be limited to a handful of customers. Please contact PubNub Support (support@pubnub.com) if you wish to discuss the incident.

Posted Aug 12, 2023 - 02:55 UTC

Update

We are continuing to monitor for any further issues for the next 30-60 minutes. We will continue to provide updates here.

Posted Aug 12, 2023 - 02:34 UTC

Monitoring

This issue has been resolved and latency has returned to normal levels. We will continue to monitor services for the next 30-60 minutes. We will continue to provide updates here.

Posted Aug 12, 2023 - 01:43 UTC

Identified

We believe the issue has been identified and a fix is being implemented. We will provide updates as they become available.

Posted Aug 12, 2023 - 01:33 UTC

Investigating

On August 11 at 23:45 UTC, we began to observe increased latency for channel group subscribes in our US-West PoP, which could result in delays in receiving messages. PubNub Technical Staff is investigating, and more information will be posted as it becomes available.

We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service.

Posted Aug 12, 2023 - 01:05 UTC

This incident affected: Realtime Network (Publish/Subscribe Service) and Points of Presence (North America Points of Presence).