The Presence service is experiencing elevated latencies in the all PoPs

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

At 14:02 UTC on 2021/11/12, we observed increased latency with the Presence service in all of our PoPs. This was a repeat issue from the prior day so we already knew the quickest path to mitigating the effects of this incident. We contacted our service provider to provide insights while we restarted our presence services to clear the backlog of requests. and the issue was resolved at 15:01 UTC.

This issue occurred because there wasn't sufficient alerting between our system and our service provider when there is a rapid increase in Presence requests.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future our service provider has upgraded the infrastructure to handle higher network throughput.

Posted Nov 16, 2021 - 22:44 UTC

Resolved

The incident has not resurfaced in the past hour. We are resolving this issue, and we will follow up with a post-mortem soon.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service.

Posted Nov 12, 2021 - 16:12 UTC

Monitoring

The Presence service latency has returned to expected levels and has been operating as expected for the past 10 minutes. We will continue to monitor this closely for at least the next 60 minutes before we determine that this incident is resolved.

If you continue to experience issues that you believe are related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Nov 12, 2021 - 15:08 UTC

Investigating

Around {02:00 UTC (06:00 PST)}, the Presence service began to experience elevated latencies in all of the PoPs. PubNub Technical Staff is investigating and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Nov 12, 2021 - 14:37 UTC

This incident affected: Points of Presence (North America Points of Presence, European Points of Presence, Asia Pacific Points of Presence, Southern Asia Points of Presence) and Realtime Network (Presence Service).