Push Notifications Latency in Multiple PoPs

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

At 18:16 UTC on June 13, 2024 we observed increased latency for delivery of mobile push messages in our Frankfurt and US-East points of presence. In response, we increased the resources available to the services and redeployed the service.The issue was resolved at 21:21 UTC on June 13, 2024.

Upon further investigation, we identified this issue occurred due to malformed message payloads creating a backlog in the message queue.

‌

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future, we increased the memory for the service to handle similar malformed payloads, as well as added additional monitoring.

Posted Jul 01, 2024 - 16:19 UTC

Resolved

This incident has been resolved, and mobile push notifications continue to be delivered normally. We will follow up with a root cause analysis soon.

We sincerely apologize for any impact on our customers and their users. If you believe you have experienced production impact due to this issue and would like to discuss it, please reach out to support@pubnub.com.

Posted Jun 13, 2024 - 21:53 UTC

Monitoring

Push notifications are now being delivered normally. We are monitoring the system to ensure no further issues.

Posted Jun 13, 2024 - 21:21 UTC

Update

We are continuing to investigate this issue.

Posted Jun 13, 2024 - 20:56 UTC

Update

We continue investigating delayed push notifications in our Frankfurt point-of-presence. Push notifications are now being delivered normally in our other regions. We will continue to provide updates here.

Posted Jun 13, 2024 - 20:55 UTC

Update

Our investigation continues and we will continue to provide updates here.

Posted Jun 13, 2024 - 20:49 UTC

Update

Our Engineering teams continue actively investigating the issue. We will continue to provide updates here.

Posted Jun 13, 2024 - 20:17 UTC

Update

We are continuing to investigate this issue.

Posted Jun 13, 2024 - 19:42 UTC

Update

We are continuing to investigate this issue.

Posted Jun 13, 2024 - 19:40 UTC

Update

We have discovered that push notifications in our US-East point-of-presence are also affected, with push notifications being delivered latently. We continue to investigate and will provide updates here.

Posted Jun 13, 2024 - 19:39 UTC

Update

We are continuing to investigate this issue.

Posted Jun 13, 2024 - 19:26 UTC

Update

We have discovered that push notifications in our Mumbai point-of-presence are also affected, with push notifications being delivered latently. Push notifications in our Frankfurt point-of-presence are also being delivered latently. We continue to investigate and will provide updates here.

Posted Jun 13, 2024 - 19:25 UTC

Investigating

We have discovered an issue where push notifications are being delivered latently in our Frankfurt point-of-presence since the last 30 minutes. Our Engineering teams are actively investigating the issue and we will provide updates here.

If you believe you have experienced production impact due to this issue and would like to discuss it, please report impact to support@pubnub.com.

Posted Jun 13, 2024 - 18:51 UTC

This incident affected: Points of Presence (North America Points of Presence, European Points of Presence, Asia Pacific Points of Presence, Southern Asia Points of Presence) and Realtime Network (Mobile Push Gateway).