Failures for Presence Webhooks

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

Around 14:45 UTC, March 1, 2024, we observed the Presence service started experiencing missed Presence webhook calls. The missed webhook calls were due to a migration of the Presence webhooks across multiple customers. We rolled back the migration for all customers in case the issue was broader and created a status page for transparency. Further investigation showed the missed webhook calls were isolated to a small subset of customers.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring, we added and deployed redirect functionality in our Events & Actions service and added monitoring for future webhook migrations.

Posted Mar 08, 2024 - 16:07 UTC

Resolved

This incident has been resolved with no errors observed for the last 30 minutes. We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service. An RCA will be provided soon.

Posted Mar 02, 2024 - 02:39 UTC

Monitoring

A fix was implemented at 01:45 UTC. We are monitoring the results for the next 30 minutes.

Posted Mar 02, 2024 - 02:08 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Mar 02, 2024 - 01:37 UTC

Investigating

`Around 14:45 UTC (06:45 PT) March 01, the Presence service began to experience missed Presence webhook calls for some users. PubNub Technical Staff is investigating and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Mar 02, 2024 - 01:26 UTC

This incident affected: Points of Presence (North America Points of Presence, European Points of Presence, Asia Pacific Points of Presence, Southern Asia Points of Presence) and Realtime Network (Presence Service).