We are experiencing elevated latencies in our presence service
Incident Report for PubNub
Postmortem

Incident date and time: 7/1/18 Intermittently between 12:05 and 13:15 UTC

Affected Services: PubNub Presence

Problem Description, Impact and Resolution:

A subset of customers experienced higher latencies and error rates for presence operations. The initial impact was caused by a spike in system resources which overloaded a subset of customer’s presence counting. Configuration was deployed to mitigate the impacted resources which allowed for recovery. We also identified a bug in our presence applications that prevented them from recovering as quickly as expected.

Mitigation Steps and Recommended Future Preventative Measures:

We are actively improving our presence automated healing configuration. In this incident we feel confident our tools enabled us to identify and resolve the issue, but as always we strive to improve speed to resolution. Additionally, we are working to resolve the identified bugs and ensure they will not happen in the future.

Posted Jul 04, 2018 - 15:43 UTC

Resolved
Latencies continue to be normal and we are considering this incident resolved.
Posted Jul 01, 2018 - 16:37 UTC
Monitoring
Presence latencies have returned to normal. We are continuing to monitor but all services are fully operational.
Posted Jul 01, 2018 - 15:02 UTC
Investigating
We are experiencing elevated latencies in our presence service. We will keep this status page up to date as we get more information.
Posted Jul 01, 2018 - 14:24 UTC
This incident affected: Realtime Network (Presence Service).