Elevated error rate for presence calls in our US West and US East PoPs.
Incident Report for PubNub
Postmortem

Problem Description, Impact, and Resolution

From 12:10 PM - 12:37 PM PST, some customers may have experienced an elevated rate of errors and timeouts for presence calls in our US West and US East PoPs.

This occurred because we received an unusual traffic pattern that significantly elevated our 90th percentile latencies. This caused delays which lead to timeouts for some customers. We were able to mitigate this traffic and reduce the latency and thus eliminate the timeouts and errors.

Mitigation Steps and Recommended Future Preventative Measures

We have already implemented more proactive measures to prevent this in the future. Additionally, we are always working to ensure the utmost robustness of our service so that any traffic can be handled without incident.

Posted Dec 10, 2019 - 20:41 UTC

Resolved
All services are still fully operational and we are resolving this incident.
Posted Dec 05, 2019 - 22:01 UTC
Monitoring
From 12:10 PM - 12:37 PM PST, some customers may have experienced elevated error rate for presence calls in our US West and US East PoPs. This is recovered but we are continuing to monitor the situation.
Posted Dec 05, 2019 - 20:49 UTC
This incident affected: Points of Presence (North America Points of Presence) and Realtime Network (Presence Service).