High Latency for History and Channel Groups
Incident Report for PubNub
Postmortem

Incident date and time: 7/18/2018, Intermittently between 21:05 and 21:14 UTC

Affected Services: History and Channel Groups

Problem Description, Impact and Resolution:

We received a large spike in traffic which temporarily caused high latency and small error rates for Channel Groups and History. Both automatic and manual operational resiliency measures were deployed, which caused latency and errors to return to normal.

Mitigation Steps and Recommended Future Preventative Measures:

As we always continue to be the most reliable network some of the manual intervention that took place will soon be automated to reduce the time to resolution. While we are happy this incident was identified and resolved by internal alerting and monitoring, we are actively improving automated traffic shaping and elasticity.

Posted Jul 18, 2018 - 23:36 UTC

Resolved
This is resolved.
Posted Jul 18, 2018 - 21:53 UTC
Update
We are continuing to monitor for any further issues.
Posted Jul 18, 2018 - 21:25 UTC
Monitoring
Service has returned to normal after configuration changes. We will continue to monitor this event.
Posted Jul 18, 2018 - 21:24 UTC
Identified
We are experiencing high latency for history calls and channel group operations. The issue is identified and we are actively deploying a fix.
Posted Jul 18, 2018 - 21:15 UTC
This incident affected: Realtime Network (Storage and Playback Service, Stream Controller Service).