Elevated rate of latency in India Point of Presence
Incident Report for PubNub
Postmortem

Problem Description, Impact, and Resolution

On April 21, 2023 at 12:04 UTC, we observed increased latency for Publish API calls to our Mumbai Point of Presence. The issue specifically affected some customers whose publish calls are routed through the PubNub Functions service. We scaled up capacity, and the issue was resolved on April 21, 2023 at 13:40 UTC.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future, we are implementing changes to our scaling process to better address these traffic patterns and adding improved alerting and monitoring for earlier warning of such issues. This should assist with faster responses if there are repeated intervals of high load then lower load.

Posted Apr 25, 2023 - 14:28 UTC

Resolved
This incident has been resolved with no spikes observed since around 13:30 UTC.
Posted Apr 21, 2023 - 15:05 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 21, 2023 - 14:31 UTC
Identified
The issue has been identified and we have not seen latency spike in the past 30 minutes.
Posted Apr 21, 2023 - 14:12 UTC
Investigating
Starting at 12:05 UTC, Customers using our India PoP might be experiencing spikes of increased latencies across multiple services.
Posted Apr 21, 2023 - 13:44 UTC
This incident affected: Points of Presence (Southern Asia Points of Presence).