At 11:04 UTC on 2021-04-20, we observed errors and elevated latencies with the Objects service in the EU PoP. We immediately moved to add more capacity to the underlying database service and the issue was resolved at 12:30 UTC. This issue occurred because our monitoring was not set to alert us soon enough that the Objects database was nearing capacity limits.
Mitigation Steps and Recommended Future Preventative Measures
To prevent a similar issue from occurring in the future we have increased the default capacity limits of the Objects service and set a more sensitive threshold for triggering alerts that the service may be nearing capacity.
Posted Apr 22, 2021 - 21:44 UTC
Resolved
The incident has not resurfaced since 12:30 UTC. We are resolving this issue, and we will follow up with a post-mortem soon.
We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service.
Posted Apr 20, 2021 - 13:39 UTC
Monitoring
The Objects service has been operating as expected for more than 30 minutes. We will continue to monitor this closely for at least the next 60 minutes before we determine that this incident is resolved.
If you continue to experience issues that you believe are related to this incident, please report the details to PubNub Support (support@pubnub.com).
Posted Apr 20, 2021 - 13:27 UTC
Identified
The engineering team has identified the issue and is taking the necessary steps to mitigate and resolve it as quickly as possible. We are no longer seeing any errors and engineering is focused on adding capacity to the Objects Service database.
Posted Apr 20, 2021 - 13:06 UTC
Update
We are continuing to investigate this issue.
Posted Apr 20, 2021 - 12:26 UTC
Investigating
At about 11:04 UTC, the Objects service began to report errors and experience elevated latencies in the EU PoP. PubNub Technical Staff is investigating and more information will be posted as it becomes available.
If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).
Posted Apr 20, 2021 - 12:24 UTC
This incident affected: Points of Presence (European Points of Presence) and Realtime Network (Objects Service).