Admin Portal Slow to Respond

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

On August 30, 2022 at 8:50 PM UTC, we observed slow or failed logins to our Admin Portal, as well as errors using some functionality within the portal once logged in. We investigated, and found that a third-party provider we use for subscription management was having an operational incident. We did what we could to work around the provider’s issue, and escalated with them. The provider resolved their issue, and the incident was resolved at 5:20 AM UTC on August 31.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future we have added work to our roadmap to reduce our dependency on the third-party for Admin Portal logins, preferring to allow a login and alert the user to the reduced functionality in case of an error at the provider. We also added work to our operations backlog to better alert us to errors of this kind from the service provider.

Posted Sep 02, 2022 - 00:10 UTC

Resolved

The incident has not resurfaced for the past hour. We are resolving this issue, and we will follow up with a post-mortem soon.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service.

Posted Aug 31, 2022 - 05:18 UTC

Monitoring

The admin portal has been operating as expected after a fix was deployed by our provider. We will continue to monitor this closely for at least the next 60 minutes before we determine that this incident is resolved.

If you continue to experience issues that you believe are related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Aug 31, 2022 - 04:16 UTC

Update

We are continuing to work with the provider to fix the issue, and some improvements are already implemented and apparent.

Posted Aug 31, 2022 - 03:10 UTC

Identified

The engineering team has confirmed the issue is with a downstream provider, and is taking necessary steps to escalate with them and resolve as quickly as possible.

Posted Aug 31, 2022 - 02:34 UTC

Investigating

Due to an issue at a downstream provider, the admin portal at admin.pubnub.com is currently taking an abnormally long time to respond, and some functionality within the portal may time out. We are investigating the issue. Other PubNub services are unaffected, and are operating normally.

Posted Aug 31, 2022 - 01:56 UTC

This incident affected: Website and Portals (Administration Portal).