At 12:08 pm PDT on 2020/06/21, encrypted end-user requests into the PubNub Data Stream Network began failing for origins including pubsub.pubnub.com and other subdomains of pubnub.com. This resulted in failed requests to all PubNub services for which TLS and SSL were attempted for these domains until service was restored globally at approximately 1:25 pm PDT. All other origins, including custom origins, and subdomains of pubnubapi.com and pndsn.com were unaffected.
The root cause was determined to be a TLS/SSL certificate for *.pubnub.com origins that expired at 12:08 pm PDT on 2020/06/21.
Between 2020/06/21 12:08 pm PDT and 1:25 pm PDT, all new end-user connections into the PubNub Data Stream Network began failing for origins including pubsub.pubnub.com and other subdomains of pubnub.com. This includes requests to all PubNub services for these origins for which TLS and SSL were utilized, including Publish/Subscribe, Storage, Presence, Analytics, Functions, and Access Manager.
Other origins, including *.pubnubapi.com and *.pndsn.com, were unaffected. Also, existing, long-lived connections to *.pubsub.com origins may not have been impacted.
Initial mitigation involved migrating pubsub.pubnub.com and other *.pubnub.com origins via DNS to our new edge systems which included an updated TLS certificate. This change was completed at 1:19 pm PDT and may have required an additional 300 seconds of DNS propagation. This restored service for all TLS end-users.
However these systems do not support SSLv3 (a very small fraction of overall traffic), so further changes required a global deployment of a new certificate and migration back to the original edge system which was completed at 11:33 am PDT on 6/22/2020. During the window of 1:19 pm, PDT on 6/21/2020 and 11:33 am PDT on 6/22/2020, all TLS 1.0, TLS 1.1 and TLS 1.2 requests should have succeeded, however, SSLv3 requests would have failed.