At 19:20 UTC on Tuesday June 13th, 2023 we observed increased error rates and latency for our Authorization services at our US East facility. In response, we redirected authorization services from US East to US West, and the issue was mitigated at 20:59 UTC on Tuesday June 13th, 2023. During this time, we identified the root cause of the issue was due to a third-party service incident. After confirming the third-party service incident was resolved, we rerouted the Authorization traffic back to US East at 22:58 UTC on Tuesday, June 13th 2023.
To prevent a similar issue from occurring in the future we are developing a comprehensive failover plan to more quickly move services from one region to other regions.. In the next few weeks we will be implementing new processes to allow mitigation of regional service issues.