Incident Summary

For several hours on January 5 2018, clients experienced elevated latencies for subscribe calls from Europe. Other types of requests and other regions were unaffected.

Root Cause

Due to a DNS configuration issue made during some standard maintenance we were performing, we experienced an unexpectedly large increase in traffic in Europe. Some of our data centers, mainly in APAC, were directing traffic to Europe which lead to the excess load. We were able to add more capacity and the latencies returned to normal. The DNS configuration problem was also fixed and load dropped to normal levels. We will be making a number of changes to our change management process around DNS to ensure that this issue or similar is not repeated again.