At 07:23 UTC on December 9th, we received alerts indicating high error levels related to storage writer operations in one of our data centers. Shortly after, one of our third-party service providers reported service disruption in their environment. The service provider began the process of replacing the affected nodes. Throughout the restoration process, we closely monitored our systems to assess how the issue impacted our environment. Once all nodes were successfully restored, error levels returned to normal, and all associated alerts were resolved.
We have worked with our vendor to ensure that nodes of this type will be on redundant infrastructure going forward, so that there is less exposure to this kind of incident.