A widespread network failure that halted trading at CME Group for over 10 hours has been traced to human error at a data center operated by CyrusOne. This incident highlights significant vulnerabilities within global financial infrastructure.
Reports indicate the cause was an error made by a technician during maintenance work in Aurora, Illinois. The breakdown, which occurred on November 28, brought critical global financial markets to a standstill and exposed weaknesses in the infrastructure supporting derivative trading worldwide.
CyrusOne stated that the incident stemmed from on-site staff and contractors failing to adhere to "standard procedures for draining cooling towers" before colder temperatures arrived. As external temperatures dropped, the internal facility temperature also decreased when its cooling system malfunctioned. This led to the failure of several key cooling units, resulting in elevated temperatures within the server rooms.
The servers hosting CME’s Globex trading platform began to overheat. Automated safety systems then initiated a shutdown of the equipment to prevent damage. With core servers offline, CME’s markets for futures, options, and products linked to commodities, currencies, and Treasury instruments abruptly ceased trade processing.
The trading pause had a significant impact across the global financial system. Traders in Asia, Europe, and the United States were disconnected from live pricing data and were unable to manage their existing positions. In some of the largest markets, liquidity completely vanished, as seen in gold and Treasury futures, leaving some traders exposed as they awaited system restoration.
Markets Shudder as the Shutdown Reverberates Around the World
The outage began in the early morning hours in the United States, prompting traders to assess the developing situation. For many traders in Europe and Asia, markets had already opened when prices on CME-connected products suddenly stopped updating. Futures on prominent U.S. stock indexes ceased to move. The commodity markets, including crude oil and gold, experienced an unnerving silence. U.S. Treasury futures, a crucial component of global fixed-income trading, were offline for several hours.
Currency trading markets were also thrown into disarray as CME’s electronic brokering system for currency trading, Globex, went offline. Traders found themselves unable to hedge against exposure or secure profits. Institutional desks encountered similar difficulties, with some reporting that their standard risk management tools had become ineffective.
This disruption underscored the modern markets' profound reliance on a limited number of data centers. With CME’s primary trading engine offline, virtually no segment of global trading could access one of its most critical pricing feeds. Some firms reported minor internal delays as activity shifted to alternative platforms, while others warned that the inability to exit positions posed substantial risks. They noted that a sharp sell-off in equities during the halt could have resulted in devastating losses.
CME’s extended delay in resuming trading on Black Friday was attributed to the venue being disconnected to allow for the stabilization of its cooling systems and the subsequent restoration of servers to an online state.
CME Rethinks Data Center Strategy After Human Error
Following the determination that human error was the primary cause of the incident, CyrusOne announced that it has already implemented more rigorous cold-weather procedures. These enhancements include increasing on-site personnel during extreme weather conditions, bolstering cooling infrastructure, hardening systems, and improving redundant systems to prevent the simultaneous failure of multiple units.
The incident has reignited discussions regarding CME’s decision to depend on a data center it no longer owns. CME sold the Aurora site in 2016 and has since relied on CyrusOne to manage the infrastructure that supports a significant portion of its electronic trading operations. The outage has also raised questions about whether this arrangement provides CME with adequate control over its essential systems.
Furthermore, questions have emerged regarding CME’s decision not to activate its backup facility. Emergency plans stipulated relocating operations to a secondary site, but CME opted against activating this backup location, having received initial indications that the outage would be brief. However, the limited period of cooling loss proved to be far more serious, and this delay ensured that the outage would extend beyond ten hours.
Industry analysts suggest that this episode should serve as a critical warning. Modern financial markets are not solely dependent on software; hardware, including power supply and cooling systems, is equally vital. A failure in any of these components can lead to the cessation of global market operations.