Wall Street mayhem traced to employee who forgot to turn something off

More than 1100 kilometres from Wall Street, the New York Stock Exchange’s backup data centre on Cermak Road in Chicago is supposed to safeguard US markets, standing by at all hours in case disaster ever strikes the world’s largest venue for trading shares.

When markets are closed, it participates in a well-worn routine, with NYSE staffers turning on and off systems to ensure everything works. But heading into Tuesday, an NYSE employee failed to properly shut down Cermak’s disaster-recovery system — leading to a disaster.

That human error, described by people with direct knowledge of NYSE’s internal operations, is what triggered wild market swings when trading opened Tuesday morning in Manhattan. The chaos affected more than 250 companies including Wells Fargo & Co., McDonald’s, Walmart and Morgan Stanley, in some cases sending stock prices swinging by 25 percentage points in a matter of minutes.

The episode has prompted the exchange to cancel thousands of trades at a cost that’s still being determined. Meanwhile, market professionals and day traders are rattled and waiting for the exchange to elaborate on what it publicly called a “manual error” involving its “disaster recovery configuration.”

“They’re going to need to come up with something better” to assuage investors and regulators, said Joseph Saluzzi, a partner and co-founder of Themis Trading LLC, whose firm avoided losses. “Though systems fail, and we understand that, there’s zero tolerance when it comes to the opening and the close.

In plainer English, according to one of the people, the turmoil resulted because the backup system in Cermak was left running. That misled the exchange’s computers to treat the 9.30am opening bell as a continuation of trading, and so they skipped the day’s opening auctions that neatly set initial prices.

Loading

Without that routine step, orders flowed through at prices all over the place. That soon tripped circuit breakers designed to prevent violent market swings, setting off alarms on screens across the investing world.

NYSE executives spent hours pinpointing the problem until they were confident there wouldn’t be further fallout, said the people, who spoke on the condition they not be identified discussing the internal matter. Officials also began reviewing whether the dislocated trades could be labelled “clearly erroneous” under the market’s rules and then cancelled. Shortly before 3pm on Tuesday, the exchange announced that it would unwind the most egregious transactions.