28 JanKnight Trading; how to guard against mad machines. The myth of the drop copy.
In August l2012, a software glitch at the US brokerage house led to the firm losing control of their orders and making such huge losses that the firm had to be taken over. What went wrong and could it have been prevented?
Knight got themselves into trouble when the software controlling their order book and the way in which those orders were sent into the market went out of control. For more detail, click here. The “talking heads” were immediately talking about this and what needed to be done. To my mind, they missed the point.
In electronic trading there is a current buzz word: “kill switch”. This is banking’s equivalent of the emergency handle on a train. The idea being that if in doubt you cancel all the orders. This type of thing is useful for example if you have a credit event with a client and want to cancel the orders. This type of function might have been useful in the Knight case, but in practice the orders were in the market, electronically speaking, so fast that this is unlikely to have helped. And to use the kill switch you have to have some cause; something that tells you things seem wrong.
The next call to arms in the game of “bullshit bingo” that is a feature of the coverage of these events was the idea of a “drop copy”. The idea here is that the exchange makes an electronic copy of the orders and executions available to the member. This is a useful tool, but not for this. There are a number of cases where as a bank or broker you want to reconcile what you have to what the external service has. That might be an exchange, a CCP (Central Counterparty), a CSD (Central Securities Depositary) or even just your custodian. You can compare open orders, or open settlement instructions or active trades. The latter very useful at a CCP where you may have very long dated trades. However, what this does is allow you to say: “I have 100 orders, they have 100, happy days.” If you have sent 10 orders today, 9 of them in error, a comparison of your 10 vs. the 10 at the exchange will not help you. The “score” alone will not tell you that you have a problem.
In this case, I believe the missing control is one of “normal”. Systems and the people that use them need to know what is “normal”. In this case, it might have been higher than average volume, or higher than average value. The latter is hard to control for, since the average order in a stock might vary considerably. One Apple share is quite expensive vs. a share in Citigroup. Berkshire Hathaway is even more expensive. Volume is perhaps easier to monitor; number of shares in a stock might be easier to detect patterns in. This is an area known as Business Activity Monitoring (BAM). It is basically about having some traffic lights help you see the state of your business. The software that is available in this area can show you actual vs. expected activity. It requires good initial user input. Some of the systems are adaptive, in that they learn over time. If you have this, then if you think something is wrong, you need a “kill switch” to quickly cancel orders. That needs to be something pre-programmed and ready to launch instantly; all orders across the board, all for one client. And it needs to be backed-up with a re-submit function.
Lessons Learned: Electronic trading is not without its dangers. Kill switches are a necessary tool in the box to manage that. As too are drop copies. Activity monitoring that shows what is not normal is though the decisive factor. In a high frequency trading environment that needs to be a control that is monitored by a real person supported by technology.
More on the design side of this next week.
A personal request: Comments and feedback from regular readers are very welcome. Thank you for each and every one of them. If you find this Blog useful, please subscribe. There is an E-Mail tool and an RSS link on the right hand side of the main Blog page. If you like it enough to share, please share this with a friend or two and ask them to subscribe too. If I am wide of the mark and not offering anything of use, please comment or make contact directly via E-Mail