8 ms·

Trading Program Ran Amok, With No ‘Off’ Switch

143 points by mindblink 14y ago

nmcfarl 14y ago
They lost $440 million (and amount greater than their market cap), and possibly the company, on what the world knows to be incompetence. At some point if I couldn’t stop it - I’d be tempted to just kill the power to the server rooms, all of them. There just has to be a way to cut your losses.
- slurgfest 14y ago
  Agreed, the incompetence of pushing changes like that to production (with such high stakes) is pretty bad... but not even having a switch to flip?
- rorrr 14y ago
  It happened very quickly, after years of operations. Technically, it can take seconds to lose that much. What they did wrong is do all their trading with the same algorithm. Way to put all eggs in one basket.
  - patdennis 14y ago
    According to the article, it took them a half hour to shut it down. That's more than a few seconds. But as a torrent of faulty trades spewed Wednesday morning from a Knight Capital Group trading program, no one at the firm managed to stop it for more than a half-hour.
  - lostlogin 14y ago
    It wasn't after years of operation. The article states that the system was brand new.
  - sp332 14y ago
    I'm pretty sure they don't do all their trading with one algorithm. What makes you think they do?
- jonhendry 14y ago
  The server rooms might not be nearby. They might be located closer to the big exchanges' servers.
  - jonah 14y ago
    Very likely. Speed of light matters in these situations.
- forgotusername 14y ago
  I'd love to know what qualifies you to throw a word like incompetence around here. My best guess is the reason it took 45 minutes to shut it off was due to a judgement call: burn through free cash, or take out all their customers too. Bear in mind some of the largest retail brokerages in the world hang off Knight. Their primary functions are acting as an order destination and a market-maker, for efficiency's sake an obvious conclusion would be that both functions are combined in the same software (in a market where microseconds matter). So given the choice of taking a cash hit (a potentially short term affair), or a reputation hit (a much longer term and most likely fatal affair), it's entirely possible Knight knowingly made the right decision. It's worth note that the eventual deficit amounts to somewhere in the region of one year's net income, hardly insurmountable (and how many investment opportunities promise close to 100% return in a single year?). Listening to the CEO on Bloomberg, it was clear that minimizing damage to customers was their primary goal (he made this point several times in the 5 minute interview), and that he appeared comfortable with the outcome.
  - rhizome 14y ago
    I'm sure they would have tried pushing code in that time at least a few times.
  - fauigerzigerk 14y ago
    $440 million is four times their 2011 net income. I doubt the CEO is comfortable with the outcome of being on the brink of bankruptcy. He is trying to arrange a fire sale of the company as we speak. But I think you are right that they tried to avoid an outage. The incompetence, if any, is that they apparently did not know how much money they were losing and still kept the system going. It wasn't a caclulated risk but rather an incaculable one. I imagine it's not easy to know how much you're losing at any moment in time. They certainly knew they were building huge positions, but knowing how much they were going to lose on those positions requires an estimate of the price at which the positions can be closed (or a hedge). What I cannot imagine is that it is common practice to leave this kind of decision to an individual's judgement call. There have to be rules for a situation like this. And there's only one sensible rule for a rogue algo racking up unknowable losses. Kill it and deal with the consequences later. Anything else is negligent.
- iskander 14y ago
  One nitpick: You can't "kill the power" on a market-making algorithm co-located with the exchange somewhere in New Jersey.
sanxiyn 14y ago
Pure speculation. Maybe there was an off switch, which used to work, but not regularly tested, and silently broken? Wouldn't surprise me.
- CaveTech 14y ago
  Highly doubt it.
- alexchamberlain 14y ago
  Sounds highly likely!
jonah 14y ago
"Knight is also working with Goldman Sachs to help unwind the trades behind its extensive loss, according to people briefed on the matter. "Goldman has agreed to buy, at a discount, the shares that the trading firm had accumulated. Such a move would help Knight by taking the portfolio off its hands and freeing up capital." What does this mean? Why would GS do this? Why would Knight do this? Couldn't they just sell them on the open market at a better price instead?
- jkimmel 14y ago
  Yes and no. Given the kind of volume Knight purchased during the faulty trades, it could be difficult to offload that many shares on the open market in a timely manner. Maybe those stocks are hot Monday, maybe they're not. Knight needs capital yesterday to keep floating, so they're likely looking to sell everything in one basket. As for GS's motivation, they're buying at a discount. Due to the time sensitive nature of Knight's predicament, they're probably trading the portfolio to Goldman at a reduced rate. Unlike Knight, Goldman has the cash to sit on it for a while and sell the shares directly out into the open market, even if it takes a few days. Given the discount they bought the shares at, they're likely selling with a decent margin.
  - jonah 14y ago
    So GS is primarily the go-to since they're willing to leisurely unload stock they bought at a discount and they've got the cash in hand to do so. Makes sense.
  - robryan 14y ago
    Would be interesting if Goldman also made a stack off the original trades.
fsckin 14y ago
I would rarely suggest this, but if something is so incredibly broken that you're loosing money at a rate of 800 million dollars per hour, screw the customers. Turn it off at any cost. If you are forthcoming and transparent, customers will understand.
- pheon 14y ago
  Point is you dont know what the loss is. 1) They bought too much stock (incorrectly) 2) realized WTF, stopped everything 2a) more likely their clients said WTF is wrong first 3) had to sell the stock for the rest of the day. Its only after they sold everything did the $440MM price tag surface. Hopefully they sold most of their positions to goldman (instead free market) so one of their investors made a boatload of cash.. giving them favorable terms for a line of credit.
  - sp332 14y ago
    This is wrong. The algorithm was buying and selling constantly, sometimes losing small amounts of money (usually about $15) each time, sometimes as often as 20-40 times per second for each of about 150 symbols.
- wtracy 14y ago
  Another thread here has a link to a blog post with (admittedly speculative) evidence that, at two different times, they tried rebooting the system only to have it come back up and start making random trades again.
svdad 14y ago
What I wonder, following this story this week, is how the software quality controls at a place like Knight compare with those for life-critical systems like those in, e.g., aviation. On one hand, you'd think the QA in finance would be pretty solid, considering that the survival of the company could be at stake (witness Knight). On the other hand, I have a feeling that even there, people just don't take it that seriously. Would love to hear from anyone with more experience writing software for these industries.
- izaidi 14y ago
  Unlike high-frequency trading, aviation is highly regulated. In the United States the FAA specifies pretty detailed development standards for avionics software (e.g., DO-178B: http://en.wikipedia.org/wiki/DO-178B http://en.wikipedia.org/wiki/DO-178B). We're unlikely to see similarly strict requirements for financial software anytime soon.
  - wpietri 14y ago
    Having talked with people who write life-critical code, the regulation isn't really what makes it safe. Safety comes from good engineering. The regulation just makes it much harder to bring an unsafe product to market, and makes it clearer who to blame when people die.
    - wglb 14y ago
      But don't you think the existence of regulation influences the culture?
      - toomuchtodo 14y ago
        Penalties influence the culture. As we all know, the first lesson in economics is that incentives matter. Sometimes, the right people aren't being incentivized to do the right thing.
    - Retric 14y ago
      much harder to bring an unsafe product to market I would assume that's the point of the regulation in the first place? Nothing guarantees great software, but say requiring companies to pay for independent 3rd party testing adds significant barriers.
killerdark 14y ago
isnt this how you launder money?
elmarks 14y ago
Doesn't this mean that others made a killing, taking advantage of all the mispriced orders?
- jonah 14y ago
  Most likely. c.f. the recent JPMorgan losses[1]. [1] http://www.pbs.org/newshour/businessdesk/2012/06/who-benefited-from-jp-morgans.html http://www.pbs.org/newshour/businessdesk/2012/06/who-benefit...
- deleted 14y ago
  [deleted]
sahilz79 14y ago
This was apparently an infrastructure problem of some sort: http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-business-1ZpjUmh0TlevIFSAZ2UwxA.html http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-... Infrastructure changes can be notoriously difficult to back out by simply using an "off" switch, particularly if this was some type of a firmware upgrade that impacted all of their production servers. Backing it out at a minimum would require some type of a reboot, which would cause problems with an active trades. It could very well be that they were running an Active-Active environment, they had to go Active-Passive, back out the changes from the passive environment, reboot, and surgically cut over to the passive environment. This could easily take 30 minutes.
teekarja 14y ago
Strange article. Lots of text but missing the main thing I was looking for. What kind "erroneus trades"? where did the money go? If you buy stock at the market you did not intend to buy, why not just sell them the next day?
- jonah 14y ago
  Seems like maybe they couldn't hold on to the stock for long enough to unload the enormous volume they were dealing with. It sounded like at one point they were doing AS MUCH VOLUME AS EVERYONE ELSE on the exchange combined. http://news.ycombinator.com/item?id=4337750 http://news.ycombinator.com/item?id=4337750
  - Devilboy 14y ago
    Since there's 2 parties to every trade doesn't that make 50% the limit?
    - jonah 14y ago
      Trading volume is number of shares traded, not number of trades. Re-reading, it was actually ~600% greater volume. "The difference reached a peak at 9:58 a.m., when the volume was six times greater." That's pretty noticable!
    - sp332 14y ago
      Nope, sometimes they bought stock they sold themselves!
      - Devilboy 14y ago
        If you sell stock to yourself... why even bother to go through the exchange? Why would the exchange even allow such trades?
        sp332 14y ago
        Knight has different programs running. It was handling >10% of all NYSE, so it must have been running a lot of servers. When the berserk algorithm wanted to buy or sell a stock where Knight was the only "market-maker", another Knight server would usually intercept the order after it had been posted on the exchange. Here's one way it might have happened: http://www.nanex.net/aqck2/3525.html http://www.nanex.net/aqck2/3525.html
SagelyGuru 14y ago
'Prediction is difficult, especially with regard to the future'. It looks like they were using some new algorithm, which should have made them a lot of money, had the market gone up after their massive purchases. In that case, they would have pocketed fat bonuses and would not be on the news. However, it has not happened, so the crying and the search for a scapegoat is on. It sounds like the case of the banking business as usual: 'heads I win, tails you lose'. Ultimately, there is a really serious problem with the concept of limited personal liability for companies engaging in speculation. It is an assymetric arrangement, whereby the directors are entitled to the profits but are never personally responsible for the losses. With such rules of the game, it is advantageous to take crazy risks. Expect to see a lot more of this and many more taxpayer funded bailouts.
- justincormack 14y ago
  They were not taking big positions expecting the market to go up, they appear to largely have been burning money buying and selling fast. There is no government money involved.
- SagelyGuru 14y ago
  Of course, buying high and selling low is always the 'reason' for making a loss. In this case, I think the program caught itself out by manipulating the market, which it perhaps naively assumed to be non-manipulable. In other words, it was creating so much volume that, when buying (or selling), it made the market go up (or down). It was then reading the price as going up (or down) and jumping on its own bandwagon. This, of itself, would create growing oscillations in the market and growing losses. For this to work for you, you need to first create a trend and then sit back and let the suckers pile in on it and take the losses. You then return only when you want to reverse the trend again, at a profitable level (for you). I suspect the program was just too fast for its own good and not a match for the human Masters of this art.
  - beagle3 14y ago
    That's not what happened, according to nanex. It just kept making markets in reverse (instead of joining the bid and the offer, it bid on the offer and offered on the bid). Nanex speculates that Knight ran their tester software on the real market (the tester losing money on purpose to the main algorithm). Alternatively, it could simply be a bug that sends a bid instead of an offer and vice versa. One bit flip at the wrong place could cause that.
bagosm 14y ago
So, some of the owners were looking for a way out, and magically this thing broke loose and started giving away (basically) free money to undisclosed receipients. In the meantime all the technicians were fast asleep and couldnt kick the machines down or something, while they were losing milions of dollars per minute. This article is a completely honest recap by completely honest people, about completely honest traders/bankers (bankers are not people). Edit: on a COMPLETELY unrelated note, trading firms/banks are known to actively pursue the extraction of money from their clients with bogus trades/advice http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-goldman-sachs.html?pagewanted=all http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-g...
0x0 14y ago
Why isn't automated high-frequency trading banned already? Does it not go directly against the spirit and purpose of having a stock market with proper investors?
- retube 14y ago
  this is issue nothing to do with speculative trading. Knight is a broker. They provide an interface to the market for retail brokers, spread betting outfits and so on. Their algos execute orders placed by their clients. Their new algo had a bug. That's it.
  - 0x0 14y ago
    I don't know the details of retail brokers vs HFT, but this writeup http://www.nanex.net/aqck2/3522.html http://www.nanex.net/aqck2/3522.html has a lot of charts showing trades with 25 millisecond intervals. Just looking at things in a big perspective, the fact that the system is designed for allowing trades at such frequencies makes it seem like markets these days no longer exist for the benefit of the listed companies. Then again, maybe I don't know wtf I'm talking about :-S I guess I don't understand how real value can be created from such a system.
    - grn 14y ago
      Christ Stucchio wrote a HFT apology: http://www.chrisstucchio.com/blog/2012/hft_apology.html http://www.chrisstucchio.com/blog/2012/hft_apology.html
    - asmithmd1 14y ago
      Check out this TED talk about how these algos. are actively changing the surface of the planet: http://www.ted.com/talks/kevin_slavin_how_algorithms_shape_our_world.html http://www.ted.com/talks/kevin_slavin_how_algorithms_shape_o... The speaker makes an arguement that no one can understand how these algos. interact and are being studied like natural phenomena.
    - retube 14y ago
      they have hundreds if not thousands of clients some of who in turn each have thousands of customers... so yeah they will be executing a high throughput. And often large orders get broken up into smaller pieces so as to minimised market impact.
retube 14y ago
There seems to be a lot of confusion around market making, brokering, execution algorithms and HFT in this thread.
sgt101 14y ago
I wonder if the lack of a kill switch was linked to the power black outs in India - what if they set it going, got cut (power) and then only came back online 45 mins later? Could be that just one link - say a local exchange or power for an FTTP line failed.
brokenparser 14y ago
Can anyone provide some context on this matter? What happened wednesday and where?
- davvid 14y ago
  http://blogs.wsj.com/marketbeat/2012/08/02/knight-capital-trading-error-cost-firm-440-million/ http://blogs.wsj.com/marketbeat/2012/08/02/knight-capital-tr... Basically, they deployed a new HFT algo and it started buying high and selling low. oops!
reddiric 14y ago
Can someone smarter than me please explain why a system where trades got matched / executed at a granularity of once per second or once per several seconds wouldn't work? What would be the problem with exchanges accumulating and keeping secret buy and sell orders and executing them at a reasonable interval?
SeanDav 14y ago
This is an extreme example of what what can happen when what should be a software company thinks it is some other sort of company. I am sure they thought they were a trading company and software development was the necessary evil required to get things done. Well 400 million odd dollars in the red later I doubt they still feel that. I do feel sorry for them and they probably didn't deserve this huge loss. Hopefully valuable lessons can be learned.
tlogan 14y ago
I don't believe the problem was caused by "bug" - this reminds me of "rouge trader" stories. It is kinda weird that all problems in Wall Street are caused by "bugs in software" or "rouge traders": while executives are never hold accountable.
teyc 14y ago
I read the nanex article. Regardless whether it is true or not, the general trend is towards development of more sophisticated load testing programs. The most benign ones were developed for use in IT systems. E.g. Apache bench. While these can cause disruptions if aimed at production services, this does not necessarily threaten the health of an entire enterprise. However, the trend is that all software sectors are starting to adopt this particular technique of testing software with not sufficient regard to what happens if it is released into live systems. For example, we have chaos monkey, from Netflix, which randomly shutdown services in a cloud based system. What would happen if software which simulated meltdown at a nuclear facility was accidentally bundled into the build system by a tired operator? Or some one does the same with flight software? The main software running trading platforms would presumably be supervised by another program to ensure that bad algorithms do not lose e company too much money. However there was no such tool for the component that generated the test data. To me, it sounds like the supervision should be done at a higher level, e.g. A wrapper around existing APIs. All software running against live systems must call into the wrapper. Secondly, test software should conduct some kind of verification. E.g. Check for evidence that it is testing against a Test system. This might be the presence of a nonexistent company, et c. I am more than happy to compile any other ideas you may have so that the IT industry is able to build more fail safes into software. We are starting to see some of these fail safes in practice. E.g. When you try to send out an email to everyone in the organization, email software may warn you if you are sure you want to do that. The problem is we haven't thought enough about these scenarios that we don't adequately address them. Incidentally, over in Australia, the Commonwealth Bank suffered a major downtime when it's outsourcer HP accidentally pushed out system wide updates instead of doing this to select machines as originally intended.
deleted 14y ago
[deleted]

143 points by mindblink 14y ago

They lost $440 million (and amount greater than their market cap), and possibly the company, on what the world knows to be incompetence. At some point if I couldn’t stop it - I’d be tempted to just kill the power to the server rooms, all of them. There just has to be a way to cut your losses.

slurgfest 14y ago
Agreed, the incompetence of pushing changes like that to production (with such high stakes) is pretty bad... but not even having a switch to flip?
rorrr 14y ago
It happened very quickly, after years of operations. Technically, it can take seconds to lose that much. What they did wrong is do all their trading with the same algorithm. Way to put all eggs in one basket.
- patdennis 14y ago
  According to the article, it took them a half hour to shut it down. That's more than a few seconds. But as a torrent of faulty trades spewed Wednesday morning from a Knight Capital Group trading program, no one at the firm managed to stop it for more than a half-hour.
- lostlogin 14y ago
  It wasn't after years of operation. The article states that the system was brand new.
- sp332 14y ago
  I'm pretty sure they don't do all their trading with one algorithm. What makes you think they do?
jonhendry 14y ago
The server rooms might not be nearby. They might be located closer to the big exchanges' servers.
- jonah 14y ago
  Very likely. Speed of light matters in these situations.
forgotusername 14y ago
I'd love to know what qualifies you to throw a word like incompetence around here. My best guess is the reason it took 45 minutes to shut it off was due to a judgement call: burn through free cash, or take out all their customers too. Bear in mind some of the largest retail brokerages in the world hang off Knight. Their primary functions are acting as an order destination and a market-maker, for efficiency's sake an obvious conclusion would be that both functions are combined in the same software (in a market where microseconds matter). So given the choice of taking a cash hit (a potentially short term affair), or a reputation hit (a much longer term and most likely fatal affair), it's entirely possible Knight knowingly made the right decision. It's worth note that the eventual deficit amounts to somewhere in the region of one year's net income, hardly insurmountable (and how many investment opportunities promise close to 100% return in a single year?). Listening to the CEO on Bloomberg, it was clear that minimizing damage to customers was their primary goal (he made this point several times in the 5 minute interview), and that he appeared comfortable with the outcome.
- rhizome 14y ago
  I'm sure they would have tried pushing code in that time at least a few times.
- fauigerzigerk 14y ago
  $440 million is four times their 2011 net income. I doubt the CEO is comfortable with the outcome of being on the brink of bankruptcy. He is trying to arrange a fire sale of the company as we speak. But I think you are right that they tried to avoid an outage. The incompetence, if any, is that they apparently did not know how much money they were losing and still kept the system going. It wasn't a caclulated risk but rather an incaculable one. I imagine it's not easy to know how much you're losing at any moment in time. They certainly knew they were building huge positions, but knowing how much they were going to lose on those positions requires an estimate of the price at which the positions can be closed (or a hedge). What I cannot imagine is that it is common practice to leave this kind of decision to an individual's judgement call. There have to be rules for a situation like this. And there's only one sensible rule for a rogue algo racking up unknowable losses. Kill it and deal with the consequences later. Anything else is negligent.
iskander 14y ago
One nitpick: You can't "kill the power" on a market-making algorithm co-located with the exchange somewhere in New Jersey.

sanxiyn 14y ago

Pure speculation. Maybe there was an off switch, which used to work, but not regularly tested, and silently broken? Wouldn't surprise me.

CaveTech 14y ago
Highly doubt it.
alexchamberlain 14y ago
Sounds highly likely!

jonah 14y ago

"Knight is also working with Goldman Sachs to help unwind the trades behind its extensive loss, according to people briefed on the matter. "Goldman has agreed to buy, at a discount, the shares that the trading firm had accumulated. Such a move would help Knight by taking the portfolio off its hands and freeing up capital." What does this mean? Why would GS do this? Why would Knight do this? Couldn't they just sell them on the open market at a better price instead?

jkimmel 14y ago
Yes and no. Given the kind of volume Knight purchased during the faulty trades, it could be difficult to offload that many shares on the open market in a timely manner. Maybe those stocks are hot Monday, maybe they're not. Knight needs capital yesterday to keep floating, so they're likely looking to sell everything in one basket. As for GS's motivation, they're buying at a discount. Due to the time sensitive nature of Knight's predicament, they're probably trading the portfolio to Goldman at a reduced rate. Unlike Knight, Goldman has the cash to sit on it for a while and sell the shares directly out into the open market, even if it takes a few days. Given the discount they bought the shares at, they're likely selling with a decent margin.
- jonah 14y ago
  So GS is primarily the go-to since they're willing to leisurely unload stock they bought at a discount and they've got the cash in hand to do so. Makes sense.
- robryan 14y ago
  Would be interesting if Goldman also made a stack off the original trades.

fsckin 14y ago

I would rarely suggest this, but if something is so incredibly broken that you're loosing money at a rate of 800 million dollars per hour, screw the customers. Turn it off at any cost. If you are forthcoming and transparent, customers will understand.

pheon 14y ago
Point is you dont know what the loss is. 1) They bought too much stock (incorrectly) 2) realized WTF, stopped everything 2a) more likely their clients said WTF is wrong first 3) had to sell the stock for the rest of the day. Its only after they sold everything did the $440MM price tag surface. Hopefully they sold most of their positions to goldman (instead free market) so one of their investors made a boatload of cash.. giving them favorable terms for a line of credit.
- sp332 14y ago
  This is wrong. The algorithm was buying and selling constantly, sometimes losing small amounts of money (usually about $15) each time, sometimes as often as 20-40 times per second for each of about 150 symbols.
wtracy 14y ago
Another thread here has a link to a blog post with (admittedly speculative) evidence that, at two different times, they tried rebooting the system only to have it come back up and start making random trades again.

svdad 14y ago

What I wonder, following this story this week, is how the software quality controls at a place like Knight compare with those for life-critical systems like those in, e.g., aviation. On one hand, you'd think the QA in finance would be pretty solid, considering that the survival of the company could be at stake (witness Knight). On the other hand, I have a feeling that even there, people just don't take it that seriously. Would love to hear from anyone with more experience writing software for these industries.

izaidi 14y ago
Unlike high-frequency trading, aviation is highly regulated. In the United States the FAA specifies pretty detailed development standards for avionics software (e.g., DO-178B: http://en.wikipedia.org/wiki/DO-178B http://en.wikipedia.org/wiki/DO-178B). We're unlikely to see similarly strict requirements for financial software anytime soon.
- wpietri 14y ago
  Having talked with people who write life-critical code, the regulation isn't really what makes it safe. Safety comes from good engineering. The regulation just makes it much harder to bring an unsafe product to market, and makes it clearer who to blame when people die.
  - wglb 14y ago
    But don't you think the existence of regulation influences the culture?
    - toomuchtodo 14y ago
      Penalties influence the culture. As we all know, the first lesson in economics is that incentives matter. Sometimes, the right people aren't being incentivized to do the right thing.
  - Retric 14y ago
    much harder to bring an unsafe product to market I would assume that's the point of the regulation in the first place? Nothing guarantees great software, but say requiring companies to pay for independent 3rd party testing adds significant barriers.

killerdark 14y ago

isnt this how you launder money?

elmarks 14y ago

Doesn't this mean that others made a killing, taking advantage of all the mispriced orders?

jonah 14y ago
Most likely. c.f. the recent JPMorgan losses[1]. [1] http://www.pbs.org/newshour/businessdesk/2012/06/who-benefited-from-jp-morgans.html http://www.pbs.org/newshour/businessdesk/2012/06/who-benefit...
deleted 14y ago
[deleted]

sahilz79 14y ago

This was apparently an infrastructure problem of some sort: http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-business-1ZpjUmh0TlevIFSAZ2UwxA.html http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-... Infrastructure changes can be notoriously difficult to back out by simply using an "off" switch, particularly if this was some type of a firmware upgrade that impacted all of their production servers. Backing it out at a minimum would require some type of a reboot, which would cause problems with an active trades. It could very well be that they were running an Active-Active environment, they had to go Active-Passive, back out the changes from the passive environment, reboot, and surgically cut over to the passive environment. This could easily take 30 minutes.

teekarja 14y ago

Strange article. Lots of text but missing the main thing I was looking for. What kind "erroneus trades"? where did the money go? If you buy stock at the market you did not intend to buy, why not just sell them the next day?

jonah 14y ago
Seems like maybe they couldn't hold on to the stock for long enough to unload the enormous volume they were dealing with. It sounded like at one point they were doing AS MUCH VOLUME AS EVERYONE ELSE on the exchange combined. http://news.ycombinator.com/item?id=4337750 http://news.ycombinator.com/item?id=4337750
- Devilboy 14y ago
  Since there's 2 parties to every trade doesn't that make 50% the limit?
  - jonah 14y ago
    Trading volume is number of shares traded, not number of trades. Re-reading, it was actually ~600% greater volume. "The difference reached a peak at 9:58 a.m., when the volume was six times greater." That's pretty noticable!
  - sp332 14y ago
    Nope, sometimes they bought stock they sold themselves!
    - Devilboy 14y ago
      If you sell stock to yourself... why even bother to go through the exchange? Why would the exchange even allow such trades?
      - sp332 14y ago
        Knight has different programs running. It was handling >10% of all NYSE, so it must have been running a lot of servers. When the berserk algorithm wanted to buy or sell a stock where Knight was the only "market-maker", another Knight server would usually intercept the order after it had been posted on the exchange. Here's one way it might have happened: http://www.nanex.net/aqck2/3525.html http://www.nanex.net/aqck2/3525.html

SagelyGuru 14y ago

'Prediction is difficult, especially with regard to the future'. It looks like they were using some new algorithm, which should have made them a lot of money, had the market gone up after their massive purchases. In that case, they would have pocketed fat bonuses and would not be on the news. However, it has not happened, so the crying and the search for a scapegoat is on. It sounds like the case of the banking business as usual: 'heads I win, tails you lose'. Ultimately, there is a really serious problem with the concept of limited personal liability for companies engaging in speculation. It is an assymetric arrangement, whereby the directors are entitled to the profits but are never personally responsible for the losses. With such rules of the game, it is advantageous to take crazy risks. Expect to see a lot more of this and many more taxpayer funded bailouts.

justincormack 14y ago
They were not taking big positions expecting the market to go up, they appear to largely have been burning money buying and selling fast. There is no government money involved.
SagelyGuru 14y ago
Of course, buying high and selling low is always the 'reason' for making a loss. In this case, I think the program caught itself out by manipulating the market, which it perhaps naively assumed to be non-manipulable. In other words, it was creating so much volume that, when buying (or selling), it made the market go up (or down). It was then reading the price as going up (or down) and jumping on its own bandwagon. This, of itself, would create growing oscillations in the market and growing losses. For this to work for you, you need to first create a trend and then sit back and let the suckers pile in on it and take the losses. You then return only when you want to reverse the trend again, at a profitable level (for you). I suspect the program was just too fast for its own good and not a match for the human Masters of this art.
- beagle3 14y ago
  That's not what happened, according to nanex. It just kept making markets in reverse (instead of joining the bid and the offer, it bid on the offer and offered on the bid). Nanex speculates that Knight ran their tester software on the real market (the tester losing money on purpose to the main algorithm). Alternatively, it could simply be a bug that sends a bid instead of an offer and vice versa. One bit flip at the wrong place could cause that.

bagosm 14y ago

So, some of the owners were looking for a way out, and magically this thing broke loose and started giving away (basically) free money to undisclosed receipients. In the meantime all the technicians were fast asleep and couldnt kick the machines down or something, while they were losing milions of dollars per minute. This article is a completely honest recap by completely honest people, about completely honest traders/bankers (bankers are not people). Edit: on a COMPLETELY unrelated note, trading firms/banks are known to actively pursue the extraction of money from their clients with bogus trades/advice http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-goldman-sachs.html?pagewanted=all http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-g...

0x0 14y ago

Why isn't automated high-frequency trading banned already? Does it not go directly against the spirit and purpose of having a stock market with proper investors?

retube 14y ago
this is issue nothing to do with speculative trading. Knight is a broker. They provide an interface to the market for retail brokers, spread betting outfits and so on. Their algos execute orders placed by their clients. Their new algo had a bug. That's it.
- 0x0 14y ago
  I don't know the details of retail brokers vs HFT, but this writeup http://www.nanex.net/aqck2/3522.html http://www.nanex.net/aqck2/3522.html has a lot of charts showing trades with 25 millisecond intervals. Just looking at things in a big perspective, the fact that the system is designed for allowing trades at such frequencies makes it seem like markets these days no longer exist for the benefit of the listed companies. Then again, maybe I don't know wtf I'm talking about :-S I guess I don't understand how real value can be created from such a system.
  - grn 14y ago
    Christ Stucchio wrote a HFT apology: http://www.chrisstucchio.com/blog/2012/hft_apology.html http://www.chrisstucchio.com/blog/2012/hft_apology.html
  - asmithmd1 14y ago
    Check out this TED talk about how these algos. are actively changing the surface of the planet: http://www.ted.com/talks/kevin_slavin_how_algorithms_shape_our_world.html http://www.ted.com/talks/kevin_slavin_how_algorithms_shape_o... The speaker makes an arguement that no one can understand how these algos. interact and are being studied like natural phenomena.
  - retube 14y ago
    they have hundreds if not thousands of clients some of who in turn each have thousands of customers... so yeah they will be executing a high throughput. And often large orders get broken up into smaller pieces so as to minimised market impact.

retube 14y ago

There seems to be a lot of confusion around market making, brokering, execution algorithms and HFT in this thread.

sgt101 14y ago

I wonder if the lack of a kill switch was linked to the power black outs in India - what if they set it going, got cut (power) and then only came back online 45 mins later? Could be that just one link - say a local exchange or power for an FTTP line failed.

brokenparser 14y ago

Can anyone provide some context on this matter? What happened wednesday and where?

davvid 14y ago
http://blogs.wsj.com/marketbeat/2012/08/02/knight-capital-trading-error-cost-firm-440-million/ http://blogs.wsj.com/marketbeat/2012/08/02/knight-capital-tr... Basically, they deployed a new HFT algo and it started buying high and selling low. oops!

reddiric 14y ago

Can someone smarter than me please explain why a system where trades got matched / executed at a granularity of once per second or once per several seconds wouldn't work? What would be the problem with exchanges accumulating and keeping secret buy and sell orders and executing them at a reasonable interval?

SeanDav 14y ago

This is an extreme example of what what can happen when what should be a software company thinks it is some other sort of company. I am sure they thought they were a trading company and software development was the necessary evil required to get things done. Well 400 million odd dollars in the red later I doubt they still feel that. I do feel sorry for them and they probably didn't deserve this huge loss. Hopefully valuable lessons can be learned.

tlogan 14y ago

I don't believe the problem was caused by "bug" - this reminds me of "rouge trader" stories. It is kinda weird that all problems in Wall Street are caused by "bugs in software" or "rouge traders": while executives are never hold accountable.

teyc 14y ago

I read the nanex article. Regardless whether it is true or not, the general trend is towards development of more sophisticated load testing programs. The most benign ones were developed for use in IT systems. E.g. Apache bench. While these can cause disruptions if aimed at production services, this does not necessarily threaten the health of an entire enterprise. However, the trend is that all software sectors are starting to adopt this particular technique of testing software with not sufficient regard to what happens if it is released into live systems. For example, we have chaos monkey, from Netflix, which randomly shutdown services in a cloud based system. What would happen if software which simulated meltdown at a nuclear facility was accidentally bundled into the build system by a tired operator? Or some one does the same with flight software? The main software running trading platforms would presumably be supervised by another program to ensure that bad algorithms do not lose e company too much money. However there was no such tool for the component that generated the test data. To me, it sounds like the supervision should be done at a higher level, e.g. A wrapper around existing APIs. All software running against live systems must call into the wrapper. Secondly, test software should conduct some kind of verification. E.g. Check for evidence that it is testing against a Test system. This might be the presence of a nonexistent company, et c. I am more than happy to compile any other ideas you may have so that the IT industry is able to build more fail safes into software. We are starting to see some of these fail safes in practice. E.g. When you try to send out an email to everyone in the organization, email software may warn you if you are sure you want to do that. The problem is we haven't thought enough about these scenarios that we don't adequately address them. Incidentally, over in Australia, the Commonwealth Bank suffered a major downtime when it's outsourcer HP accidentally pushed out system wide updates instead of doing this to select machines as originally intended.

deleted 14y ago

[deleted]