4 ms·

Kraken Goes Down for 40 Hours, Drawing Mt. Gox Comparisons

84 points by yonilevy 9y ago

verroq 9y ago
Pretty irresponsible to completely brick production servers when doing an upgrade. https://status.kraken.com https://status.kraken.com Sounds like their staff has being going without sleep for a few days now.
- Shank 9y ago
  > Sounds like their staff has being going without sleep for a few days now. Given that sleep deprivation has effects similar to being blackout drunk, I'd consider that equally irresponsible[0]. [0]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1739867/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1739867/
  - switch007 9y ago
    I've lost count the number of times "watched the sunrise" was the climax of an IT war story. Bonus points for two nights in a row ! :D
    - sambull 9y ago
      Worked for a IT managed services place that made staff pull 24+ hrs non stop regularly. I walked away from that career and spent years recovering from a 38 hr shift where I was begging my managers to get me relief. "Your the only guy that knows this well enough" is a sign you work for a failing organization pinning that shit on you. I learned that the "business" and sales dudes absolutely will trade your health/lively hood for a few extra bucks on commission.
- tzakrajs 9y ago
  I can't help but feel less stressed about my own job after reading this status page.
- bufferoverflow 9y ago
  It's not an upgrade, it's a bugfix. Probably an important bug, they don't want to risk tens/hundreds of millions of dollars.
  - verroq 9y ago
    They could have left the legacy system running before switching irreversibly to the new bugged system. That way they can rollback the change.
    - evgen 9y ago
      Unless the bug being fixed was some sort of critical security failure that enabled people to game the exchange or manipulate balances. I am guessing this is more of a screw-up by amateurs than a 'take it all down until this hole is patched' situation, but there is a slim possibility that keeping it up was a bigger risk than the reputation hit of extended downtime.
      - hrpnk 9y ago
        Looks like these are announced improvements to the Exchange engine. Really look forward to a post-mortem on this issue. Curious if they had issues with database migrations or such, that required recovery from backup in case of errors. Plus, for sure there were bugs in the code itself.
    - bufferoverflow 9y ago
      No, they couldn't. It's a large trading platform. Any serious bug can lead to the theft of tens of millions of dollars worth of cryptocurrencies.
frik 9y ago
As someone wrote It took three days of downtime for MtGox to admit that bitcoins were missing. @krakenfx is on day 2. I'm hoping the same thing isn't happening at Kraken, but not confident. Not so nice they tried to hide the news on HN twice by flagging them: https://news.ycombinator.com/item?id=16131750 https://news.ycombinator.com/item?id=16131750 and https://news.ycombinator.com/item?id=16127180 https://news.ycombinator.com/item?id=16127180 (both hidden after a short time)
hal9000xp 9y ago
Latest Kraken update says that: > We are still working to track down an elusive bug which is holding up launch. This bug did not appear in our many weeks of testing and only emerged in the production environment. Unfortunately, it is not consistently reproducible and we cannot launch until this issue is resolved. Sounds pretty realistic to me. During my 8+ years of programming, I had a lot of pain and sleepless nights trying to catch elusive bugs which can be reproduced only in production servers and only sometimes. Many times, my clueless managers got nervous and demanded estimated time from me. Unfortunately, it's impossible to give realistic estimations. It could take hours, days or weeks. Also, searching for such bugs doesn't scale well. It means if you have 5 devs looking for elusive bug, it doesn't mean they will find a bug 5 times faster.
- koonsolo 9y ago
  I even had clueless managers who demanded I explained the technically complex problem to them, because "maybe our insights could shed new light on the problem". They just don't get that best thing to do is leave us alone and let us do our job. They are just nervous because they are totally powerless in such situation. In a factory, a manager might be smarter than the one doing manual labour. In tech, it's the other way around.
  - setr 9y ago
    If you assume potential malice (ie a lazy dev just not handling the bug at all) then a manager is required to ask about the bug, if only to try to verify that you've put real effort into the problem (based on say, confidence in whatever you spew). The worst possible scenario is that five months later, it turns out no one actually looked into properly: they just claimed to.
    - koonsolo 9y ago
      In that case, the manager should have fired that dev a long time ago. Best managers I worked for had trust in their developers, and asked for estimates on planning (not bugs). The worst managers didn't trust us, and pushed their own planning on us because they thought it would make us work faster. They thought planning can be negotiated. I can tell you which projects were within 7% of the deadline, with good quality. And which ones we were always putting out fires, and some never finished.
hrpnk 9y ago
They're back with free trading until end of January: https://blog.kraken.com/post/1449/kraken-returns-with-free-trading/ https://blog.kraken.com/post/1449/kraken-returns-with-free-t...
- jnordwick 9y ago
  Down again. Amateur hour. Lol.
  - Bombthecat 9y ago
    Still no withdraw..
azzhohle 9y ago
Kraken is notoriously slow, unresponsive and buggy exchange and frankly I dont know why they get away with it. For months or year(s) you often run into site unavailable errors (load balancing issues?). Or if you are lucky to get to the trading screen you are unable to get trades through because of engine issues (capacity issues again?). Sometimes all you see is an empty red background rectangle where the error text should be indicating no dice but then minutes/hours later you might get one or more of the positions you tried to open despite only having gotten (ambiguous) errors. Let's hope they finally fixed their engine and purchased more capacity.
valdiorn 9y ago
Upgrade a 750 million dollar a day trading system with no rollback strategy. Amateur hour, and if this were a "real" exchange, they'd lose their licence and have lawsuits and government agents knocking down their door (real-money MTFs have strict uptime and infrastructure requirements for anyone who didn't know)
arisAlexis 9y ago
everything is up and trading is free for the rest of the month. </drama>
- Turbots 9y ago
  But nobody can withdraw their cash or transfer to other exchanges... They are literally holding their customers hostage. I just want out of this hellhole of a website, but they won't let me. Criminal!
lowlevel 9y ago
They say the next big thing is here, that the revolution's near, but to me it seems quite clear that it's all just a little bit of history repeating
rootsudo 9y ago
IT sounds like this week is going to be profitable on Kraken if you trust their status reports.
hndamien 9y ago
I tend to trust them for security because they are handling the Gox bankruptcy. For uptime... not so much.

84 points by yonilevy 9y ago

Pretty irresponsible to completely brick production servers when doing an upgrade. https://status.kraken.com https://status.kraken.com Sounds like their staff has being going without sleep for a few days now.

Shank 9y ago
> Sounds like their staff has being going without sleep for a few days now. Given that sleep deprivation has effects similar to being blackout drunk, I'd consider that equally irresponsible[0]. [0]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1739867/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1739867/
- switch007 9y ago
  I've lost count the number of times "watched the sunrise" was the climax of an IT war story. Bonus points for two nights in a row ! :D
  - sambull 9y ago
    Worked for a IT managed services place that made staff pull 24+ hrs non stop regularly. I walked away from that career and spent years recovering from a 38 hr shift where I was begging my managers to get me relief. "Your the only guy that knows this well enough" is a sign you work for a failing organization pinning that shit on you. I learned that the "business" and sales dudes absolutely will trade your health/lively hood for a few extra bucks on commission.
tzakrajs 9y ago
I can't help but feel less stressed about my own job after reading this status page.
bufferoverflow 9y ago
It's not an upgrade, it's a bugfix. Probably an important bug, they don't want to risk tens/hundreds of millions of dollars.
- verroq 9y ago
  They could have left the legacy system running before switching irreversibly to the new bugged system. That way they can rollback the change.
  - evgen 9y ago
    Unless the bug being fixed was some sort of critical security failure that enabled people to game the exchange or manipulate balances. I am guessing this is more of a screw-up by amateurs than a 'take it all down until this hole is patched' situation, but there is a slim possibility that keeping it up was a bigger risk than the reputation hit of extended downtime.
    - hrpnk 9y ago
      Looks like these are announced improvements to the Exchange engine. Really look forward to a post-mortem on this issue. Curious if they had issues with database migrations or such, that required recovery from backup in case of errors. Plus, for sure there were bugs in the code itself.
  - bufferoverflow 9y ago
    No, they couldn't. It's a large trading platform. Any serious bug can lead to the theft of tens of millions of dollars worth of cryptocurrencies.

frik 9y ago

As someone wrote It took three days of downtime for MtGox to admit that bitcoins were missing. @krakenfx is on day 2. I'm hoping the same thing isn't happening at Kraken, but not confident. Not so nice they tried to hide the news on HN twice by flagging them: https://news.ycombinator.com/item?id=16131750 https://news.ycombinator.com/item?id=16131750 and https://news.ycombinator.com/item?id=16127180 https://news.ycombinator.com/item?id=16127180 (both hidden after a short time)

hal9000xp 9y ago

Latest Kraken update says that: > We are still working to track down an elusive bug which is holding up launch. This bug did not appear in our many weeks of testing and only emerged in the production environment. Unfortunately, it is not consistently reproducible and we cannot launch until this issue is resolved. Sounds pretty realistic to me. During my 8+ years of programming, I had a lot of pain and sleepless nights trying to catch elusive bugs which can be reproduced only in production servers and only sometimes. Many times, my clueless managers got nervous and demanded estimated time from me. Unfortunately, it's impossible to give realistic estimations. It could take hours, days or weeks. Also, searching for such bugs doesn't scale well. It means if you have 5 devs looking for elusive bug, it doesn't mean they will find a bug 5 times faster.

koonsolo 9y ago
I even had clueless managers who demanded I explained the technically complex problem to them, because "maybe our insights could shed new light on the problem". They just don't get that best thing to do is leave us alone and let us do our job. They are just nervous because they are totally powerless in such situation. In a factory, a manager might be smarter than the one doing manual labour. In tech, it's the other way around.
- setr 9y ago
  If you assume potential malice (ie a lazy dev just not handling the bug at all) then a manager is required to ask about the bug, if only to try to verify that you've put real effort into the problem (based on say, confidence in whatever you spew). The worst possible scenario is that five months later, it turns out no one actually looked into properly: they just claimed to.
  - koonsolo 9y ago
    In that case, the manager should have fired that dev a long time ago. Best managers I worked for had trust in their developers, and asked for estimates on planning (not bugs). The worst managers didn't trust us, and pushed their own planning on us because they thought it would make us work faster. They thought planning can be negotiated. I can tell you which projects were within 7% of the deadline, with good quality. And which ones we were always putting out fires, and some never finished.

hrpnk 9y ago

They're back with free trading until end of January: https://blog.kraken.com/post/1449/kraken-returns-with-free-trading/ https://blog.kraken.com/post/1449/kraken-returns-with-free-t...

jnordwick 9y ago
Down again. Amateur hour. Lol.
- Bombthecat 9y ago
  Still no withdraw..

azzhohle 9y ago

Kraken is notoriously slow, unresponsive and buggy exchange and frankly I dont know why they get away with it. For months or year(s) you often run into site unavailable errors (load balancing issues?). Or if you are lucky to get to the trading screen you are unable to get trades through because of engine issues (capacity issues again?). Sometimes all you see is an empty red background rectangle where the error text should be indicating no dice but then minutes/hours later you might get one or more of the positions you tried to open despite only having gotten (ambiguous) errors. Let's hope they finally fixed their engine and purchased more capacity.

valdiorn 9y ago

Upgrade a 750 million dollar a day trading system with no rollback strategy. Amateur hour, and if this were a "real" exchange, they'd lose their licence and have lawsuits and government agents knocking down their door (real-money MTFs have strict uptime and infrastructure requirements for anyone who didn't know)

arisAlexis 9y ago

everything is up and trading is free for the rest of the month. </drama>

Turbots 9y ago
But nobody can withdraw their cash or transfer to other exchanges... They are literally holding their customers hostage. I just want out of this hellhole of a website, but they won't let me. Criminal!

lowlevel 9y ago

They say the next big thing is here, that the revolution's near, but to me it seems quite clear that it's all just a little bit of history repeating

rootsudo 9y ago

IT sounds like this week is going to be profitable on Kraken if you trust their status reports.

hndamien 9y ago

I tend to trust them for security because they are handling the Gox bankruptcy. For uptime... not so much.