11 ms·
With 61 Seconds in a Minute, Markets Brace for Trouble
- dijit 11y agoLast time this happened I was the only sysadmin in a fairly small ecommerce company dependant on mysql. The leap second caused our databases to all peg at 100% CPU/memory consumption.. Given the wide adoption and commercial nature of mysql, I can't begin to imagine how systems designed to be super precise will handle this..
- valleyer 11y agoWhat was the root cause of the problem?
- soup10 11y agoif(seconds>60) while(1) malloc(1);
- deleted 11y ago[deleted]
- ars 11y agoIt was a linux kernel bug, not a MySQL bug: https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/comment-page-1/#comment-1471 https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-se...
- calineczka 11y agoI experienced similar problems in other kinds of applications as well. 100% cpu usage until restart.
- jerf 11y agoIf you experience that problem this year, this supposedly can be fixed without even restarting the process by running "date -s now". I say "supposedly" because I can't validate, but based on my reading of the problem and why that works, I have no reason to believe it won't.
- mootothemax 11y agoThe leap second caused our databases to all peg at 100% CPU/memory consumption. I remember being utterly ridiculed that weekend, after swearing about fixing servers on Facebook. It didn't take long for apologies to arrive on the following Monday, once the corporate sysadmins logged in and saw anything Java-based had also massively, massively failed. It's the small (and petty!) victories.
- kbd 11y ago> anything Java-based had also massively, massively failed. What was the error in Java?
- xedarius 11y agoWonder why they don't add an average weight to the time rather than this troublesome extra second.
- bad_user 11y agoWhat do you mean?
- to3m 11y agoYou could add an extra second by having the previous 1,000 seconds (16 hours) last 1,001ms each, say. Then the scope for confusion is more likely to be limited to software that reads human time, and counts milliseconds, and requires the two to match over time. Probably a much smaller percentage of all programs. When you know what the problems are likely to be, you can of course pick the approach that's least likely to cause you problems! But if you're going in blind, then this seems like it could be a safer way to do it. (You might use some kind of a curve to do this, so it's smooth like. I imagine this is what the comment about weighting refers to.)
- ucho 11y agoThat's how Google does that: http://googlecloudplatform.blogspot.com/2015/05/Got-a-second-A-leap-second-that-is-Be-ready-for-June-30th.html http://googlecloudplatform.blogspot.com/2015/05/Got-a-second... . The drawback is that their clock will be "wrong" for most of that day.
- deleted 11y ago[deleted]
- bad_user 11y agoSo the problem is that seconds were measured relative to the duration of a day, but that standard changed and now seconds are measured with atomic clocks, making a day to not be equal to 24 hours, yet we pretend that it is, hence the need for leap seconds. I understand that this isn't ideal for people dealing with actual dates and times, but the upside is that timespans expressed in seconds are now universal, so for example Unix timestamps actually have meaning that doesn't change according to earth's rotation.
- lucb1e 11y agoThis sounds rather exaggerated to me. 10% of the systems will have trouble with it? Perhaps during the leap second something will bug or look weird, but outside of that second I doubt it's even 1% that will have trouble recovering. Also, many systems don't even support the leap second, nothing will happen at all until they sync with time servers the next time and correct for the second. Many systems are off a second or so anyway. As a quick example, Javascript is not designed to handle leap seconds correctly, so I doubt we'll ever see the value 60 out of new Date().getSeconds(). Things will just be off by 1 second for a second. Edit: Another example from the Windows Time service: > The Windows Time service does not indicate the value of the Leap Indicator when the Windows Time service receives a packet that includes a leap second. [...] Therefore, after the leap second occurs, the NTP client that is running Windows Time service is one second faster than the actual time. This time difference is resolved at the next time synchronization. It will just be a second off and resynchronize next time.
- spacefight 11y agoIf your system is one of those 10% and if your trading algo is executing fucked up trades because of timing issues and you lose real money - I wouldn't call that exaggerated. Also, not sure if any of the systems mentioned in the article run JS for critical server side code.
- lucb1e 11y agoHigh-frequency trading and other important realtime processes that handle money should of course be tested for this scenario (either by simply not trading for a few seconds before and after, or by properly handling it). The vast majority of the computer systems around the world are however not HFT systems and the world will continue to work normally. No Y2K-like doom scenarios needed like the article suggests. We've been here before, June 30th 2012 also contained a leap second (see Wikipedia). The article only forecasts doom without looking back at previous leap seconds and evaluating its effect.
- ars 11y ago> We've been here before, June 30th 2012 also contained a leap second (see Wikipedia). The article only forecasts doom without looking back at previous leap seconds and evaluating its effect. Clearly you didn't read the article. If you had you would have seen that they noted that last time it was on a weekend when all the markets were closed.
- tempodox 11y agoAh, this is one high-profile example of the CAP theorem in full action (not counting Linux kernel bugs).
- jimmcslim 11y agoThe article states that the planet's speed of rotation is slowing down, albeit very slowly. Is the implication that at some point in the (distant) future the rotation will stop completely, or will it just reach a new equilibrium? Could it in fact speed up again?
- anon4 11y agoI believe the time-scale is such, that the Sun will swallow the Earth long before it stops spinning.
- noir_lord 11y agoIt can't speed up without an external force and at current slow rate it'll take about 2 trillion years to stop (this assumes a linear rate of slowing over time, which won't happen). Much of the slowing is caused by tidal friction with the Moon (which recedes slightly each year), iirc eventually the Earth will become tidally locked with the Sun (same side facing all the time).
- saiya-jin 11y agowe will lose moon well before that, it's going further away every year (don't recall exact value, should be few cm per year)
- bad_user 11y agoOur sun will turn into a red giant and destroy earth long before earth has a chance to stop its rotation, as we're talking about an increase in milliseconds per century. Since 1820 the earth's rotation slowed down by about 2.5 milliseconds.
- valleyer 11y agoThat sounds like it contradicts with the fact that we've had dozens of leap seconds in the last several decades. Explain?
- 11y ago
- h43k3r 11y agoIf 1 second can cause so much of havoc, I am wondering, to what precision are the servers in markets synchronized? Are they made in a distributed fashion where it is assumed that two clocks can never be synchronized with full precision. PS - I am a noob in markets and trading.
- kasey_junk 11y agoClock sync is one of the areas where the markets and their participants actually deserve their reputation for sophistication. That said, I'd be surprised if keeping the clocks together is what is causing the worry. Much more troubling are third party bugs (kernel, jvm, db, etc) and incorrect assumptions in their code. This same story can be equally applied to any industry honestly, but the markets make for better headlines.
- easytiger 11y agoGPS + PTP +- < 1us GPS + NTP +- 12us
- spacecowboy_lon 11y agoAs opposed to the situation in Greece I have lost over £1k since the Friday close and its only Lunch time :-)
- cnvogel 11y agoWow, that's a pretty bad illustration of leap smearing. I think a much nicer illustration would have been roughly like this (with obvious limitations caused by the ASCII spacing): Real: true time, showing the 61st second Comp: computer time, only counting 60 seconds not smeared, time counted at regular speed ...--->| |<---... Real 54 55 56 57 58 59 60 00 01 02 03 Comp 54 55 56 57 58 59 00 01 02 03 |<- smearing ----------->| time counting slowed down by 20%
- MereInterest 11y agoAlso, who the heck decided to call it a "leap smear"? There isn't anything "leap"ing about it anymore. If you are changing from "leap second" to something indicating smearing, "smear second" is the logical choice.
- pdpi 11y agoWell, you're turning a leap into a smear, so "leap smearing" seems like a perfectly apt description.
- nadocrew 11y agoAre there any security implications for leap seconds? Do pseudo random number generators have any issues with 61 seconds in a minute?
- delinka 11y agoI have yet to meet a [CS]PRNG that cares about the passage of time.
- jessaustin 11y agoSome protocols that use prngs might make assumptions about the number of seconds in a minute, however.
- alkonaut 11y agoAnother reason to just have artificial delays in trading. If every exchange had a minimum of, say, a minute after an order is submitted until it is executed (during which time it couldn't be amended or cancelled of course) it seems you would greatly reduce the risk of "flash crashes"and unnatural manipulation of the market. I know "market liquidity" is an argument for allowing HFT, but haven't hard a good explanation about why the economy would suffer so much if the ultra high frequency trading just wouldn't exist. Note: I have about as much knowledge about this as people who suggest how SpaceX should improve. Just sayin. Of course computer systems everywhere will have issues with an extra seconds (deltas being negative that can't be etc) but most of those systems aren't trading systems. Why is it that "markets" are in the focus of this? At y2k we were worried about planes falling out of the sky. Surely there must be worse things than stock market computer systems being confused?
- kasey_junk 11y ago> but haven't hard a good explanation about why the economy would suffer so much if the ultra high frequency trading just wouldn't exist. The short explanation is that the speed is a tool for managing risk (like most other market tools) and by removing it, you remove that ability. That risk must be priced into the market somewhere and that will be in the spread that everyone pays. Will that be worse than what we currently have? Who knows, but what we have is working pretty well with regard to providing liquidity (it is cheaper and easier to get in more markets than ever before and the margins are as low as they have ever been on providing it), so why mess with something that has so few down sides? > Surely there must be worse things than stock market computer systems being confused? There are, but Bloomberg doesn't specialize in them, and they don't garner nearly the HN upvotes.
- denim_chicken 11y ago> it is cheaper and easier to get in more markets than ever before and the margins are as low as they have ever been on providing it High enough that the firms are at least receiving a normal profit. And given the opportunity cost that all these smart people incur by working in this industry, this normal profit is rather big, and no doubt that the extraction of this normal profit has non-neglible deleterious effects on the economy.
- rm_-rf_slash 11y agoPeople who were losing their shit over Y2K had no idea what normal life would become.
- Lawtonfogle 11y agoCan someone please explain why this keeps being explained as 'earth's rotation is slowing down'? Having a scheduled addition of the leap second only corrects an issue resulting from our timing being slightly off (by about 1 second every couple decades). Now, if the second had to be add in changing frequency... the changing in frequency would be a result of the earth's rotation changing... but even still that wouldn't be why the second itself is being added, is it?
- mxyzpt1k 11y agoFrom the very end of the article... In the meantime, the Paris-based International Earth Rotation and Reference Systems Service will keep track of the gradual slowing of the planet, caused in part by drag created by the Moon. Millions of years ago days were 22 hours long.
- Lawtonfogle 11y agoI know the speed of rotation changes slowly. But that would be a reason for lengthening a day, not adding a leap second. The leap second is added because our concept of time is slightly off, but it keeps being presented as if we had to add an extra second because of the slowing of the rotation. If that was the case, it would be like those 2 hours that were added in the last few million years. We don't consider them leap years because they aren't a correction but a redefinition (ignore for a moment that humans weren't around to define as a day as 22 hours back then). Consider the leap year for a second. We add it because our years have 365 days is slightly incorrect (by about .25 days a year). It isn't that the number of rotations per revolution is or isn't changing. That isn't relevant to our use of a 1 day correction every 4 years. If some force caused us to revolve around the sun slightly faster and it took 365 exact, then we could remove the leap year. Removing the leap year would be a result of our revolution being slightly faster. But the leap year itself is not. In short, the delta of the leap correction is a result of the change in the system being monitored, but the leap correction itself is because our system of measurement is off.
- jessaustin 11y ago
- lappa 11y agoThis seems to be something that would benefit from standardization and modularity... Pausing for a second, going back a second in time or changing how long a second is all are hacks that are bound to cause some edge case failure somewhere. I'd like to see a minute have a 60th second just as February has a 29th day.
- mikeash 11y agoThe problem is that the effect being compensated for is so tiny that it's really noisy. It's not like leap years, where all you have to do is measure the actual period of the Earth's orbit very precisely and come up with a nice way to fit the appropriate correction factor into the calendar. The Earth's rotation is slowing, but it's not a steady process. For example, the Boxing Day earthquake in 2004 permanently decreased the length of the day by about 3 microseconds. Here is a graph of the changes over the past few decades: https://upload.wikimedia.org/wikipedia/commons/5/5b/Deviation_of_day_length_from_SI_day.svg https://upload.wikimedia.org/wikipedia/commons/5/5b/Deviatio...
- lappa 11y agoAs long as we have a standardized way of determining when leap seconds are going to happen, some authority determining and announcing, there shouldn't be problem. Leap years are predictable, but leap seconds can be accounted for a decade after an event disrupts the pattern. Also, given all that variance and the fact that we only seem to be shifting around 1 minute per century at the current rate, we probably could get by having a leap minute every few decades when necessary. This works fine for any system trying to interact with other computers and humans while keeping consensus on the time. Any system that is trying to determine earths yaw would no longer be able to rely on F(time), but that is only a minor inconvenience.
- mikeash 11y agoHow is that different from what's being done now? There is a standardized way of determining when leap seconds are going to happen (they always happen on June 30th or December 31st when the deviation reaches about 0.6s), there is an authority that determines and announces it (the International Earth Rotation and Reference Systems Service) and it is, in fact, done by causing the last minute on the day in question to have 61 seconds instead of the usual 60. The complication is just that there are a whole lot of systems out there that assume that every minute has exactly 60 seconds with no exceptions, so you end up with crazy workarounds like smearing to hide that 61st second from such systems.
- mikeash 11y agoWhy aren't leap seconds treated as a time zone change, like a daylight saving time transition? Keep the underlying clock counting the number of actual seconds since midnight, January 1st, 1970 (or whatever your epoch of choice is) and just add one more second into the offset to local time. Is there some big problem with this approach I haven't thought of, or is there just too much history with doing it the other way to change now?
- Someone1234 11y agoI don't understand why either. Counting up from 1/1/1970 and then making adjustments for rendering only seems like a simple, reliable, and time tested solution. I guess people don't want a database of leap seconds and a lookup each time time is rendered, but ultimately it seems like the best solution which is least likely to break things.
- Scramblejams 11y agoYeah. I'd like to see us all set our hardware clocks to TAI and everything else comes from the time zone files. It won't solve everything because lots of protocols have broken notions of time, but at least we could all agree on what the hardware time is supposed to mean and there would be no worry about leap seconds at that level.
- danielweber 11y agoThe nice thing about time standards is that there are so many to choose from! One drawback of your approach is that I can't just see if a time = 0 mod 60 to check if we are on a minute boundary. Your way is what is used by the GPS network, IIRC, and it works for them, so there you go.
- mikeash 11y agoIf your dates and times and locations vary enough, you already can't do a mod 60 check on UTC to detect a minute boundary, as various historical time zones include seconds-level offsets from UTC. For example, US cities were on local solar time until the latter part of the 19th century, and the changeover to uniform time zones involved lots of seconds-level offsets. Interesting stories here: http://historymatters.gmu.edu/d/5748 http://historymatters.gmu.edu/d/5748
- Balgair 11y agoMy brother used to work for an atomic clock company. Their array of clocks was the 2nd most precise next to NIST in Colorado, he said, and when small fluctuations in the earth occurred, their's was another point of contact for GPS timekeeping (allegedly). Reading through the OP, I remember that this sort of mid year leap second happened a few years back with the Japanese major earthquake. My brother said that they had to reset all the clocks about a millisecond due to the shifts in the rotation of the earth because of the quake. One would think that the programmers would have written into these market timing devices a simpler way to adjust these calculations, seeing as these timing updates keep on happening.
- lifeisstillgood 11y agoI would suggest the whole thing can be avoided by the exchanges setting their own ticks. Any and all Bid/offers are queued during the tick and then applied, in order, during the next tick. before I did some research I thought this would be a good "anti-HFT abuse" idea but I still think it has considerable merits.
- InclinedPlane 11y agoWhy are leap seconds still considered a good idea? They have almost zero practical value and a huge practical cost.