7 ms·
Some DNS lookups causing 5xx errors due to leap second bug
- aburan28 10y agoI was wondering who this leap second was going to affect!
- karlhughes 10y agoThis was shared a while ago, but it's relevant again: http://www.madore.org/~david/computers/unix-leap-seconds.html http://www.madore.org/~david/computers/unix-leap-seconds.htm...
- deleted 10y ago[deleted]
- gamegoblin 10y agoI guessed most big services would be using something akin to time smearing [1] since the first big leap-second outages years ago. Is there any reason why cloudfare would be unable to use this technique? [1] https://developers.google.com/time/smear https://developers.google.com/time/smear
- wyager 10y agoIt's pretty lame that a lot of software is so fragile that it breaks if we give it the correct time. The solution here is that any software that relies on accurate timing and/or breaks when you change the time should be using epoch seconds, not any sort of human-oriented time format.
- rspeer 10y agoLeap seconds are a change to the number of UTC epoch seconds.
- wyager 10y ago> Leap seconds are a change to the number of UTC epoch seconds. UTC is different from epoch time. Epoch time by definition does not count leap seconds. https://en.m.wikipedia.org/wiki/Unix_time https://en.m.wikipedia.org/wiki/Unix_time
- zerd 10y agoBut it is affected by leap seconds. According to the example labeled "Unix time across midnight when a UTC leap second was inserted on 1 January 1999" the unix time went backwards. So using epoch/unix time does not help.
- EGreg 10y agoIs it safe to say that time() in php and Date.now() in JS do not care about leap seconds?
- perennate 10y agoFrom [1], it is likely messier than that: > The time() function will resolve to the system time of that server. If the server is running an NTP daemon then it will be leap second aware and adjust accordingly. PHP has no knowledge of this, but the system does. > It took me a long time to understand this, but what happens is that the timestamp increases during the leap second, and when the leap second is over, the timestamp (but not the UTC!) jumps back one second. [1] http://stackoverflow.com/questions/7780464/is-time-guaranteed-to-be-leap-second-aware http://stackoverflow.com/questions/7780464/is-time-guarantee...
- dingaling 10y agoAnd alternatively I guessed that big services like Cloudflare, responsible for fronting 2.5 million websites, would have been running a preproduction environment clocked-forward to 2017.
- jlgaddis 10y agoI'm guessing CloudFlare runs their own custom DNS server software?
- ckdarby 10y agoIs this not it? https://github.com/cloudflare/dns https://github.com/cloudflare/dns
- alexforster 10y agoYeah, and Go doesn't expose a monotonic clocksource in its stdlib[1]. I'd bet that's what this boils down to. [1] https://github.com/golang/go/issues/12914 https://github.com/golang/go/issues/12914
- raverbashing 10y agoWow, that discussion is cringeworthy I really thought anyone discussing systems programming should be aware of the need for a monotonically increasing clock source REALLY
- kondbg 10y agoI'm confused on why a DNS server would need to rely on a monotonic clock for its use cases. Is there a part of DNS that relies on the assumption of synchronized, monotonic time? (Perhaps TTL/expiry of records? But I still don't see why having a non monotonic clock source would harm if CF is using Go timers for expiry)
- nullc 10y agoMy CDMA phone dropped service for a few minutes after the leap second. It's absurd that we continue to keep subjecting ourselves to these disruptions and the considerable amount of work that goes into handling leap seconds for the systems that aren't disrupted by them. Leap seconds serve no useful purpose. Applications that care about solar time care usually care about the local solar time, while UT1 is a 'mean solar time' that doesn't really have much physical meaning (it's not a quantity that can be observed anywhere, but a model parameter). It would take on the order of 4000 years for time to slip even one hour. If we found that we cared about this thousands of years from now: we could simply adopt timezones one hour over after 2000 years, existing systems already handle devices in a mix of timezones. [And a fun aside: it appears likely that in less than 4000 years we would need more than two leapseconds per year, sooner if warming melts the icecaps. So even the things that correctly handle leapseconds now will eventually fail. Having to deal with the changing rotation speed of the earth eventually can't be avoided but we can avoid suffering over and over again now.] There are so many hard problems that can't just easily be solved that we should be spending our efforts on. Leapseconds are a folly purely made by man which we can choose to stop at any time. Discontinuing leapseconds is completely backwards compatible with virtually every existing system. The very few specialized systems (astronomy) that actually want mean solar time should already be using UT1 directly to avoid the 0.9 second error between UTC and UT1. For all else that is required is that we choose to stop issuing them (a decision of the ITU), or that we stop listening to them (a decision of various technology industries to move from using UTC to TAI+offset). The recent leap smear moves are an example of the latter course but a half-hearted one that adds a lot of complexity and additional failure modes. (In fact for the astronomy applications that leap seconds theoretically help they _still_ add additional complication because it is harder to apply corrections from UTC to an astronomical time base due to UTC having discontinuities in it.)
- userbinator 10y agoI'm curious what if anything would be problematic if everything just effectively "ignored" leap seconds (i.e. would this outage not have occurred?) --- one minute is always 60 seconds, an hour is always 60 minutes, and a day always 24h. I mean, if you consider the fact that human society has managed to function perfectly well with almost everyone not knowing nor caring what a leap second is, and yet apparently some software does --- leading to problems like this --- something doesn't feel right.
- na85 10y agoFinancial systems need very precise timekeeping. I'm sure other fields do as well.
- userbinator 10y agoThey do, and actually HFT did come to mind when I was writing that comment, but then I realised that, as explained in https://news.ycombinator.com/item?id=13294747 https://news.ycombinator.com/item?id=13294747 , they have no need to precisely synchronise time with the rotation of the Earth, and would be fine without leap seconds.
- rspeer 10y agoBut financial systems don't care if the position of the sun in the sky is a couple of seconds off from where a model says it should be. Astronomers would care about that, but they already don't use UTC.
- XorNot 10y agoBut they do care what GPS reference time is. Which are satellites very much dependent on holding an accurate position in the sky (which is dependent on the Earth's rotational speed, which changes, which is why we have leap seconds). So go figure: which part of this system should be broken because people keep ignoring that leap seconds happen?
- 10y ago
- ComputerGuru 10y agoI was at a relative's and tried to load two different web sites.. my first thought was that their wifi sucked. My second was "will we finally learn a lesson today about the disturbing trend towards constant re-centralization of all our online services?"
- mikehollinger 10y agoI'll just leave this here: Have a look at a j excellent video that explains why time algorithms are hard to sort out: https://m.youtube.com/watch?v=-5wpm-gesOY https://m.youtube.com/watch?v=-5wpm-gesOY Happy New Year from Austin!
- zitterbewegung 10y agoSo you can basically see if a tool or company hasn't experienced a leap second if their system goes down because of it.
- homero 10y agoI didn't see an outage at all
- ChuckMcM 10y agoOnce again we're screwed by different people wanting "time" to mean different things. There is no hope for humanity once we start traveling anywhere close to light speed into and out of the solar system. I propose a new "non-time" time system. It has exactly two real values which range from 0 to tau and an integer, the first real number is radians of earth rotation, and the second is radians of the rotation around the Sun. The integer reflects the number of complete cycles. So lunch time in Greenwich 'pi'. It has the benefit that its "source" is actually the planet, so we can use a telescope at Greenwich to pick a certain alignment of stars as the "zero", "zero" point and then each time it realigns to that exact point, you can increment the "year" count. I believe we can build a robust system to support this out of stone. We'll need to create a circle of stones but using a small hole drilled through a stone and a marker on the ground we can always identify 0.0,0.0, 0.0,pi/2, 0.0, pi, and 0.0, 3*pi/2.
- EGreg 10y agoWhat if the stones move?
- ChuckMcM 10y agoGuess we better use really large stones :-)
- Godel_unicode 10y agoYou do know that stars move, right? And not even all in the same direction or at an easily predictable speed. It's illustrative about how hard time is that you tried to create a new system from scratch, with the express purpose of being future proof for space travel, and it's already broken because the fixed point you chose is not, in fact, fixed. Edit: grammar
- carbocation 10y agoI believe that ChuckMcM is making a lighthearted, elaborate reference to Stonehenge. (If not, nevermind!)
- thisrod 10y agoThere is a higher order issue here. DNS time stamps have been stable for decades. Why has anyone written new code to format them since the last leap second?
- zkms 10y agoWhat causes real-world problems with leap seconds is actually unrelated to the nasty interactions of metrology and solar time -- it's a specific and avoidable problem with how NTP (and many OSes/languages) represent time -- it's a types issue. The right way for computers to represent time is with a number that represents the number of constant-rate ticks that have elapsed past a some agreed-upon epoch. If you know what the epoch is and how long each tick is (lots of people use 1 / 9.192 GHz), it is easy to know how many ticks are between any two time values, and you can convert a time value with one epoch to one with a different epoch and tick rate -- you can do everything people expect to do with time. There are no numbers that represent an invalid time value, and for each moment, there is a unique time value that represents it. There's a one-to-one mapping with no nasty edge cases. Leap seconds are a step function that is added to a constant-rate timescale (whose name is "TAI") in order to generate a discontinuous timescale (whose name is "UTC") that never is too different from solar time. There is nothing fundamentally abhorrent about leap seconds -- there are just good and bad ways to represent, disseminate, and compute with timescales that involve leap seconds. The right way to handle leap seconds can be seen with many GNSSes and PTP (very high precision hardware-assisted time synchronization over Ethernet). GPS, BeiDou, Galileo, and PTP all involve dissemination and computation on time values -- and with dire consequences for failure/downtime/inaccuracy. The designers of those systems all somehow converged on the choice to separate out the nice, predictable, constant-rate and discontinuity-free part of UTC from the nasty step function (the leap second offset). Times in all those systems are represented as the tuple (TAI time at t, leap offset at t). This means that the entire system can calculate and work with (discontinuity-free and constant-rate) TAI times but also truck around the leap offsets so when time values need to be presented to a user (or anything that requires a UTC time), the leap offset can be added then. Crucially, all the maths that are done on time values are done on TAI values, so calculating a time difference or a frequency is easy and the result is always correct, regardless of the leap second state of affairs. Representing UTC time as a tuple makes the semantics of that data type easy to reason about -- the "time" bit is in the first element and is completely harmless -- the edge cases have all live in the second half of the tuple. NTP and Unix (and everything descending and affected by those) have made the mistake of representing and transmitting time as a single integer, TAI(t) + leap offset(t). This is not a data representation that has sensical semantics and it is very hard to reason about it. First of all, the leap second offset is nondeterministic and also unknown -- there is no way to get it from NTP and there is no good way to know the time of the next leap event. Second of all, there are repeated time values for different moments in time (and when a negative leap second will happen, there will be time values that represent no moments in time). Predictably, introducing nondeterministic discontinuities doesn't work so well in the real world. There are a bunch of bugs in NTP software and OS kernels and applications that make themselves shown every time there is a leap second. It's not even just NTP clients that struggle -- 40% of public Stratum-1 NTP servers had erroneous behavior [0] related to the 2015 leap second! Given that level of repeated and widespread failure, the right solution is not to blame programmers -- it should be to blame the standard. The UTC standard and how NTP disseminates UTC are fundamentally not fit for computer timekeeping. GNSS receivers and PTP hardware get used in mission-critical applications (synchronizing power grids and multi-axis industrial processes, timestamping data from test flights and particle accelerators) all the time -- and even worse, there's no way to conveniently schedule downtime/maintenance windows during leap second events! "Leap smear" isn't an acceptable solution for those applications, either -- you can't lie about how long a second is to the Large Hadron Collider. GNSS and PTP systems handle leap second timescales without a hitch by representing UTC time with the right data type -- a tuple that properly separates two values that have the same unit (seconds) but have vastly different semantics. The NTP and unix timestamp approach of directly baking the discontinuities into the time values reliably causes problems and outages. The leap second debacle is not about solar time vs atomic time; it's about the need for data types that accurately represent the semantics of what they describe. [0]: http://crin.eng.uts.edu.au/~darryl/Publications/LeapSecond_camera.pdf http://crin.eng.uts.edu.au/~darryl/Publications/LeapSecond_c...
- known 10y agoI just did sudo rdate -s time-a.nist.gov
- iopq 10y agoIs that why Google Maps was down?
- web007 10y agoAre there any public "skewing" NTP pools that distribute the leap seconds as lag / gain over 24 or 48 hours as some of the large providers do? That seems to be the generally accepted answer to leap-second chaos, and certainly seems simpler than all of the hidden bugs in systems all over the place trying to deal with :60 on a clock.
- benjiweber 10y agoGoogle have https://developers.google.com/time/smear https://developers.google.com/time/smear but it can introduce different problems.
- tscs37 10y agoI knew it. I knew that something is going to break somehow because for some reason people continue to falsely believe that 1 minute always has 60 seconds.
- brongondwana 10y agoWas glad things have improved since 4 years ago! https://blog.fastmail.com/2012/07/03/a-story-of-leaping-seconds/ https://blog.fastmail.com/2012/07/03/a-story-of-leaping-seco... This time I didn't get paged for anything on leap second day :)
- justinholmes 10y agoFunny that they wrote about it in 2014 https://blog.cloudflare.com/its-go-time-on-linux/ https://blog.cloudflare.com/its-go-time-on-linux/
- dmd 10y agoHalf the people posting here need to read https://qntm.org/calendar https://qntm.org/calendar
- tyingq 10y agoThey apparently run their own DNS proxy called "RRDNS", written in golang. https://blog.cloudflare.com/tag/rrdns/ https://blog.cloudflare.com/tag/rrdns/
- tim_hutton 10y agoSurely now we can agree that it is FINALLY time to adjust our planet's orbit to correct for this problem once and for all.