12 ms·
NTP at NIST Boulder Has Lost Power
- crazydoggers 9mo agoStatus of NIST time servers: https://tf.nist.gov/tf-cgi/servers.cgi https://tf.nist.gov/tf-cgi/servers.cgi
- renewiltord 9mo agoWell, where did NTP at NIST last put it? Did they look there?
- Y_Y 9mo agoYou misunderstand, there's been a coup
- renewiltord 9mo agoWe have to stop those knaves pushing PTP! NTP must prevail!
- adastra22 9mo agoOf course there is. Where else would they put the reference standard chickens?
- qmarchi 9mo agoMan, they're having a hell of a time up in Boulder.
- Animats 9mo agoNIST campus status: Due to elevated fire risk and a power outage for the Boulder area, the DOC Boulder Labs campus is CLOSED on December 19 for onsite business and no public access is permitted; previously approved accesses are revoked.[1] WWV still seems to be up, including voice phone access. NIST Boulder has a recorded phone number for site status, and it says that as of December 20, the site is closed with no access. NIST's main web site says they put status info on various social media accounts, but there's no announcement about this. [1] https://www.nist.gov/campus-status https://www.nist.gov/campus-status
- Scaevolus 9mo agoNote that WWV is a good 60 miles NNE of Boulder on the outskirts of Fort Collins.
- cdfuller 9mo agoCan anybody expand on the implications of this? Being unfamiliar with it, it's hard to tell if this is a minor blip that happens all the time, or if it's potentially a major issue that could cause cascading errors equal to the hype of Y2K.
- ThrowawayTestr 9mo agoIf your computer was using it as your time server and you didn't have alternatives configured your clock my have drifted a few seconds.
- Roark66 9mo agoI never checked it, but how much a typical's pc/server's clock does actually drift over a week or a month? I always thought it's well under a second.
- soared 9mo agoI have an extremely cheap and extremely low power automatic cat feed - it’s been on 2 D batteries for 18 months. I just reset it after it had drifted 19 minutes, so about 1 minute a month, or 15 seconds a week!
- bhouston 9mo agoClocks do drift. Seconds a week is definitely possible. I think there are varying quality of internal clocks in electronic devices, and the cheaper the quality the more drift there is. I think small cheap microcontrollers can drift seconds per day.
- ycui1986 9mo agocheap microcontrollers use RC oscillators. If they only drift a few seconds a day, that would be an achievement by itself. RC oscillator is poor enough that early days USB communication would fail if running on RC clock.
- 9mo ago
- lovich 9mo agoThis was an NTP 0 server right? What is the actual failback mechanism when that level of NTP server fails? This is some level of eldritch magic that I am aware of, but not familiar with but am interested in learning.
- lambdaone 9mo agoThere are lots of Stratum 0 servers out there; basically anything with an atomic clock will do. They all count seconds independently from one another, all slowly diverging over time, with offset intervals being measured by mutual synchronization using a number of means (how is this done is interesting all by itself). Some atomic clocks are more accurate than others, and an ensemble of these is typically regarded as 'the' master clock. To quote the ITU: "UTC is based on about 450 atomic clocks, which are maintained in 85 national time laboratories around the world." https://www.itu.int/hub/2023/07/coordinated-universal-time-an-overview/ https://www.itu.int/hub/2023/07/coordinated-universal-time-a... Beyond this, as other commenters have said, anyone who is really dependent on having exact time (such as telcos, broadcasters, and those running global synchronized databases) should have their own atomic clock fleets. There are thousands and thousands of atomic clocks in these fleets worldwide. Moreover, GPS time, used by many to act as their time reference, is distributed by yet other means. Nothing bad will happen, except to those who have deliberately made these specific Stratum 0 clocks their only reference time. Anyone who has either left their computer at its factory settings or has set up their NTP configuration in accordance to recommended settings will be unaffected by this.
- Maxious 9mo agoThere's two other sites for the time.nist.gov service so it'll be okay. Probably more interesting is how you get a tier 0 site back in sync - NIST rents out these cyberpunk looking units you can use to get your local frequency standards up to scratch for ~$700/month https://www.nist.gov/programs-projects/frequency-measurement-and-analysis-service-fmas https://www.nist.gov/programs-projects/frequency-measurement...
- lovich 9mo agoWhat happens in the event all the sites for time.nist.gov go down? is it included in the spec? Also thank you for that link, this is exactly the kind of esoteric knowledge that I enjoy learning about
- arn3n 9mo agoWind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.
- Maxion 9mo agoSomewhat interesting that they themselves don't have access to the site. You'd think there would have been some disaster plans put in place?
- glkindlmann 9mo agoOf the various internet .+P, NTP is one I never learned about as a student, so now I'm looking at its web page [1] by its creator David L. Mills (1938-2024). I've found one video of him giving a retrospective of his extensive internet work; he talks about NTP at 34:51 [2] and later at 56:26 [3]. [1] https://www.eecis.udel.edu/~mills/ntp.html https://www.eecis.udel.edu/~mills/ntp.html [2] https://youtu.be/08jBmCvxkv4?si=WXJCV_v0qlZQK3m4&t=2092 https://youtu.be/08jBmCvxkv4?si=WXJCV_v0qlZQK3m4&t=2092 [3] https://youtu.be/08jBmCvxkv4?si=K80ThtYZWcOAxUga&t=3386 https://youtu.be/08jBmCvxkv4?si=K80ThtYZWcOAxUga&t=3386
- ssl-3 9mo agoHN discussion shortly after Dave Mills died, early in 2024: https://news.ycombinator.com/item?id=39051246 https://news.ycombinator.com/item?id=39051246
- torcete 9mo agoIn [3] he mentions that one can use NTP to observe frequency deviations and use it as an early warning system for fire and AC failure. That really intrigues me. Can you actually? Has this ever been implemented?
- magicalhippo 9mo agoOscillators of all kinds are temperature dependent. That's why the most stable ones are insulated and ovenized[1]. So an AC failure which would lead to higher room temperatures would lead to stronger or more frequent correction by the NTP client, as the local oscillator would drift more. Not sure about the fire case though. I mean the same applies there but I'm not imaginative enough to think of a realistic scenario where NTP would be useful for averting a fire. [1]: https://blog.bliley.com/anatomy-of-an-ocxo-oven-controlled-crystal-oscillator https://blog.bliley.com/anatomy-of-an-ocxo-oven-controlled-c...
- eichin 9mo agoI knew of some experiments in this space back in the late 1980s or early 1990s - but it was specifically with DECstation hardware that had terrible clocks (not used for alerting, just "this graphs nicely against temperature".) https://groups.csail.mit.edu/ana/Publications/PubPDFs/Greg.Troxel.Thesis.1994.pdf https://groups.csail.mit.edu/ana/Publications/PubPDFs/Greg.T... (PDF) 4.2.1 does talk about explaining local clock frequency changes with office temperature changes (because they overwhelm a clock-aging model) but it doesn't have graphs so perhaps they weren't clear enough to include (or just not relevant enough to Time Surveying.)
- themafia 9mo ago> Facility operators anticipated needing to shutdown the heat-exchange infrastructure providing air cooling to many parts of the building, including some internal networking closets. As a result, many of these too were preemptively shutdown with the result that our group lacks much of the monitoring and control capabilities we ordinarily have Having a parallel low bandwidth, low power, low waste heat network infrastructure for this suddenly seems useful.
- Ey7NFZ3P0nzAe 9mo agoLike reticulum? https://github.com/markqvist/Reticulum/ https://github.com/markqvist/Reticulum/
- DamonHD 9mo agoSo far I think I'm still seeing one of them in my peers list for my public-ish NTP server: remote refid st t when poll reach delay offset jitter ============================================================================== +time-e-b.nist.g .NIST. 1 u 372 1024 377 125.260 1.314 0.280
- DamonHD 9mo ago...and maybe it's gone: #time-e-b.nist.g .NIST. 1 u 1071 1024 377 125.260 1.314 0.280
- amelius 9mo agoThis makes me wonder, if you take the average time of all wristwatches on the planet, accounting for timezones and throwing out outliers, how close would you get to NTP time? And how many randomly chosen wristwatches would you need to get anything reasonable?
- varjag 9mo agoClose but unlikely to be precise in metrology sense. There's unlikely even a billion wrist watches being worn.
- b112 9mo agoOne. One watch. POTUS's watch. And in fact, that's why Boulder is currently shuttered... they disagreed.
- usea 9mo agoI think this comment is referencing the government's recent announcement[0] to shut down the National Center for Atmospheric Research in Boulder. They do climate research at the Mesa Laboratory there. It's open to the public for visits. They have a small science museum, offices, a library, etc. I highly recommend anyone with interest and opportunity to visit the mesa lab soon. It may not be open much longer. The view alone is worth the trip, and the building is cool too. [0]: https://x.com/russvought/status/2001099488774033692 https://x.com/russvought/status/2001099488774033692
- deleted 9mo ago[deleted]
- throwup238 9mo agoYou’re the person Douglas Adams warned us about.
- Intermernet 9mo agoComment of the year.
- keepamovin 9mo agoFor future reference of civilization: if a facility is critical, it must have a SMR.
- gilrain 9mo agoIt’d be a good idea to protect our infrastructure from the climate we created. It’s just a good idea, though, not a greedy one… so it won’t happen.
- sgnelson 9mo agoOne question I have is did DOGE decisions have anything to do with this? Because I know they took knives to NIST.
- cramcgrab 9mo agoActually DOGE involvement at the highest level would have resulted in Tesla solar and Tesla powerwall battery backups.
- redbush237 9mo agoLol, lmao. Some relevant DOGE’s effects: -time and frequency division director quit -NIST emergency management staff at least 50% vacant -NIST director of safety retired, and NIST safety was already understaffed when compared to DOE labs -NOAA emergency manager on the same Boulder campus laid off etc
- the_gipsy 9mo agoAny day now
- throw0101c 9mo ago> […] Tesla solar and Tesla powerwall battery backups. Don't forget Solar Roof.
- cramcgrab 9mo agoApparently no engineers here anymore, all armchair politicians.
- hiddencost 9mo agoWhen the politicians take a hatchet to the underpinnings of society that we built, we have to get political.
- cramcgrab 9mo agoSo sad, we used to innovate and stay one step ahead of them. That’s how wealth is created.
- sgnelson 9mo agoFYI, this was posted a month ago when discussing thermal effects of clock drift. I thought it was quite interesting view of what the WWVB location looks like: https://jila.colorado.edu/news-events/articles/spare-time https://jila.colorado.edu/news-events/articles/spare-time Discussed here: https://news.ycombinator.com/item?id=46042946 https://news.ycombinator.com/item?id=46042946
- 8organicbits 9mo agoThe referenced mailing list is this Google Group (https://groups.google.com/a/list.nist.gov/g/internet-time-service/c/o0dDDcr1a8I https://groups.google.com/a/list.nist.gov/g/internet-time-se...) which has some other posts about this incident.
- creatonez 9mo agoWhere are you seeing the other posts?
- jb1991 9mo agoThis is terrible. It’s always sad to hear about things like this.
- samch 9mo agoCan anybody speak to the current best practices around running underground power lines? I see these types of articles about above-ground distribution systems from time-to-time, particularly in California. I feel lucky that my area has underground power, but that was installed back in the 1980s. Would it be prohibitively expensive for Boulder’s utility provider to move to underground distribution? I can’t help but think it could be worth the cost to reduce wildfire risk and offer more reliable service.
- shadowpho 9mo ago1/3-1/2 of the cost of the electricity we pay is distribution. Some of it is physical infrastructure (transformers, wire, poles), but a lot of it is labor. Labor is expensive in US. It’s a lot of labor to do, plus they’ll likely need regulatory approval, buying out land, working through easements. At the same time you have people screaming about how expensive energy is. Furthermore they have higher priorities, replacing ancient aging infrastructure that’s crumbling and being put on higher load every day.
- pseudonymidy 9mo agoThink of it like this: overhead power lines require you to dig a 5-7’ deep hole that’s 2’ in diameter every 90’. Underground power supplied through cable requires you to bury the cable minimum 3’ in the ground in rigid ductwork the entire 90’. Any time that cable runs under a roadway that ductwork needs to be encased in concrete. In urban and semi urban areas you also compete with other buried infrastructure for space - sewer, city/municipal infrastructure, gas, electrical transmission, etc. While underground distribution systems are less prone to interruption from bad weather it depends on the circuit design. If the underground portion of the circuit is fed from overhead power lines coming from the distribution substation you will still experience interruptions from faults on the overhead. These faults can also occur on overhead transmission circuits (the lines feeding the distribution substations and/or very large industrial customers). Underground distribution comes at a cost premium compared to overhead distribution. It’s akin to the cost of building a picket fence vs installing a geothermal heating system for your home. This is why new sub divisions will commonly have underground cable installed as the entire neighborhood is being constructed - there’s no need to retrofit underground cable into an existing area and so the costs are lower and borne upfront. It’s more cost effective for them to turn the power off as a storm rolls through, patrol, make repairs and reenergize then to move everything underground. Lost revenue during that period is a small fraction of the cost of taking an existing grid and rebuilding it underground. This is especially true for transmission circuits that are strung between steel towers over enormous distances.
- deleted 9mo ago[deleted]
- stackghost 9mo agoDirect link, rather than a repost: https://groups.google.com/a/list.nist.gov/g/internet-time-service/c/o0dDDcr1a8I https://groups.google.com/a/list.nist.gov/g/internet-time-se...
- digdigdag 9mo agoMy local GPS + PPS based NTP server finally pays off.
- tguvot 9mo agoWe had some fun requesting key for accessing nist time servers. the process is (quoted from website) NIST currently offers this service free of charge. We require written requests to arrive by U.S. mail or fax containing: Your organization’s name, physical address, fax number (if desired as a reply method). One or more point-of-contact personnel or system operators authorized to receive key data and other correspondence: names, phone numbers, email addresses. Up to four static IPv4 network addresses under the user’s control which will be allowed to use the unique key. By special arrangement, additional addresses or address ranges may be requested. Desired hash function (“key type”). NIST currently supports MD5, SHA1, SHA256, and HMAC-SHA256. Please list any limitations your client software places on key values, if known: maximum length, characters used, or whether hexadecimal key representations are required. If you prefer, please share details about your client software or NTP appliance so we can anticipate key format issues. Desired method for NIST’s reply: U.S. mail, fax, or a secure download service operated by Department of Commerce. NIST will not use email for sending key data. ps. there actually seems to be improvement over what they had year ago. they added "secure download service". and previously they had message that nobody assigned to actively monitor mailbox so if you didn't get key, please email us so we will check it
- JackAcid 9mo agoGenerator maintenance is a thing.
- user3939382 9mo agoIt’s fine. The public pays for sci-fi clocks used by NIST and the Navy and we get shit latency over NTP and a WWVB signal that barely reaches a huge chunk of the country. CLOCKS WE PAID FOR. Jane Street gets lightning access to clocks and our pension managers get their NTP trades front run. NTP is a disgrace and an insult when it is working.
- royce 9mo agoThe rest of the snark/ bitterness aside ... NTP takes latency as a constraint, and accommodates for it. And far from a disgrace, NTP was a brilliant design for its time, and has proven to be far more future-proof than should have been expected.
- user3939382 9mo ago“snark bitterness side” I see so I’m not entitled to my feelings and I’m holding it wrong. No. It’s a disgrace. The latency is shit the precision is shit and WWVB to go with it is an insult. Future proof? Are you living in the same future I am where our trades are front run? I notice you didn’t engage with the substance of my attack at all.
- HocusLocus 9mo agoOld sci-fi story of mine, maybe the only one ever inspired by NTP THE TIME RIFT OF 2100: How We lost the Future --- and Gained the Past. https://tech.slashdot.org/comments.pl?sid=7132077&cid=49308245 https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...
- bear8642 9mo agoCool story - thanks for sharing it!
- deleted 9mo ago[deleted]
- wolfbyte8 9mo agoI just got home from a holiday party where two people from NIST in Boulder work on the NIST-F1 Atomic Clock. The one scientist described the outage and the failures but reported that things are still shut down even after power was restored. She told me that she believes that the Trump administration is in the process of shutting down the program and server in Boulder. And there was low confidence in that the Washington DC superiors understood what they are doing. God I hope this isn't more Tina Peters retribution crap. Cutting off our nose to spite our face.