6 ms·
Out of control Java processes when dealing with a leap second?
- Maxious 14y agoReddit is also down due to Java https://twitter.com/redditstatus/status/219244389044731904 https://twitter.com/redditstatus/status/219244389044731904
- cmorrisrsg 14y agoWe saw this on every Ubuntu Lucid machine we run. Restarting the JVM was insufficient - we ended up fully rebooting the machines.
- silviosantoz 14y agoLeap second + Java + Linux fix (without reboot): /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; -- then restart java
- timdoug 14y agoWorks wonders for me. Thanks!
- orphan_annie 14y agoyou saved the orphanage!
- zship 14y agoThank you! Was banging my head against the wall on RHEL6's tomcat. Went on Hacker News to stop thinking about it for a few minutes. Son of a bitch, the solution was right on the front page.
- smokestack 14y agoThis is what worked for us. Credit to https://bugzilla.mozilla.org/show_bug.cgi?id=769972#c5 https://bugzilla.mozilla.org/show_bug.cgi?id=769972#c5
- stevencorona 14y agofixed it for Twitpic too
- _thekev 14y agoA lot simpler: date --set "`date`"
- larelli 14y agoOn a localized box, I had to run "unset LANG" first, because otherwise date doesn't seem to understand its own output.
- juiceandjuice 14y agoDoesn't work for everyone (Didn't work with our java processes on RHEL 6.2)
- henrikschroder 14y agoWhere were you 11 hours ago when we rebooted all of our cassandra and mysql machines? :-)
- aSig 14y agoI saw the same across all of our Ubuntu servers. Some running Tomcat others running SmartFox Server (a java based game server). A reboot + app restart fixed it.
- rbanffy 14y agoI experienced no problem today. Is this Java-specific?
- NDizzle 14y agoYeah this is what is slaying linkedin at the moment.
- cedrichurst 14y agoMySQL is also apparently affected, according to the #mysql IRC channel.
- codeka 14y agoYes, this happened on my development MySQL instance and also one of my production instances of MySQL -- but not another one that I have. I'm not sure why it only affected one, maybe different versions of MySQL?
- thristian 14y agoNo, it's a problem with futexes. If your production code doesn't use many (any) threads, you should be fine. People are reporting the same issue with MySQLd, I had it with Firefox.
- spullara 14y agoHas anyone seen this not on Ubuntu?
- cmorrisrsg 14y agoWe saw it pop up on Centos6 as well. JRE 1.6 and 1.7.
- jerdfelt 14y agoI saw this on my Fedora 17 laptop. Chrome started eating up 100% CPU and my laptop started getting really hot. I tried restarting Chrome, updating Chrome, etc. Nothing worked. Rebooting fixed the problem. Surprisingly, none of my servers have had problems, but I don't run Java anywhere.
- smokestack 14y agoFC15
- andrewcooke 14y agoopensuse (mysql supporting the kde desktop search).
- stevencorona 14y agoEven if you're not running java in your stack, it's worth a quick check on your servers. Our SoftLayer servers use an Adaptec RAID card and the monitoring software uses the JVM and will suck up all of your CPU. All of our MySQL/Cassandra/Redis/Kestrel servers needed to be bounced because they had RAID cards. Frustrating.
- Maxious 14y agoEven if you don't have java installed but you're running on shared infrastructure that might be running MySQL/Java elsewhere it's a crazy day. My inbox is full of performance-below-acceptable-levels/high-load alerts.
- Negitivefrags 14y agoI also use Softlayer and have a pair of servers with Adaptec RAID cards. Checking up on them they were not affected. Having looked in to it, it seems our server admin decided to install OpenNTP to replace the standard NTP daemon. I'm not sure why he did that, but it seems OpenNTP just ignores leap seconds and compensates after the fact just as if your clock had drifted a second out of sync in the usual way.
- _joe 14y agoSame problem here. I am supposedly on vacation in Barcelona and was summoned to help :/ Gotta love your java.
- Sharlin 14y agoExcept that it's not a Java problem, it's a Linux futex problem.
- DEinspanjer 14y agoFun that a bug for my team happens to be the thing linked in this article. :)
- matthewlmcclure 14y agoHey Daniel, Funny to see your name there!
- matthewlmcclure 14y agoWhat's the explanation of that wacky date format? minute day hour month year . second
- emmelaich 14y agoIt's actually month day hour minute.second Originally (and perhaps still) the month day is optional; allowing you to set the date with just hour and minute.
- _thekev 14y agoI was able to restore my java app boxen (and all other boxen) by only setting the date to the current date. No restarts necessary.
- matthewlmcclure 14y agoWould love to read a root cause analysis of the relevant source code. Anyone have a link?
- ryanpers 14y agoits the futex() call that is bugged, so therefore things that are highly multi-threaded... for example mysql and java! Many unix things are not as multithreaded as those examples.
- rasur 14y agoI had a lovely morning (2am-4.30) rebooting about 50 boxes by hand. All wearing some shade of java.
- gouranga 14y agoAt the risk of shooting myself, all our jvms were fine on windows and Ubuntu.
- burningout 14y agoI was just wondering why everything stopped working at night. System.exit() didn't work anymore in any of the java processes, or just random hangs. wow.
- RahulAgrawal 14y agoThank you guys.. since yestartday night i was working on it.. searched lots of places.. some of servers were going down and some were working great.. never thought this would be the reason.
- nickzoic 14y agoI didn't really think about it at the time, but right about then (I'm in GMT+10) a Minecraft server (OpenJDK 64-bit, Ubuntu 11.10) I had running got rather upset ... 2012-07-01 09:59:59 [WARNING] Time ran backwards! Did the system time change? 2012-07-01 10:01:14 [WARNING] Can't keep up! Did the system time change, or is the server overloaded? ... and stuck on 100% CPU until I noticed it and restarted it this morning. Thankfully no other issues that I've noticed.