15 ms·
Heroku is down again
- deleted 14y ago[deleted]
- dnsco 14y agoApparently it's AWS East.
- KenCochrane 14y agoDon't think it was just heroku, lots of other sites were down as well, netflix.com, etc. most likely another AWS issue.
- devth 14y agoMy app with 2 dynos is down. Their status site is running fine altho it's not reporting errors: https://status.heroku.com/ https://status.heroku.com/ Their Helpdesk is down: https://api.heroku.com/helpdesk/login?timestamp=1341025835&return_to=https%3A%2F%2Fsupport.heroku.com%2F&locale_id=1 https://api.heroku.com/helpdesk/login?timestamp=1341025835&#... Devcenter is down: http://devcenter.heroku.com http://devcenter.heroku.com AWS isn't reporting any errors: http://status.aws.amazon.com http://status.aws.amazon.com
- shuzchen 14y agoThat's standard. You'd expect them to run everything they have on their own service except for the status site.
- KenCochrane 14y agoAmazon posted an update: 8:21 PM PDT We are investigating connectivity issues for a number of instances in the US-EAST-1 Region.
- stbullard 14y agoAt 11:25 Eastern, https://status.heroku.com/incidents/386 https://status.heroku.com/incidents/386 was posted: "We're currently experiencing a widespread application outage. We've disabled API access while engineers work on resolving the issues."
- jc4p 14y agoIt's been fifteen minutes since our site in US-East went down and AWS Status hasn't said anything yet.
- heliostatic 14y agoAs of 10:16 CDT, I can't reach Netflix or Heroku, although AWS status (http://status.aws.amazon.com/ http://status.aws.amazon.com/) is not yet reporting any current outages.
- usaar333 14y agoUgh, the EC2 administration console is done. Being in other availability zones won't save you..
- deleted 14y ago[deleted]
- sehugg 14y agoThere are US-East problems, looks like: - One AZ is down - API commands are spotty and may return incorrect results - ELB looks screwed - IP reassignments don't seem to be working - Who knows what the fuck else is broken
- dragonstyle 14y agoWe're in AWS East and definitely fighting some issues here, though we're trying to understand what is happening.
- mthreat 14y agoAWS should call them Unavailability Zones.
- alanh 14y agoI had the same thought, but it’s not HN material.
- robbiet480 14y agoMASSIVE Storms in VA area where us-east-1 is. 326,000 customers without power already, worst lightning I have seen in my 20 years of life. Sky is intense blue/green/purple. This is most likely what the issue is
- prezjordan 14y agoNo matter how powerful we become as a species with our technology, we are still at the mercy of the clouds. Pretty cool if you think about it.
- oldstrangers 14y agoOr if we just built our power grid underground like rational people.
- __mark__ 14y agoUntil earthquakes.
- mediocregopher 14y agoDepending on where you are in the world, earthquakes are much rarer then insane storms. I'm speaking as a Floridian. I'm fairly ignorant on this issue, but would it be that difficult to use one or the other depending on which natural occurrence is more likely? Or is this also a cost issue?
- gregschlom 14y agoMassive cost issue, plus some technical issues. According to this document [1]: "The North Carolina Utilities Commission studied the cost of placing Duke Power’s distribution facilities underground and found it would cost more than $41 billion, resulting in a 125 percent increase in customer rates." [1] http://www.sceg.com/NR/rdonlyres/465E6534-2FFB-4069-BF84-81465AEEF887/0/%20Undergroundvs.pdf http://www.sceg.com/NR/rdonlyres/465E6534-2FFB-4069-BF84-814...,
- ultrasaurus 14y agoYup, a lot of services served by AWS are having issues. We're seeing a huge spike in incidents being triggered in PagerDuty. (fyi: our customers are still being alerted)
- scorpion032 14y agoWhat infrastructure do you use for alerting your customers?
- drewvanstone 14y agoWe have dozens of servers that are unavailable at the moment (US-East). Obviously AWS is having major issues.
- ricardobeat 14y agoPlease link to https://status.heroku.com/ https://status.heroku.com/, pointing to a broken URL is pointless.
- sneaky_weasel 14y agoEverything still down for me. Would have expected some redundancy...
- MicahWedemeyer 14y agoCan we update the title to something like "AWS US-east-1 is down" instead of just Heroku?
- sofuture 14y ago> informative titles get changed > uninformative titles left intact hn 2012
- jdcryans 14y agoThen you'd have to change the link too?
- rplnt 14y agoAppropriate title might be that "Heroku is down due to AWS outage which is down due to power failure which happened due to storms caused by moist winds colliding with hot air that was heated over the continent by sun that....". It really doesn't matter. Heroku is down. Customers don't care.
- Apreche 14y agoWhy does the AWS dashboard show all green when that is most definitely not the case? http://status.aws.amazon.com/ http://status.aws.amazon.com/
- prezjordan 14y agoThe red check marks are in VA and can't be displayed.
- jc4p 14y agoIf only they did multi-AZ hosting like they keep telling us to do when there's outages :)
- ultrasaurus 14y agoI strongly doubt that's the case, at PagerDuty we're seeing ~100x the regular traffic. I think it's been having issues for at least 15 minutes.
- Apreche 14y agoEven now that it is updated it has yellow triangles for "performance issues" instead of red circle for service disruption. Seems like they are in denial.
- oakwhiz 14y agoPower went out = service disruption. It's underhanded to call it a "performance issue," if not an outright lie, albeit a small one.
- erichmond 14y agoThis was the disappointing thing for me as well. Our connectivity died around 8PM EST-ish, and I immediately went to status.aws and it said everything was normal. I then proceeded to waste half my night looking at our internal infrastructure trusting that page was accurate. I've learned my lesson.
- fosk 14y agoIt's not Heroku that is down, AWS is down.
- coryshaw 14y agoAhh I was just using crunchbase and its now down, must be related.
- momoro 14y agoAws East connectivity issues, 8:21pm: http://status.aws.amazon.com/rss/ec2-us-east-1.rss http://status.aws.amazon.com/rss/ec2-us-east-1.rss
- RegEx 14y agoLoadbalancers are down for me: Getting 'Response contains invalid JSON' upon attempted termination.
- darrenkopp 14y agoThink that netflix is down too. so much for chaos monkey?
- theoutlander 14y agoNo wonder my wife was just asking me why Netflix is down on Xbox live here in Seattle.
- blantonl 14y agoYup, I can confirm from here in Montana that both Netflix and Herkou are down. However, I have 20 instances on us-east. And haven't seen any problems, even during yesterday's outage on AWS. Edit: that doesn't mean this isn't an AWS outage.... It almost certainly is.
- sofuture 14y agoYeah, my ~20 instances are okay. I think we had a hiccup with our (multi-az, whooo!) RDS though.
- zenogais 14y agoConfirmed in California as well.
- undergroundhero 14y agoConfirmed in Missouri.
- deleted 14y ago[deleted]
- jaequery 14y agoi think netflix ceo just signed up for a rackspace account
- justinsb 14y agoYou're joking, but I bet Netflix's Asgard system gets support for OpenStack pretty quickly now...
- pud 14y agoThis is the motivation I needed to spread my EC2 instances across multiple availability zones. When the power comes back. (fandalism is down)
- sehugg 14y agoDon't assume that'll save you.
- deleted 14y ago[deleted]
- edouard1234567 14y agoidea for heroku : allow customers to host a "my app is down page for blah blah reason" where they host their status page (rackspace I guess?). Who think this would be useful? My users see a blank page right now when they go to ZeTrip, I'd rather show them a static page saying : "our site is down due to amazon lack of redundancy."
- devicenull 14y agoCloudflare lets you do this afaik. I'm not sure I'd trust a service to show a proper 'this site is temporarily down' page when something very bad has happened.
- reustle 14y agoI can't get to netflix.com. That is no good. Luckily all of my 75+ east servers seem to be ok.
- rorrr 14y agonetflix.com is down here as well
- stbullard 14y agoEC2 status: 8:21 PM PDT We are investigating connectivity issues for a number of instances in the US-EAST-1 Region. 8:31 PM PDT We are investigating elevated errors rates for APIs in the US-EAST-1 (Northern Virginia) region, as well as connectivity issues to instances in a single availability zone. 8:40 PM PDT We can confirm that a large number of instances in a single Availability Zone have lost power due to electrical storms in the area. We are actively working to restore power.
- joelg87 14y ago8:49 PM PDT Power has been restored to the impacted Availability Zone and we are working to bring impacted instances and volumes back online.
- stbullard 14y ago9:20 PM PDT We are continuing to work to bring the instances and volumes back online. In addition, EC2 and EBS APIs are currently experiencing elevated error rates.
- edouard1234567 14y ago"heroku status" command returns : All Systems Go: No known issues at this time.
- aquark 14y agoJust started up try using filepicker.io and it seems to be down too. Beginning to feel pretty lucky though -- this is at least the 4th AWS-East outage that has made enough of a splash to notice but missed my instances. Upgrading to multiple availability zones was scheduled for Monday anyway.
- moskie 14y agoComcast's login server looks like it's down too: http://login.comcast.net http://login.comcast.net. Prevents me from logging into HBO Go. No Netflix either. :(
- apawloski 14y agoReddit also seems to be experiencing some difficulties. Are they still on AWS?
- rgarcia 14y agoHow many times does this have to happen before heroku spreads across multiple regions?
- fragsworth 14y agoYou say this like they can just snap their fingers and provide regional services.
- inopinatus 14y agoWell, nothing of scale happens overnight. But anyone deploying a critical application to AWS makes a point of cross-region data replication. Heroku have long known that they lose potential customers to, say, Engine Yard as a result of only hosting at US-East. One can only conclude that this is a clear business decision on their part. I can hardly believe that Heroku's engineers are incapable of it. Indeed I would be very surprised to learn that they haven't brought up an instance of their platform at, say, US-West, for testing or proof-of-concept purposes. Of course, productising that is a different matter. Extending the control plane, front end, and pricing/billing systems might have considerable associated project cost. Perhaps they have concluded that the costs outweigh the additional revenue. Or, just haven't got around to it yet.
- tomjen3 14y agoIf not, why pay them rather than go to amazon directly?
- cardmagic 14y agoLike http://appfog.com/ http://appfog.com/ has?
- mechanical_fish 14y agoIt's a hard thing to engineer, especially after the fact, and especially when you are trying to hide it behind an abstraction layer. (Which is to say: You can't expect your customers to engineer their apps with multiregion in mind, or to take it kindly when you raise rates to support additional redundant hardware and bandwidth.) e.g. because region-to-region data transfer is not free, and trans-region latency is ugly, you can't just relaunch half your instance farm in another region and expect happiness. There are also routing issues: Internal IPs don't work across regions, elastic IPs don't transfer across regions...
- jaequery 14y agodid the datacenter get flooded or what? this is just "major" downage.
- apawloski 14y agoIt's likely related to the power outages across VA
- deleted 14y ago[deleted]
- knodi 14y agoCome on not another power issue, what happened to the generators... and the back up generators that they fixed few weeks back.
- excuse-me 14y agoThe little red ribbon that you pull to get the AA batteries out is stcuk underneath - they are looking for a pen to flick the battery but since everyone switched over to Fire tablets there aren't any pens.
- jordanthoms 14y agoHeroku's uptime for June is going to be.. not so hot.
- deleted 14y ago[deleted]
- bohara 14y agoFor shissle. Had to move off Heroku for my latest app. That amount of downtime would put me out of business. To bad. I really like the Heroku platform
- _nato_ 14y agowhat a winner! And they charge!! FUX.
- _nato_ 14y agoI have 3 customers -- what the are they going to do, those poor souls!!
- philip1209 14y agoCloudflare Always-Up isn't showing on my page - is Cloudflare affected too?
- ahmedaly 14y agoI can't believe that guys at heroku are not ready for such situations! They rely ONLY on virginia's instances because its the cheapest, without caring about customers.. or thinking of replicating their services in multiple locations for such issues!
- technotony 14y agoI just lost a potential hire because of this, was demoing my app to someone and it wasn't working, she thought it was because of the product. Damn you heroku!
- bsaul 14y agoGoogle Appengine's just fine. Dont't know how many AZ i'm on and don't want to know :)) The more i see about amazon failures the more i think VM are just not high enough for me in the abstraction layer...
- manishm 14y agoSimple solution to this is to have a backup or failover to a non-AWS Datacenter too, basically don't be just dependent on one Datacenter. E.g. MS Azure/Google/Rackspace This not only spreads your risks but keeps your customers happy.
- kevinprince 14y agoWe use Heroku for our event tools, thankfully we have nothing live this weekend or this would be a disaster.