9 ms·

Heroku - Bamboo Routing Performance

343 points by nigma 14y ago

RaphiePS 14y ago
Interesting -- he seems to be saying that they'll explain all about the problem, but not do anything about it.
- zevyoura 14y ago
  I don't think you can really conclude that from this statement, this section could be read to mean they plan on making changes: "Working closely with our customers to develop long-term solutions" At this point I imagine they're in damage control mode; I'm not surprised they'd be reluctant to make new promises before everyone's on the same page about the technical issues.
- spankalee 14y ago
  I'm sure he doesn't want to commit to anything too soon. This was one of the bullet points: "Working closely with our customers to develop long-term solutions"
- gojomo 14y ago
  That's not fair. What do you think "tools to understand and improve the performance of your apps" and "develop long-term solutions" from his bullets mean? But, I'm surprised they didn't wait until the "in-depth technical review" was available to apologize. And the idea that they were informed of a problem "yesterday" doesn't quite match the impression RapGenius gave, that they'd been discussing this with Heroku support for a while.
  - callum85 14y ago
    I think RaphiePS's comment is fair. "tools to understand and improve the performance of your apps" only commits them to updating their docs and tools to reflect how their system really works. It doesn't indicate any intention to fix the actual problem (the fact that requests can be routed to busy dynos), nor that they will make any kind of reimbursement to people who made business decisions based on incorrect docs. "develop long-term solutions" doesn't really mean anything.
    - RaphiePS 14y ago
      Yeah, that phrase is a bit suspect. It could easily mean "teaching people how to effectively deal with our dumb routing."
- varikin 14y ago
  The first couple bullet points are about increasing transparency into how it works and what it is doing in an app. The last point is about fixing it. | Working closely with our customers to develop long-term solutions I cannot imagine this will be simple to fix. I don't know what Heroku's deployment is like^, but I figure it is very massive and complex and solving performance issues at that scale are not done overnight. ^That is the deployment and configuration of the Heroku platform, not how we as devs deploy to Heroku.
- dragonwriter 14y ago
  Actually he is, quite explicitly, saying that they will do a short-term description of the current behavior and its impact, provide tools to assist customers in understanding and managing the effect on their specific apps, and work on long-term fixes.
encoderer 14y ago
(Wonders what this response would look like if Elon Musk was running Heroku.)
- pkulak 14y ago
  RapGenius wasn't deliberately making shit up to get page views.
damian2000 14y ago
So the issue only affects Bamboo? that's what it seems to be saying
- thenduks 14y ago
  It does seem to be what they are saying but, unfortunately, no :( Random request routing is also present on Cedar [1]. The difference is that, on Cedar, you can easily run multi-threaded or even multi-process apps (the latter being harder due to a 512mb memory limit) which can mitigate the problem, but does not solve it. Modifying your app so all of your requests are handled extremely quickly also mitigates the problem, but does not solve it. Seems to me the obvious solution is to do these things (multi-threaded app server, serve only/mostly short requests) and use at least a somewhat intelligent routing algorithm (perhaps 'least connections' would make sense). [1] - https://devcenter.heroku.com/articles/http-routing#request-distribution https://devcenter.heroku.com/articles/http-routing#request-d...
salman89 14y ago
Most people are going to come here and mention how they are not planning on fixing the problem. Put it into context. Heroku made this change 3 years ago, and also has had no issues admitting the change to users. Their documentation has lagged far behind and I believe they will be more transparent in the future. This is an engineering decision they made a long time ago that happened to get a lot of PR in the past 24 hours. Until there is a business reason (losing customers), I don't see them "fixing" the problem.
- twog 14y ago
  I think this PR has already hurt Heroku & caused them to lose customers.
  - res0nat0r 14y ago
    How many customers are running on a stack that old that are at the scale being complained about? It is probably cheaper to tell them to upgrade if they want better performance, or if they dont want to/ can't then they unfortunately can't be officially supported.
    - badgar 14y ago
      Cedar doesn't fix this for Rails. It helps marginally.
  - WestCoastJustin 14y ago
    The only thing worse than being talked about is not being talked about - OW You are almost investing in heroku by using their stack and tool chain, it isn't easy for well established customers to just up and move. This is probably a PR win for them, rather than a loss. Truth be told, it will be how they handle this in the coming months that will make them win/lose customers.
    - camus 14y ago
      If you are locked in the trunk with any cloud solution , then you are a bad programmer/syst admin/whatever taking bad decisions, period. You should be able to move your project infrastructure quickly from a service to another ,if you cant to that, well too bad when your infrastructure fails...
seivan 14y ago
Wait, so those guys were on Bamboo, and complaining? Fuck, that is so not cool. We've been on cedar ever since it launched, and been running puma threads or unicorn workers. The idea of one dyno per request is bullshit, and I wasn't sure if they were on cedar or not. A dyno is an allocated resource (512mb, not counting db, k/v store etc) How ballsy of them to complain when they are doing it wrong.
- achalkley 14y ago
  Bamboo is no longer running now.
  - benatkin 14y ago
    I think you're mistaken. New Bamboo apps can't be created, but apps that were created as Bamboo or migrated from Aspen to Bamboo before Bamboo was turned off are still running.
    - achalkley 14y ago
      Right! The Aspen stack was shut down. My mistake! I thought they were only offering Cedar.
- richcollins 14y ago
  Does Cedar solve the problem? Random routing can still end up overloading a dyno. What's the solution?
  - badgar 14y ago
    No, you are correct, and move to AWS.
- deleted 14y ago
  [deleted]
- niggler 14y ago
  "Wait, so those guys were on Bamboo, and complaining? Fuck, that is so not cool." That's a non-sequitur given that Heroku still supports it and didn't advocate migrating in their response. It's fair to complain about a bug in Lion or in Windows 7 or in other products that are stil being supported. It's unfair to rail about a windows 95 bug, for example, but not something which is still being supported. And I'm surprised heroku didnt suggest Cedar in the response itself.
antoko 14y ago
That's actually a pretty impressive response as far as it goes. Obviously there's no details at this point, but he absolutely takes responsibility, doesn't try to deflect or sugar coat it, and manages to find a tone that is both professional/serious, yet also down-to-earth and earnest. I guess the real impact will be how they go about "making it right" but in terms of a first response to the situation the tone is near perfect.
- phillmv 14y ago
  Yeah, given the crowd it operates in this could be a PR nightmare. Someone seems to know what they're doing.
- fauigerzigerk 14y ago
  I would reserve judgement on this response until we learn the truth. Whether or not a response is adequate depends on what they could have said and should have said, which obviously depends on the facts. So let's just wait for more information until we assign praise or blame.
  - antoko 14y ago
    My comment is 3 sentence long and has 3 caveats. I am definitely reserving judgement, my praise was mostly about the tone, which I maintain is near perfect. I'm really curious if it was a case of him just writing it because a response hadn't been forthcoming and there needed to be one or if there was high-level meetings dissecting every word and phrase to make it just so - or where on that continuum it fell.
- kami8845 14y ago
  No this is not impressive. This is them fucking up and misleading customers for 3 years, enjoying a great reputation and now FINALLY getting called out for their BS. They're about to lose that great reputation that they've spent the past years building up, so of course they're in major crisis mode and doing everything they can to fix this.
  - 14y ago
timothya 14y ago
Working closely with our customers to develop long-term solutions Of the five action items they listed, it seems that only the last of them is about actually solving the problem. I hope they are committed to it - better visibility of the problem can help, but I'd rather not have the problem in the first place.
- raylu 14y ago
  Actually, that's the only one that smells like BS to me. The others have clear meaning and goals.
  - dragonwriter 14y ago
    The other ones are things that are obvious and immediate responses to the problem on description that don't take any deep analysis of alternatives. Long-term fixes actually do require deep analysis of alternatives (and even what the appropriate parameters are for a solution that will deal with customers problems while maintaining Heroku's scalability), and aren't something you can do much more than make vague references to off the cuff. The key question on that point will be follow-through.
spankalee 14y ago
It's so refreshing to see this kind of communication. I don't use Heroku, and don't know much about this specific issue, but they're responses to downtime and complaints have been so direct and BS-free that I'll definitely consider them when I need a PaaS.
ivzar 14y ago
I feel like there is an answer for this, but why are two companies in the "YC family" at odds so publicly? If RapGenius is "starting beef" like is done in the music industry, I find it odd that it would happen with someone on their own "label". Perhaps this is ignorance on my behalf of how companies who have already been sold (Heroku) fit into the picture, but some explanation would be appreciated.
- damian2000 14y ago
  Heroku has been owned by Salesforce.com since dec. 2010
ibdknox 14y ago
It's a good response in that they are taking responsibility, but it is pretty obvious that they are reluctant to say anything about a fix. In my mind, "it's hard" isn't a valid excuse in this case, especially when there are relatively straightforward solutions that will solve this at a practical level. For example, you could imagine a naive form of intelligent routing that would work simply by keeping a counter per dyno: - request comes in and gets routed to the dyno with the lowest count. Inc the count. - response goes out. Dec the counter. Since they control the flow both in and out, this requires at most a sorted collection of counters and would solve the problem at a "practical" level. Is it possible to still end up with one request that backs up another one or two? Sure. Is it likely? No. While this isn't as ideal as true intelligent routing, I think it's likely the best solution in a scenario where they have incomplete information about what a random process on a dyno can reliably handle (which is the case on the cedar stack). Alternatively, they could just add some configuration that allows you to set the request density and then you could bring intelligent routing back. The couple of milliseconds that lookup/comparison would take is far better than the scenario they're in now. EDIT: I realized my comment could be read as though I'm suggesting this naive solution is "easy". At scale it certainly isn't, but I do believe it's possible and as this is their business, that's not a valid reason to do what they are.
- benatkin 14y ago
  The naive approach would fail badly if they didn't have some way of supporting a healthcheck. A customizable url path (/healthcheck is often used) where the app would return 200 if things look good would work. Otherwise you may wind up with dyno that's quickly sending back 500's where not appropriate, and since it's handling lots of requests the router would keep giving it more.
  - ibdknox 14y ago
    I'm not sure that should be a concern at the routing layer or even necessarily a concern of heroku. It's not their job to ensure that your code isn't blowing up. That being said, health checks are nice for other reasons and could be used outside of the routing layer (which you need to sail along as quickly as possible).
kevinfat 14y ago
Can someone explain, to people who know nothing about scaling infrastructure, why routing to idle dynos is a hard problem?
- apinstein 14y ago
  Because to be intelligent, you have to have the router talk to all the dynos to calculate load. Doing that in a performant way can get tricky, especially since people can hit a button and get 100 workers. The bigger the n, the more resources are required to track everything and the more things can go wrong. It's not an intractable problem, but it's not trivial, affects only a small percentage of customers, and introduces complexity for everyone. I feel pretty confident that there is a reasonable solution, and as someone that just spent the last 3 weeks building a custom buildpack and a new heroku app for an auto-scaling worker farm, I am happy to see such a quick, hopeful response.
  - oillio 14y ago
    I don't think this only affects a small percentage of customers. Maybe only a small percentage will notice. This will affect any user with a high standard deviation in their application's response time. Lets take an extreme case as an example: An application that has an average response time of 100ms, however 1% of the responses have a 3s response time. They have relatively small load and they only have 2 web dynos running. The admin thinks: We have this occasional slow response, but it should be fine. When one dyno is chewing on the 3s task, the other dyno will pick up the slack. Wrong. With random routing, when one dyno is chewing on the slow task, 50% of the incoming requests are stacking up in that dyno's queue. The other dyno may be able to easily handle it's load, but half of your responses are still getting hit with 3s+ delays. This is an extreme example, but this is not a rare issue. As the admin, unless you know about this issue, you will be perplexed by the seemingly random slow response times your users will be reporting. You won't see the problem in your logs, or your New Relic performance reports, but your customers will notice. As others have pointed out, a major selling point of Heroku is it is supposed to "just work." These sorts of issues are supposed to be intelligently handled by their super-slick infrastructure. In my opinion, this is a serious issue. The fact that this has been biting users for 3 years now and Heroku is only willing to address the problem after they get major bad press is disheartening. I have always been impressed by Heroku, especially how they constantly step up, admit their mistakes, and appear to be as transparent as possible about how they will fix their issues. This situation is seriously disappointing. I am sure this is a very difficult problem to solve at their scale, but this is really what we as customers are paying them to solve.
twog 14y ago
Honest question, why would Rapgenuis still be on Heroku if the y needed 100 dynos? Why not go directly to AWS at that scale? The cost savings would be pretty significant. Am I missing something?
- ibdknox 14y ago
  Ops guys cost a lot more than just using Heroku, not to mention the cost of simply having the responsibility of servers (even if they are virtual). Never underestimate the value of just not having to think about something, especially when you're small group of people.
  - philwelch 14y ago
    I think the amount of time and energy they've invested in studying Heroku's routing and queueing strategy counts as having to think about something.
- cbs 14y ago
  Before you undertake the more significant time and cost of migrating platforms, you see what you can wring out of what you've got. It is possible that after improving their performance enough on Heroku (or hell, even without improving it), they can not justify the up-front money and resources to migrate platforms. Or, for all we know, they could already be in the process of migrating away from Heroku, but that doesn't happen overnight and doesn't help their performance in the meantime.
tyler_grady 14y ago
Is it me not understanding disqus, or did Heroku's moderator just deleted my comment?
- benatkin 14y ago
  I think I saw your comment and that they must have deleted it. Apparently their idea of keeping it civil means keeping out links to the blog post that it was a response to.
ibrahima 14y ago
I think this is really a fine response considering the pretty terrible way the original post was written and the community responded. The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better. Don't remember the precise math, but back when I studied random processes we studied this problem and the conclusion was that randomly routing requests is generally not that much worse than doing the intelligent thing, and doing the intelligent thing is nowhere near as trivial as Rapgenius and random HN posters would have you believe. Given generally well behaved requests he random solution should be maybe 2-3x worse but nothing near 50x worse. And besides, I really don't see why someone who needs that many dynos is still on Heroku.
- tomlemon 14y ago
  Rap Genius cofounder: > The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better It's still pretty bad. Here's a graph of the relative performances of the different routing strategies when your response times are much better (50%: 50ms, 99%: 537ms, 99.9%: 898ms) http://s3.amazonaws.com/rapgenius/1360871196_routerstyles_fast.png http://s3.amazonaws.com/rapgenius/1360871196_routerstyles_fa... See http://rapgenius.com/1504222 http://rapgenius.com/1504222 for more
  - gingerlime 14y ago
    Disclaimer: I've never used heroku :) if I understand the chart correctly, using unicorn with two workers gets you pretty close to intelligent routing with no intelligence. I imagine adding up to three or four would make things even better... I don't know about puma/thin etc where you can perhaps crank it even further without too much memory tax(?) To me this seems like the easiest approach for everybody concerned. Heroku can keep using random routing without adding complexity, and most users will not get affected if they are able to split the workload on the dyno-level. On a slight tangent: On the rails app I'm working on I'm trying to religiously offload anything that might block or take too long to a resque task. It's not always feasible, but I think it's a good common-sense approach to try to avoid bottlenecks.
auggierose 14y ago
Well, let's put it like this. Those of us who know our programming shit and aren't afraid of a little math know exactly what has being going on here and that this answer is pretty much BS (what else is he supposed to say? basically he makes minimal concessions given the facts).
mhartl 14y ago
This is a great response, and I'll look forward to the follow-ups in the days to come. Kudos to the Heroku team. Bravo.
zensavona 14y ago
Maybe I'm missing something here, this response speaks specifically about Bamboo - do all new services now not run on Cedar?
wowzer 14y ago
At this point they haven't really done anything. I'm really curious to see what they come up with.
redguava 14y ago
I don't understand why people think this is a great response. They know how their routing works, just say so. It can't be that hard to give a basic overview of it before they release a more comprehensive post. As for the comment "Improving our documentation and website to accurately reflect our product". That is a very round about way of saying "our website indicates our service does things that it does not" which is a VERY bad thing. People are paying for this service based on what Heroku claims it does. If the website has been inaccurate for years, that is false advertising and really a bigger problem than they are giving credit to. If anything, I am more disappointed now that I have read this response, it has not appeased anything.
- spiralganglion 14y ago
  Documentation discrepancies happen. I've seen them with pretty much every platform I've worked on. Just yesterday, I found a critical discrepancy between the ActionScript documentation and the actual behaviour of the ActionScript compiler, costing my team a day of work. (I tried to report the issue to Adobe, but the Adobe Bug Reporting System was down. Perhaps they need a Bug Reporting System for the Bug Reporting System.) I think it's pretty heroic (yeah, pun) for Heroku to own their mistake, make the changes they've proposed, and accept the fire we've been pouring on them. They could have easily tried to weasel their way out of this, or attack the claims (Tesla/NYT comes to mind). Instead, they've accepted their own wrongdoing, and have pledged to make it right. Who cares if the explanation comes today or tomorrow? Give them a few more hours to make sure their new round of technical claims are accurate, since such accuracy is exactly what's at issue.
  - redguava 14y ago
    In answer to "who cares if the explanation comes today or tomorrow", I care if the explanation comes today or tomorrow. I use Heroku and have hit scaling issues in the last few weeks very similar to this. More information on what is going on behind the scenes will help me immediately. As for discrepancy in documentation, this is one of the most major parts of their infrastructure and directly relates to how well applications scale. To claim they have intelligent routing and then not having so, that is completely misleading and not just a minor documentation discrepancy. This isn't a tech document that got out of date, this is straight from their main "how it works" page... http://www.heroku.com/how/scale http://www.heroku.com/how/scale. Read the bit on routing.
instakill 14y ago
Bamboo routing? Is Cedar not affected?
- psynapse 14y ago
  I think Cedar is just not affected as badly because it will route > 1 request at a time to a dyno, which helps if you're using something like Unicorn.
nikcub 14y ago
There is a perverse conflict with platform service providers - the worse your scheduler performs the more profitable your service will be. You replace intelligent request scheduling with more hardware and instances, which you charge the user for. How much investment is there in platform service providers towards developing better schedulers that would reduce the number of instances required to serve an application? That answer, in this case, is "not a lot" The incentives between provider and user are not aligned, which is why I am more inclined to buy and manage at a layer lower with virtual machines. Edit: AppEngine went through a similar issue. Here is an interesting response from an engineer on their team: https://groups.google.com/forum/#!msg/google-appengine/y-LnZ2WYJ5Q/j_w13F4oSSkJ https://groups.google.com/forum/#!msg/google-appengine/y-LnZ...
- nostrademons 14y ago
  In theory, the service provider could just switch to a better scheduler and raise their rates to compensate, since the TCO for a customer would go down correspondingly. Their costs go down, their profits go up, and customers still benefit. Prices are usually determined more by competition than by technical factors. Unfortunately, there's a fair bit of lock-in to PaaS vendors, which is the real reason I'd be skeptical about building a business on them. The nice thing about the VM layer is there's a well-documented, public API between your code and the platform, so when your service provider raises their rates, you can switch to another one or operate your own hardware.
  - jules 14y ago
    What you could do is price based on the value you provide. You could price per request that you serve for your customers. Of course that just puts the perverse incentive on your customers, since now the service provider is bearing the cost of a nonoptimized application. The client doesn't care much about the weight of a single request, since the cost to him is the same (well except for latency). So what you could do is define a 'typical request' by running a suite of open source web application on the service provider. This way you see how much hardware is required for a given throughput of 'typical requests'. Put a price tag on that. Now pricing also becomes much clearer for the customers: they know that if their application's requests are roughly of the same weight as open source web application X, Y and Z, and if they serve N requests per month, then their cost will be roughly N*(price of 1 typical request) per month. Now the service provider has the incentive to optimize its hardware and services, because then it can run more typical requests at the same cost. The client has the incentive to optimize his application, because then he will use fewer typical request units, and hence pay less. You could also determine the cost of a typical request by averaging over all of your clients' applications, instead of a suite of open source applications.
aneth4 14y ago
This is a horribly inadequate response. Prices for hardware have dropped 30% over the last 3 years and heroku is admitting their performance has degraded by many orders of magnitude. It's completely unacceptable to simply say, "yeah there's a problem, we'll give you some metrics to understand it better." Sure, it's great they responded. The response should be "you're right, we are fixing it and issue credits" for revenue gained from fraudulent claims about the performance of their product and a credibility straining bait-and-switch.
xwowsersx 14y ago
What the hell? It's good he owned up...I guess. But the response basically sounds like "yeah, we've been charging the same prices over the last few years for increasingly degraded performance and we would have continued to do so, but someone finally caught on so I guess we have to now do something about this, right?"
mcgwiz 14y ago
Credit for owning the scope of the problem (allowing serious discrepancies for 3 years), which is sure to cost them trust from the community. But the skeptic in me reminds me that it's likely there was no way out of admitting it. What disheartens me is that the documentation discrepancy caused real, extremely substantial aggregate monetary impact on customers, yet there is no mention of refunds. Perhaps that will come, but in my opinion, anything short of that is just damage control. This is a time excessively demonstrate integrity, for them to go above and beyond. It's in their interest not to just paper over the whole thing.
deleted 14y ago
[deleted]
drchiu 14y ago
What I find incredibly irritating about this blog response by Heroku is that it took a very visible post on Hackernews for them to act and reconsider their way of doing business. They saw the potential loss in customers, and then acted. What this means is that they never had in mind to provide the best support and product they could for their customers before this news broke out. Sad.
programminggeek 14y ago
Wow, I feel like Heroku is really dropping the ball here. Like, they are acting punch drunk or something. Basically all this says is "we hear you and we are sorry". They could have posted that a day ago. This still says nothing about what is wrong and what they are doing to fix it. Also, I'm not sure at what point this is, but at some point around say $3-5k a month, (100+ dynos) you really should rethink using Heroku. At that point and higher, you really ought to know about your infrastructure enough to optimize for scale. The "just add more dynos" approach is stupid because adding more web fronts is often the lazy/expensive approach. Add a few queues or some smarter caching and you'll need fewer web servers. Throw in something like Varnish where you can and you need even fewer servers. Point being, at some point scaling is no longer "free", it takes work and Heroku isn't magic.
- 7952 14y ago
  At $3-5k a month Heroku may as well start offering a consultancy service rather than hosting. Wanting unlimited scaling without needing local talent is a reasonable thing to want, but its unrealistic to expect if from one single platform.
  - ceejayoz 14y ago
    A lot of their success stories at http://success.heroku.com/ http://success.heroku.com/ are sites you'd expect to be spending the $3-5k/month. If their platform can't handle higher amounts of load, they really should indicate as such.
mattquiros 14y ago
Did they just say that they have no plans to return to intelligent routing, just making naive routing more visible to you?
vicks711 14y ago
Gaand phati to heroku bola hajmola hajmola
alberth 14y ago
It seems strange for me to read in Heroku's response how forthcoming they are to accept blame and responsibility for the "a degradation in performance over the past 3 years". Yet they state their action plan to "fix" this issue is to update their DOCUMENTATION and no mention of fixing the DEGRADATION issues itself. Just bizarre.
- dragonwriter 14y ago
  > Yet they state their action plan to "fix" this issue is to update their DOCUMENTATION and no mention of fixing the DEGRADATION issues itself. This is flat out untrue. The third bullet point in their action plan is to update their documentation, and the fifth is "Working closely with our customers to develop long-term solutions". Updating the documentation to accurately reflect what the platform does is obviously critical to allow people to make decisions and manage applications on the platform as it is, so is an important and immediate part of the action plan. Long-term fixes to the problem are also important, and are explicitly part of the action plan. Its clear that they haven't identified what those solutions are, but its not at all true that they haven't mentioned them as part of the action plan.
podperson 14y ago
Seems to me like a classy response to a real problem from Heroku. We all need to remember that there are no magic bullets. The fact that Heroku can get a startup to, say, 5M uniques per day by dragging some sliders on a web panel and running up a bill on a corporate AMEX is pretty impressive. At some point scaling a web business becomes a core competency and one needs to deal with it. I'm guessing by the time scaling an app on Heroku becomes an issue, if better understanding your scaling needs and handling them directly isn't going to save you a TON of money, your business model is probably broken.
tomlemon 14y ago
Rap Genius cofounder: Our response: http://rapgenius.com/Oren-teich-bamboo-routing-performance-lyrics http://rapgenius.com/Oren-teich-bamboo-routing-performance-l...
habosa 14y ago
So do the issues in the RapGenius post only affect those on the Bamboo stack? I'm procrastinating migrating to Cedar now but this could be a very good reason. Also, I really love seeing a company take responsibility like this. I know the situations (and the stakes) are not comparable but this is a lot better than what Musk did when Tesla got a bad review. As a company just take the blame and say you can and will fix it, that's good enough for most people.
austingunter 14y ago
I'm very curious to see what the technical review turns up tomorrow. This feels like something that would have been connected to the Salesforce acquisition 3 years ago, and then making the service less efficient in order to increase profits or revenue targets on paid accounts. Not to mention saving money on the free ones. It would be a little bit like Tesla not only selling you the Model S, but also selling you the electricity you charge the vehicle with. At some point, they make the car less efficient, forcing you to charge more often, and then claiming they didn't document this very well. Frankly, there are only so many people who will be a capable enough electrical engineer (or in Heroku's case, a sysadmin) to catch the difference and measure it. The apology should be, "we misled you, and betrayed your trust. Here's how we're planning on resolving that, and working to rebuild our relationship with our customers over the next year. [Insert specific, sweeping measures...]

343 points by nigma 14y ago

RaphiePS 14y ago

Interesting -- he seems to be saying that they'll explain all about the problem, but not do anything about it.

zevyoura 14y ago
I don't think you can really conclude that from this statement, this section could be read to mean they plan on making changes: "Working closely with our customers to develop long-term solutions" At this point I imagine they're in damage control mode; I'm not surprised they'd be reluctant to make new promises before everyone's on the same page about the technical issues.
spankalee 14y ago
I'm sure he doesn't want to commit to anything too soon. This was one of the bullet points: "Working closely with our customers to develop long-term solutions"
gojomo 14y ago
That's not fair. What do you think "tools to understand and improve the performance of your apps" and "develop long-term solutions" from his bullets mean? But, I'm surprised they didn't wait until the "in-depth technical review" was available to apologize. And the idea that they were informed of a problem "yesterday" doesn't quite match the impression RapGenius gave, that they'd been discussing this with Heroku support for a while.
- callum85 14y ago
  I think RaphiePS's comment is fair. "tools to understand and improve the performance of your apps" only commits them to updating their docs and tools to reflect how their system really works. It doesn't indicate any intention to fix the actual problem (the fact that requests can be routed to busy dynos), nor that they will make any kind of reimbursement to people who made business decisions based on incorrect docs. "develop long-term solutions" doesn't really mean anything.
  - RaphiePS 14y ago
    Yeah, that phrase is a bit suspect. It could easily mean "teaching people how to effectively deal with our dumb routing."
varikin 14y ago
The first couple bullet points are about increasing transparency into how it works and what it is doing in an app. The last point is about fixing it. | Working closely with our customers to develop long-term solutions I cannot imagine this will be simple to fix. I don't know what Heroku's deployment is like^, but I figure it is very massive and complex and solving performance issues at that scale are not done overnight. ^That is the deployment and configuration of the Heroku platform, not how we as devs deploy to Heroku.
dragonwriter 14y ago
Actually he is, quite explicitly, saying that they will do a short-term description of the current behavior and its impact, provide tools to assist customers in understanding and managing the effect on their specific apps, and work on long-term fixes.

encoderer 14y ago

(Wonders what this response would look like if Elon Musk was running Heroku.)

pkulak 14y ago
RapGenius wasn't deliberately making shit up to get page views.

damian2000 14y ago

So the issue only affects Bamboo? that's what it seems to be saying

thenduks 14y ago
It does seem to be what they are saying but, unfortunately, no :( Random request routing is also present on Cedar [1]. The difference is that, on Cedar, you can easily run multi-threaded or even multi-process apps (the latter being harder due to a 512mb memory limit) which can mitigate the problem, but does not solve it. Modifying your app so all of your requests are handled extremely quickly also mitigates the problem, but does not solve it. Seems to me the obvious solution is to do these things (multi-threaded app server, serve only/mostly short requests) and use at least a somewhat intelligent routing algorithm (perhaps 'least connections' would make sense). [1] - https://devcenter.heroku.com/articles/http-routing#request-distribution https://devcenter.heroku.com/articles/http-routing#request-d...

salman89 14y ago

Most people are going to come here and mention how they are not planning on fixing the problem. Put it into context. Heroku made this change 3 years ago, and also has had no issues admitting the change to users. Their documentation has lagged far behind and I believe they will be more transparent in the future. This is an engineering decision they made a long time ago that happened to get a lot of PR in the past 24 hours. Until there is a business reason (losing customers), I don't see them "fixing" the problem.

twog 14y ago
I think this PR has already hurt Heroku & caused them to lose customers.
- res0nat0r 14y ago
  How many customers are running on a stack that old that are at the scale being complained about? It is probably cheaper to tell them to upgrade if they want better performance, or if they dont want to/ can't then they unfortunately can't be officially supported.
  - badgar 14y ago
    Cedar doesn't fix this for Rails. It helps marginally.
- WestCoastJustin 14y ago
  The only thing worse than being talked about is not being talked about - OW You are almost investing in heroku by using their stack and tool chain, it isn't easy for well established customers to just up and move. This is probably a PR win for them, rather than a loss. Truth be told, it will be how they handle this in the coming months that will make them win/lose customers.
  - camus 14y ago
    If you are locked in the trunk with any cloud solution , then you are a bad programmer/syst admin/whatever taking bad decisions, period. You should be able to move your project infrastructure quickly from a service to another ,if you cant to that, well too bad when your infrastructure fails...

seivan 14y ago

Wait, so those guys were on Bamboo, and complaining? Fuck, that is so not cool. We've been on cedar ever since it launched, and been running puma threads or unicorn workers. The idea of one dyno per request is bullshit, and I wasn't sure if they were on cedar or not. A dyno is an allocated resource (512mb, not counting db, k/v store etc) How ballsy of them to complain when they are doing it wrong.

achalkley 14y ago
Bamboo is no longer running now.
- benatkin 14y ago
  I think you're mistaken. New Bamboo apps can't be created, but apps that were created as Bamboo or migrated from Aspen to Bamboo before Bamboo was turned off are still running.
  - achalkley 14y ago
    Right! The Aspen stack was shut down. My mistake! I thought they were only offering Cedar.
richcollins 14y ago
Does Cedar solve the problem? Random routing can still end up overloading a dyno. What's the solution?
- badgar 14y ago
  No, you are correct, and move to AWS.
deleted 14y ago
[deleted]
niggler 14y ago
"Wait, so those guys were on Bamboo, and complaining? Fuck, that is so not cool." That's a non-sequitur given that Heroku still supports it and didn't advocate migrating in their response. It's fair to complain about a bug in Lion or in Windows 7 or in other products that are stil being supported. It's unfair to rail about a windows 95 bug, for example, but not something which is still being supported. And I'm surprised heroku didnt suggest Cedar in the response itself.

antoko 14y ago

That's actually a pretty impressive response as far as it goes. Obviously there's no details at this point, but he absolutely takes responsibility, doesn't try to deflect or sugar coat it, and manages to find a tone that is both professional/serious, yet also down-to-earth and earnest. I guess the real impact will be how they go about "making it right" but in terms of a first response to the situation the tone is near perfect.

phillmv 14y ago
Yeah, given the crowd it operates in this could be a PR nightmare. Someone seems to know what they're doing.
fauigerzigerk 14y ago
I would reserve judgement on this response until we learn the truth. Whether or not a response is adequate depends on what they could have said and should have said, which obviously depends on the facts. So let's just wait for more information until we assign praise or blame.
- antoko 14y ago
  My comment is 3 sentence long and has 3 caveats. I am definitely reserving judgement, my praise was mostly about the tone, which I maintain is near perfect. I'm really curious if it was a case of him just writing it because a response hadn't been forthcoming and there needed to be one or if there was high-level meetings dissecting every word and phrase to make it just so - or where on that continuum it fell.
kami8845 14y ago
No this is not impressive. This is them fucking up and misleading customers for 3 years, enjoying a great reputation and now FINALLY getting called out for their BS. They're about to lose that great reputation that they've spent the past years building up, so of course they're in major crisis mode and doing everything they can to fix this.
- 14y ago

timothya 14y ago

Working closely with our customers to develop long-term solutions Of the five action items they listed, it seems that only the last of them is about actually solving the problem. I hope they are committed to it - better visibility of the problem can help, but I'd rather not have the problem in the first place.

raylu 14y ago
Actually, that's the only one that smells like BS to me. The others have clear meaning and goals.
- dragonwriter 14y ago
  The other ones are things that are obvious and immediate responses to the problem on description that don't take any deep analysis of alternatives. Long-term fixes actually do require deep analysis of alternatives (and even what the appropriate parameters are for a solution that will deal with customers problems while maintaining Heroku's scalability), and aren't something you can do much more than make vague references to off the cuff. The key question on that point will be follow-through.

spankalee 14y ago

It's so refreshing to see this kind of communication. I don't use Heroku, and don't know much about this specific issue, but they're responses to downtime and complaints have been so direct and BS-free that I'll definitely consider them when I need a PaaS.

ivzar 14y ago

I feel like there is an answer for this, but why are two companies in the "YC family" at odds so publicly? If RapGenius is "starting beef" like is done in the music industry, I find it odd that it would happen with someone on their own "label". Perhaps this is ignorance on my behalf of how companies who have already been sold (Heroku) fit into the picture, but some explanation would be appreciated.

damian2000 14y ago
Heroku has been owned by Salesforce.com since dec. 2010

ibdknox 14y ago

It's a good response in that they are taking responsibility, but it is pretty obvious that they are reluctant to say anything about a fix. In my mind, "it's hard" isn't a valid excuse in this case, especially when there are relatively straightforward solutions that will solve this at a practical level. For example, you could imagine a naive form of intelligent routing that would work simply by keeping a counter per dyno: - request comes in and gets routed to the dyno with the lowest count. Inc the count. - response goes out. Dec the counter. Since they control the flow both in and out, this requires at most a sorted collection of counters and would solve the problem at a "practical" level. Is it possible to still end up with one request that backs up another one or two? Sure. Is it likely? No. While this isn't as ideal as true intelligent routing, I think it's likely the best solution in a scenario where they have incomplete information about what a random process on a dyno can reliably handle (which is the case on the cedar stack). Alternatively, they could just add some configuration that allows you to set the request density and then you could bring intelligent routing back. The couple of milliseconds that lookup/comparison would take is far better than the scenario they're in now. EDIT: I realized my comment could be read as though I'm suggesting this naive solution is "easy". At scale it certainly isn't, but I do believe it's possible and as this is their business, that's not a valid reason to do what they are.

benatkin 14y ago
The naive approach would fail badly if they didn't have some way of supporting a healthcheck. A customizable url path (/healthcheck is often used) where the app would return 200 if things look good would work. Otherwise you may wind up with dyno that's quickly sending back 500's where not appropriate, and since it's handling lots of requests the router would keep giving it more.
- ibdknox 14y ago
  I'm not sure that should be a concern at the routing layer or even necessarily a concern of heroku. It's not their job to ensure that your code isn't blowing up. That being said, health checks are nice for other reasons and could be used outside of the routing layer (which you need to sail along as quickly as possible).

kevinfat 14y ago

Can someone explain, to people who know nothing about scaling infrastructure, why routing to idle dynos is a hard problem?

apinstein 14y ago
Because to be intelligent, you have to have the router talk to all the dynos to calculate load. Doing that in a performant way can get tricky, especially since people can hit a button and get 100 workers. The bigger the n, the more resources are required to track everything and the more things can go wrong. It's not an intractable problem, but it's not trivial, affects only a small percentage of customers, and introduces complexity for everyone. I feel pretty confident that there is a reasonable solution, and as someone that just spent the last 3 weeks building a custom buildpack and a new heroku app for an auto-scaling worker farm, I am happy to see such a quick, hopeful response.
- oillio 14y ago
  I don't think this only affects a small percentage of customers. Maybe only a small percentage will notice. This will affect any user with a high standard deviation in their application's response time. Lets take an extreme case as an example: An application that has an average response time of 100ms, however 1% of the responses have a 3s response time. They have relatively small load and they only have 2 web dynos running. The admin thinks: We have this occasional slow response, but it should be fine. When one dyno is chewing on the 3s task, the other dyno will pick up the slack. Wrong. With random routing, when one dyno is chewing on the slow task, 50% of the incoming requests are stacking up in that dyno's queue. The other dyno may be able to easily handle it's load, but half of your responses are still getting hit with 3s+ delays. This is an extreme example, but this is not a rare issue. As the admin, unless you know about this issue, you will be perplexed by the seemingly random slow response times your users will be reporting. You won't see the problem in your logs, or your New Relic performance reports, but your customers will notice. As others have pointed out, a major selling point of Heroku is it is supposed to "just work." These sorts of issues are supposed to be intelligently handled by their super-slick infrastructure. In my opinion, this is a serious issue. The fact that this has been biting users for 3 years now and Heroku is only willing to address the problem after they get major bad press is disheartening. I have always been impressed by Heroku, especially how they constantly step up, admit their mistakes, and appear to be as transparent as possible about how they will fix their issues. This situation is seriously disappointing. I am sure this is a very difficult problem to solve at their scale, but this is really what we as customers are paying them to solve.

twog 14y ago

Honest question, why would Rapgenuis still be on Heroku if the y needed 100 dynos? Why not go directly to AWS at that scale? The cost savings would be pretty significant. Am I missing something?

ibdknox 14y ago
Ops guys cost a lot more than just using Heroku, not to mention the cost of simply having the responsibility of servers (even if they are virtual). Never underestimate the value of just not having to think about something, especially when you're small group of people.
- philwelch 14y ago
  I think the amount of time and energy they've invested in studying Heroku's routing and queueing strategy counts as having to think about something.
cbs 14y ago
Before you undertake the more significant time and cost of migrating platforms, you see what you can wring out of what you've got. It is possible that after improving their performance enough on Heroku (or hell, even without improving it), they can not justify the up-front money and resources to migrate platforms. Or, for all we know, they could already be in the process of migrating away from Heroku, but that doesn't happen overnight and doesn't help their performance in the meantime.

tyler_grady 14y ago

Is it me not understanding disqus, or did Heroku's moderator just deleted my comment?

benatkin 14y ago
I think I saw your comment and that they must have deleted it. Apparently their idea of keeping it civil means keeping out links to the blog post that it was a response to.

ibrahima 14y ago

I think this is really a fine response considering the pretty terrible way the original post was written and the community responded. The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better. Don't remember the precise math, but back when I studied random processes we studied this problem and the conclusion was that randomly routing requests is generally not that much worse than doing the intelligent thing, and doing the intelligent thing is nowhere near as trivial as Rapgenius and random HN posters would have you believe. Given generally well behaved requests he random solution should be maybe 2-3x worse but nothing near 50x worse. And besides, I really don't see why someone who needs that many dynos is still on Heroku.

tomlemon 14y ago
Rap Genius cofounder: > The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better It's still pretty bad. Here's a graph of the relative performances of the different routing strategies when your response times are much better (50%: 50ms, 99%: 537ms, 99.9%: 898ms) http://s3.amazonaws.com/rapgenius/1360871196_routerstyles_fast.png http://s3.amazonaws.com/rapgenius/1360871196_routerstyles_fa... See http://rapgenius.com/1504222 http://rapgenius.com/1504222 for more
- gingerlime 14y ago
  Disclaimer: I've never used heroku :) if I understand the chart correctly, using unicorn with two workers gets you pretty close to intelligent routing with no intelligence. I imagine adding up to three or four would make things even better... I don't know about puma/thin etc where you can perhaps crank it even further without too much memory tax(?) To me this seems like the easiest approach for everybody concerned. Heroku can keep using random routing without adding complexity, and most users will not get affected if they are able to split the workload on the dyno-level. On a slight tangent: On the rails app I'm working on I'm trying to religiously offload anything that might block or take too long to a resque task. It's not always feasible, but I think it's a good common-sense approach to try to avoid bottlenecks.

auggierose 14y ago

Well, let's put it like this. Those of us who know our programming shit and aren't afraid of a little math know exactly what has being going on here and that this answer is pretty much BS (what else is he supposed to say? basically he makes minimal concessions given the facts).

mhartl 14y ago

This is a great response, and I'll look forward to the follow-ups in the days to come. Kudos to the Heroku team. Bravo.

zensavona 14y ago

Maybe I'm missing something here, this response speaks specifically about Bamboo - do all new services now not run on Cedar?

wowzer 14y ago

At this point they haven't really done anything. I'm really curious to see what they come up with.

redguava 14y ago

I don't understand why people think this is a great response. They know how their routing works, just say so. It can't be that hard to give a basic overview of it before they release a more comprehensive post. As for the comment "Improving our documentation and website to accurately reflect our product". That is a very round about way of saying "our website indicates our service does things that it does not" which is a VERY bad thing. People are paying for this service based on what Heroku claims it does. If the website has been inaccurate for years, that is false advertising and really a bigger problem than they are giving credit to. If anything, I am more disappointed now that I have read this response, it has not appeased anything.

spiralganglion 14y ago
Documentation discrepancies happen. I've seen them with pretty much every platform I've worked on. Just yesterday, I found a critical discrepancy between the ActionScript documentation and the actual behaviour of the ActionScript compiler, costing my team a day of work. (I tried to report the issue to Adobe, but the Adobe Bug Reporting System was down. Perhaps they need a Bug Reporting System for the Bug Reporting System.) I think it's pretty heroic (yeah, pun) for Heroku to own their mistake, make the changes they've proposed, and accept the fire we've been pouring on them. They could have easily tried to weasel their way out of this, or attack the claims (Tesla/NYT comes to mind). Instead, they've accepted their own wrongdoing, and have pledged to make it right. Who cares if the explanation comes today or tomorrow? Give them a few more hours to make sure their new round of technical claims are accurate, since such accuracy is exactly what's at issue.
- redguava 14y ago
  In answer to "who cares if the explanation comes today or tomorrow", I care if the explanation comes today or tomorrow. I use Heroku and have hit scaling issues in the last few weeks very similar to this. More information on what is going on behind the scenes will help me immediately. As for discrepancy in documentation, this is one of the most major parts of their infrastructure and directly relates to how well applications scale. To claim they have intelligent routing and then not having so, that is completely misleading and not just a minor documentation discrepancy. This isn't a tech document that got out of date, this is straight from their main "how it works" page... http://www.heroku.com/how/scale http://www.heroku.com/how/scale. Read the bit on routing.

instakill 14y ago

Bamboo routing? Is Cedar not affected?

psynapse 14y ago
I think Cedar is just not affected as badly because it will route > 1 request at a time to a dyno, which helps if you're using something like Unicorn.

nikcub 14y ago

There is a perverse conflict with platform service providers - the worse your scheduler performs the more profitable your service will be. You replace intelligent request scheduling with more hardware and instances, which you charge the user for. How much investment is there in platform service providers towards developing better schedulers that would reduce the number of instances required to serve an application? That answer, in this case, is "not a lot" The incentives between provider and user are not aligned, which is why I am more inclined to buy and manage at a layer lower with virtual machines. Edit: AppEngine went through a similar issue. Here is an interesting response from an engineer on their team: https://groups.google.com/forum/#!msg/google-appengine/y-LnZ2WYJ5Q/j_w13F4oSSkJ https://groups.google.com/forum/#!msg/google-appengine/y-LnZ...

nostrademons 14y ago
In theory, the service provider could just switch to a better scheduler and raise their rates to compensate, since the TCO for a customer would go down correspondingly. Their costs go down, their profits go up, and customers still benefit. Prices are usually determined more by competition than by technical factors. Unfortunately, there's a fair bit of lock-in to PaaS vendors, which is the real reason I'd be skeptical about building a business on them. The nice thing about the VM layer is there's a well-documented, public API between your code and the platform, so when your service provider raises their rates, you can switch to another one or operate your own hardware.
- jules 14y ago
  What you could do is price based on the value you provide. You could price per request that you serve for your customers. Of course that just puts the perverse incentive on your customers, since now the service provider is bearing the cost of a nonoptimized application. The client doesn't care much about the weight of a single request, since the cost to him is the same (well except for latency). So what you could do is define a 'typical request' by running a suite of open source web application on the service provider. This way you see how much hardware is required for a given throughput of 'typical requests'. Put a price tag on that. Now pricing also becomes much clearer for the customers: they know that if their application's requests are roughly of the same weight as open source web application X, Y and Z, and if they serve N requests per month, then their cost will be roughly N*(price of 1 typical request) per month. Now the service provider has the incentive to optimize its hardware and services, because then it can run more typical requests at the same cost. The client has the incentive to optimize his application, because then he will use fewer typical request units, and hence pay less. You could also determine the cost of a typical request by averaging over all of your clients' applications, instead of a suite of open source applications.

aneth4 14y ago

This is a horribly inadequate response. Prices for hardware have dropped 30% over the last 3 years and heroku is admitting their performance has degraded by many orders of magnitude. It's completely unacceptable to simply say, "yeah there's a problem, we'll give you some metrics to understand it better." Sure, it's great they responded. The response should be "you're right, we are fixing it and issue credits" for revenue gained from fraudulent claims about the performance of their product and a credibility straining bait-and-switch.

xwowsersx 14y ago

What the hell? It's good he owned up...I guess. But the response basically sounds like "yeah, we've been charging the same prices over the last few years for increasingly degraded performance and we would have continued to do so, but someone finally caught on so I guess we have to now do something about this, right?"

mcgwiz 14y ago

Credit for owning the scope of the problem (allowing serious discrepancies for 3 years), which is sure to cost them trust from the community. But the skeptic in me reminds me that it's likely there was no way out of admitting it. What disheartens me is that the documentation discrepancy caused real, extremely substantial aggregate monetary impact on customers, yet there is no mention of refunds. Perhaps that will come, but in my opinion, anything short of that is just damage control. This is a time excessively demonstrate integrity, for them to go above and beyond. It's in their interest not to just paper over the whole thing.

deleted 14y ago

[deleted]

drchiu 14y ago

What I find incredibly irritating about this blog response by Heroku is that it took a very visible post on Hackernews for them to act and reconsider their way of doing business. They saw the potential loss in customers, and then acted. What this means is that they never had in mind to provide the best support and product they could for their customers before this news broke out. Sad.

programminggeek 14y ago

Wow, I feel like Heroku is really dropping the ball here. Like, they are acting punch drunk or something. Basically all this says is "we hear you and we are sorry". They could have posted that a day ago. This still says nothing about what is wrong and what they are doing to fix it. Also, I'm not sure at what point this is, but at some point around say $3-5k a month, (100+ dynos) you really should rethink using Heroku. At that point and higher, you really ought to know about your infrastructure enough to optimize for scale. The "just add more dynos" approach is stupid because adding more web fronts is often the lazy/expensive approach. Add a few queues or some smarter caching and you'll need fewer web servers. Throw in something like Varnish where you can and you need even fewer servers. Point being, at some point scaling is no longer "free", it takes work and Heroku isn't magic.

7952 14y ago
At $3-5k a month Heroku may as well start offering a consultancy service rather than hosting. Wanting unlimited scaling without needing local talent is a reasonable thing to want, but its unrealistic to expect if from one single platform.
- ceejayoz 14y ago
  A lot of their success stories at http://success.heroku.com/ http://success.heroku.com/ are sites you'd expect to be spending the $3-5k/month. If their platform can't handle higher amounts of load, they really should indicate as such.

mattquiros 14y ago

Did they just say that they have no plans to return to intelligent routing, just making naive routing more visible to you?

vicks711 14y ago

Gaand phati to heroku bola hajmola hajmola

alberth 14y ago

It seems strange for me to read in Heroku's response how forthcoming they are to accept blame and responsibility for the "a degradation in performance over the past 3 years". Yet they state their action plan to "fix" this issue is to update their DOCUMENTATION and no mention of fixing the DEGRADATION issues itself. Just bizarre.

dragonwriter 14y ago
> Yet they state their action plan to "fix" this issue is to update their DOCUMENTATION and no mention of fixing the DEGRADATION issues itself. This is flat out untrue. The third bullet point in their action plan is to update their documentation, and the fifth is "Working closely with our customers to develop long-term solutions". Updating the documentation to accurately reflect what the platform does is obviously critical to allow people to make decisions and manage applications on the platform as it is, so is an important and immediate part of the action plan. Long-term fixes to the problem are also important, and are explicitly part of the action plan. Its clear that they haven't identified what those solutions are, but its not at all true that they haven't mentioned them as part of the action plan.

podperson 14y ago

Seems to me like a classy response to a real problem from Heroku. We all need to remember that there are no magic bullets. The fact that Heroku can get a startup to, say, 5M uniques per day by dragging some sliders on a web panel and running up a bill on a corporate AMEX is pretty impressive. At some point scaling a web business becomes a core competency and one needs to deal with it. I'm guessing by the time scaling an app on Heroku becomes an issue, if better understanding your scaling needs and handling them directly isn't going to save you a TON of money, your business model is probably broken.

tomlemon 14y ago

Rap Genius cofounder: Our response: http://rapgenius.com/Oren-teich-bamboo-routing-performance-lyrics http://rapgenius.com/Oren-teich-bamboo-routing-performance-l...

habosa 14y ago

So do the issues in the RapGenius post only affect those on the Bamboo stack? I'm procrastinating migrating to Cedar now but this could be a very good reason. Also, I really love seeing a company take responsibility like this. I know the situations (and the stakes) are not comparable but this is a lot better than what Musk did when Tesla got a bad review. As a company just take the blame and say you can and will fix it, that's good enough for most people.

austingunter 14y ago

I'm very curious to see what the technical review turns up tomorrow. This feels like something that would have been connected to the Salesforce acquisition 3 years ago, and then making the service less efficient in order to increase profits or revenue targets on paid accounts. Not to mention saving money on the free ones. It would be a little bit like Tesla not only selling you the Model S, but also selling you the electricity you charge the vehicle with. At some point, they make the car less efficient, forcing you to charge more often, and then claiming they didn't document this very well. Frankly, there are only so many people who will be a capable enough electrical engineer (or in Heroku's case, a sysadmin) to catch the difference and measure it. The apology should be, "we misled you, and betrayed your trust. Here's how we're planning on resolving that, and working to rebuild our relationship with our customers over the next year. [Insert specific, sweeping measures...]