8 ms·
Rubygems.org AWS bill for Feb 2014 [pdf]
- colinbartlett 13y agoI had no idea it cost this much to host rubygems.org. The website says that hosting is provided by BlueBox?
- howardr 13y agoThey might provide by paying for it rather than hosting themselves
- dschwartz88 13y agoThis seems to be mostly their CDN bill. Not sure, but I don't really consider a CDN as part of hosting fees, more of a general infrastructure fee.
- davidradcliffe 13y agoThis includes all our compute fees too.
- qrush 13y agoBluebox hosts one of the gem mirrors...AWS is the primary source though.
- reustle 13y agoThat's not as bad as I was expecting. I was once working with a startups infrastructure (>100 servers) and it was near 20k/mo (mostly reserved instances)
- jayvanguard 13y agoYes, this seems quite reasonable considering the scale it handles.
- patio11 13y agoWhile one could probably knock a couple thousand bucks off that if one cared to (which is probably penny wise and pound foolish but invariably comes up in HN discussions of hosting costs), the amazing thing is that hundreds of thousands of people worldwide are able to use core infrastructure which costs less than the fully-loaded cost of a single billing clerk in your local municipal water department.
- duaneb 13y ago> which costs less than the fully-loaded cost of a single billing clerk in your local municipal water department. To be fair, a lot of maintenance value goes into the software that is never quantified. Broken software breaks hard, not partially, so maintenance is even more crucial.
- fivethree 13y agoWhen a levy breaks people die. Software maintenance and damage is nothing compared to real engineering.
- _petronius 13y agoThat may be true of web development, but certainly isn't of software as a whole: http://en.m.wikipedia.org/wiki/Therac-25 http://en.m.wikipedia.org/wiki/Therac-25
- jonaldomo 13y agoI disagree with the web development comment. What if there was a web interface on top of Therac-25 that had an error in it?
- doyoulikeworms 13y agoWhat about the defense industry? People die if you screw up. I mean, people die if you don't, too, but you know what I mean.
- 13y ago
- incision 13y agoAt a glance, this looks like AWS being used like a dedicated host, which as demonstrated, isn't exactly cheap. There's no spot or even reserved pricing, just a bunch of on-demand instances that were up 24/7 for all 28 days in February. Seems like a genuine dedicated host, reserved instances or an architecture that leverages the elastic in elastic compute cloud would be worth considering.
- saurik 13y agoA lot of the price is bandwidth. They are effectively being reamed by using CloudFront instead of negotiating a better rate with a "real" CDN (which will also give then much better performance, as CloudFront doesn't have many edge locations). (Although, actually, while I verified their total dollars spent is greater than what would be required to get a fundamentally better deal on bandwidth, I didn't take into consideration that once you slash their costs the amount they would be paying might no longer be ;P.)
- vertis 13y agoCould probably fix some of this just by talking to Amazon about it. It's not like this is a 'for profit' setup.
- ksec 13y agoExactly, and if you include a little Powered by AWS CloudFront i am pretty sure they could drive down the price a lot. Or, they could start talking to Fastly, I am pretty sure they can work out a much better deal while being faster.
- ilaksh 13y agoThe thing is there are many providers who can do the same and most of them will do it for less than half of this. Some less than 1/5th. I think they should move this to Digital Ocean and save $5000. The bias towards AWS for this type of application is ridiculous and a big waste of money.
- ghshephard 13y agoWhenever anybody makes this type of statement, I'm alway interested in knowing if they've ever run a site with this type of traffic, and this many customers. In particular, have you ever run a site that consistently serves over 25 Terabytes of traffic/month, or have you worked with someone who has? I guarantee you that no company I have worked for in the last 15 years, could have ever run this type of infrastructure for $7K/month. Its absolutely amazing.
- ilaksh 13y agoHave you worked for Rackspace, Linode or Digital Ocean?
- ghshephard 13y agoI've worked for one of their (direct) large competitors, but haven't worked for those three companies. I've currently got active accounts with all three of those VPS providers - I love them, and use them every day - particularly Linode, but also Slicehost/Rackspace, and DigitalOcean. I even have a bare metal server at ServerBeach - which I realize I need to shut down... At this exact instant I have six terminal windows open across DO/Linode. I host a moderately popular California Food Blog, and have about 15 years experience in various companies that have had hosting responsibilities. I'm not saying you can't do great things with the VPS providers - I'm just suggesting that the tradeoff between saving $2-$3k (at most) with Digital Ocean, would be more than made up by the technology risk, hassle of having to re-invent a lot of the services that you get automatically from AWS. That could change sometime in the (near) future - but right now, AWS is an easy (and honestly, all things considered, relatively cheap) solution for this type of application.
- ww520 13y agoInteresting. Looks like most are bandwidth cost.
- stickydink 13y agoRight now we're a top 25 grossing iPhone game developer. The last AWS bill I saw was January's, a little under $200k. I'm not on the server team, so I don't know exactly what contributes most to it. But part of me really thinks it could be reduced!
- jcampbell1 13y agoThis bill is 2/3 bandwidth, and 1/3 compute. Some games require massive amounts of compute, but the bandwidth to deliver the assets is generally paid by Apple. I can guarantee you, your company is paying a metric fuck-ton more. It is called Apple's 30% cut. Your company is paying AWS $200k to pass json messages around for analytics and social aspects of the game. You are paying Apple something like $1 million per week to distribute, market, and collect payments for the game. I am not saying your company is dumb, or Apple is evil. I am saying your experience and anecdote isn't relevant to Ruby Gems, and offering a different way to think about the games industry vs. the open source software distribution world.
- stickydink 13y agoWe aren't paying that much in cut just yet. We're a small team (6 engineers in total). You don't have to be pulling in millions per week to get high on the grossing charts. We're probably around 1/4 of what you estimated. Though you mention delivering the assets. Actually (like a lot of games) we make a big effort in getting under 50MB over-the-air limit on the App Store. The total content for retina iPhone is ~300MB, delivered in parts as you progress in the game. That's kept on S3, downloaded through CloudFront. But yes! You're right, it's mostly a hell of a lot of JSON flying around.
- nknighthb 13y agoHave somebody spend a day or two looking for low-hanging performance fruit. Start with your JSON library, there are some slow ones out there. Also see if you might be unnecessarily de/serializing data structures multiple times in a single thread or process, I've seen that kind of thing creep up over time in reasonably modular codebases.
- amalag 13y agoBTW this is paid for by : http://rubycentral.org/ http://rubycentral.org/
- senthilnayagam 13y agoassuming there is a direct correlation between requests and projects, we can do a guestimate on ruby developers and projects which are active
- jpfuentes2 13y agoThere's already 30+ comments on this thread and no one has pointed out the obvious: this is all for the peanut gallery to laugh at Npm, Inc. If the bill remained relatively consistent they could host Rubygems.org for ~28 months with 200K.
- ne0lithic 13y agoWith most of this being bandwidth costs, it seems like switching to a host like Digital Ocean would make more sense here. The bandwidth costs are a fraction of Amazon's in comparison. As for the CDN, switching to something like Cloudflare might make more sense rather than relying on Cloudfront. At the least, there's a "US and EU only" option for edge locations to use which si considerably cheaper than the default option of all edge locations.
- jtrtoo 13y agoI presume you mentioned Cloudflare because of their "unlimited bandwidth". That comes with some constraints as to the use/application: https://www.cloudflare.com/terms.html https://www.cloudflare.com/terms.html It's possible RubyGems.org would be classified under one of the "not really allowed here" terms.
- mje__ 13y agoWow, as someone who uses rubygems all day and is not in "US and EU only", I'm glad you're not involved in this project.
- ceejayoz 13y ago> With most of this being bandwidth costs, it seems like switching to a host like Digital Ocean would make more sense here. The bandwidth costs are a fraction of Amazon's in comparison. That's just replacing bandwidth costs with build-and-run-your-own-CDN costs.
- rebyn 13y agoHow can I donate to Ruby Central? Checked out their "Support" page yet it didn't help much. Any easier ways like donating via Paypal?
- evanphx 13y agoI thought I'd answer some of your questions, as the person that pays the bill. 1. This can be cheaper on AWS. We've been meaning to move to reserve instances, paying a year at a time, for a while and simply haven't done it yet. 2. Fastly has already donate CDN usage to us, but we haven't fully utilized it yet as we're (slowly) sort out some issues between primary gem serving and the bundler APIs. 3. RubyCentral pays the bill and can afford to do so via the proceeds generated from RubyConf and RailsConf. 4. The administration is an all volunteer (myself included) effort. Because of that, paying a premium to use AWS has it's advantages because it allows more volunteers have help out given the well traveled platform. In the past, RubyGems was hosted on dedicated hardware within Rackspace. While this was certainly cheaper, it created administrative issues. Granted those can be solved without using AWS, but we get back to again desiring to have as low of friction on the administration as possible. Any other questions?
- teeparham 13y agoDo you know who are the biggest consumers of bandwidth? I would guess the CI servers (Travis, Circle)
- evanphx 13y agoA very good question. I'll see about crunching some of the logs to break it down by subnet.
- teeparham 13y agoGreat. Whoever the major commercial users are have a financial incentive to keep the service performant. They should all at least be sponsors at some level if they're not already.
- evanphx 13y agoHere is a partial log, every /24 that had more than 10k hits in the last 24 hours: https://gist.github.com/evanphx/9361755 https://gist.github.com/evanphx/9361755
- wbond 13y agoPackage Control is a far cry from the scale of RubyGems. PC uses a little over 2TB a month, whereas my calculations show RubyGems using around 50TB. That said, early on I chose Linode because of their generous bandwidth that is included with the boxes. For the price of less than 1TB of AWS bandwidth, I get 8TB, plus a decent box. The bigger boxes have an even bigger proportion. I'm not posting this to give any suggestions for RubyGems - I know nothing of the complexity of that setup. Mostly just figured I'd share the research I did for finding reasonably priced bandwidth.
- soheil 13y agoThat's it?
- SeoxyS 13y agoWhile by no means insignificant, this bill is no where near what I'd imagine would warrant a HN post. I wouldn't be surprised if most startups beat this regularly. The startup whose backend I co-created racks up an AWS bill that hovers around a half million dollars a month. We make use of all of the ways to save with Amazon: pre-paid reserved instances, negotiated deals, etc. And we're not even that big; imagine what Netflix's AWS bill must cost? We've tried other providers, toyed with co-locating, but at the end of the day the flexibility and cost benefit of IaaS outweighed the lower base price of CPU cycles when you roll it yourself.
- twotwotwo 13y ago> this bill is no where near what I'd imagine would warrant a HN post. Can only guess at why folks like any post, but it's not necessarily how large the bill is. Maybe it's how low it is for a service that's widely relied on, or maybe it's the level of transparency, which turned out to include evanphx above showing up to answer questions about the project.
- matthewrudy 13y agoAbsolutely, this is a transparency thing. Compared to npm asking for $300,000 in donations to keep the thing running. I'm glad RubyGems can run for relatively so little, and be transparent in doing so.
- reillyse 13y agoThis seems reasonable to me? Why is this a newsworthy item?
- trustfundbaby 13y agoexactly. I came here expecting to see a $200k/mo bill tbh
- pataphysician 13y agoTransparency is nice and the guy paying the bill answered questions for curious minds.
- jtrtoo 13y agoSince it can take a bit of time to read through the invoice, here's a summary of the bill: CloudFront $1,071 Data Transfer $3,597 EC2 $2,184 S3 $ 228 While "bandwidth" costs equate to ~$4,668/month, only $1,071 is CDN (CloudFront), with the balance just raw Data Transfer. Since lots of folks are commenting, and not everyone realizes the difference it's also a good time to point out the CloudFront vs. Data Transfer distinction. Using Amazon's terms... Data Transfer means anything directly served/coming from EC2 or S3 (or a few other services which aren't relevant here), but NOT anything for CloudFront (which is, obviously, a separate line item, as shown above). The bulk of CDN (CloudFront) usage ($735 worth or 69%) is US. The bulk of Data raw bandwidth (Data Transfer) usage ($2,931 ~80%) is US East.
- jtrtoo 13y agoIs any of this good/bad/right/wrong? I have no idea. That depends quite a bit on what THEY are doing with it and why. For example, it can be cheaper to distribute from CloudFront versus straight from S3 for some use cases. Though, generally, you are not only looking at using CloudFront to save money over S3 ...there's typically a performance reason. And sometimes the hosting costs simply don't matter. It's easy for us engineers - siting here on HN - to sit at our keyboards and play around with hypothetical ways to save money. This isn't necessarily a bad thing, but there are numerous things in IT that it doesn't make sense to optimize. Why? Because the ROI on the engineering time, CapEx, and OpEx (and the time, energy, and focus of ANYONE involved or impacted at all) to do the optimization doesn't outweigh the opportunity cost. Sometimes there are simply better uses of our limited capital and time. Not everything needs to be optimized. And the argument gets stronger when there are other factors more difficult to factor in: adopting a platform that isn't as widely known or isn't backed by a similar level of maturity (even with it's quirks, at least they are well known), etc. The risks/concerns not only vary between organizations, but often from one period of an organization's growth to the next. The beauty is every organization gets to make their own decision ...and none of them have to give a damn if the HN community agrees or not. :-)
- hurrycane 13y agoWe run into the same cost-related problems for our CDN. What we did to solve it was to rent dedicated servers that are near AWS regions. We used Route53 latency based routing to route traffic to that dedicated servers + Nginx + LUA. We're serving 300+ TB of traffic per month and the total price is just a percent of the RubyGems AWS Bill. There is some maintenance included with this solution and the problem is finding the right dedicated server providers.
- deleted 13y ago[deleted]
- tyw 13y agocould easily knock multiple thousand bucks off of that by just reserving the ec2 servers you know you'll need, plus reserve the cloudfront bandwidth you know you'll need (for the amount of data served I believe you should be able to cut CF costs by at least half). 3 year heavy EC2 reservations pay for themselves in ~7 months, cloudfront reserved bandwidth is just a 12 month agreement so that costs nothing up front. You might want to experiment with some different instance types though, depending on your resource utilization. Personally I really like using the new c3.large instances for my web servers and anything else that needs more CPU than memory, proportionately. If the standard instances suit your needs better you still might want to move to the m3 class. Aside from those two items it looks like you are sending out a considerable amount of stuff from EC2->internet (27 TB transfer out from US-East to internet). I'd recommend looking at whether you could set up a cloudfront distribution with your EC2 servers as its origin.
- dushyant 13y agoWHY DOWNLOAD!!! WHY
- sampierson 13y agoWhy was this even posted? Looking for help reducing it? Complaining about the amount spent? Looking for a pat on the back? I saw a talk at Ruby/RailsConf about the work spent building and maintaining rubygems.org. It smelled a bit martyrish. "Look at the thankless work we perform behind the scenes". Well, if help is required building or operating rubygems.org, please just say so. As a seasoned Ruby developer I'd be more than happy to contribute development time, and as a daily user I'd be willing to commit financially in a small way towards operating costs. Not that that is required - given all the offers of free hosting this post received in response. If we don't know about a problem, we can't help. Just ask if help is what you want. It's not like the Ruby community doesn't have great communication channels.
- lassebunk 13y agoCould it be possible to cache the version list locally and then just update it incrementally, e.g. via Git? Wouldn't this save both download time (for us), and bandwidth (for RubyGems)?