5 ms·
Reddit: 2012 State of the Servers
- cluda01 15y agoI'm unfamiliar with hosting costs or really any costs running a site as popular as reddit. Anyone with experience in this area have a ballpark figure for how much it would cost per month to run this sort of setup?
- rdouble 15y ago$300K
- someone13 15y agoWhere do you get this estimate from? (Not disbelieving you, just curious)
- joering1 15y agoand also -- any idea how much they can bring id ad revenues?
- plasma 15y agoUsually 1,000 ad impressions is around $1.00 (varies greatly though, can be lower like $0.15c and higher like $4.00+). Assuming $1.00 per 1,000 impressions, and taking their 2.07billion impressions/month figure: Roughly 2,070,000,000 / 1,000 = $2,070,000 in ad revenue per month? Wild guess.
- dangrossman 15y agoMany/most page views on Reddit don't have any ads. Promoted stories only appear on story lists, not individual story pages, and don't appear 100% of the time (the space is also used to promote random new submissions). The graphical slot in the sidebar is almost 100% non-paid in-house ads.
- cluda01 15y agoI've always wondered why they don't contract out with other ad networks when they cannot fill the ad content themselves. Say for example their self serve ad can't fill the page request why not put in a google text ad link on the right side where the banner is? That to me seems like a straightforward way to massively increase revenues.
- redslazer 15y agoThey dont do it because they really care about the user. Just sticking up random google ads isnt going to make anybody happier and with an internet savy crowd like reddit ad clicks are likley to be low. Sure very targeted ads like the ones that self-serve currently delivers work because its redditors advetising to redditors.
- deleted 15y ago[deleted]
- daintynews 15y agoI have the same question. I tried googling it, but no luck.
- meenriquez 15y agome too.
- deleted 15y ago[deleted]
- bru 15y agoOne year and half ago, it was calculated and then confirmed by an admin[1] that the monthly cost was around 22K/month, or 270K/year. jedberg added that they were projecting to be around 350K/year by the end of 2010. Supposing that the cost increased linearly with the number of users (which sounds like a bad hypothesis, but is a start), the cost at the end of 2011 could be around 1M/year... That's impressive, but nowhere near the 300K/month proposed by rdouble. So I would say that the monthy cost of reddit's infrastructure is around 90K. Which is really impressive. 1: http://www.reddit.com/r/blog/comments/ctz7c/your_gold_dollars_at_work/c0v8yby&context=2 http://www.reddit.com/r/blog/comments/ctz7c/your_gold_dollar...
- dhbanes 15y agoThanks for the clarification, I thought $300k sounded a little off since cluda01 asked about estimated monthly (not annual) cost.
- rdouble 15y agoYou're probably right as I calculated with expensive instances. Also, when I made my estimate I was guessing at image storage costs, forgetting that the images are coming from image sharing sites.
- ketralnis 15y agoA year and a half is a long time in reddit time
- deleted 15y ago[deleted]
- cmer 15y agoThere's no way in hell it costs $300k per month to run Reddit!
- sp332 15y agoTrue, that's probably per-year instead.
- davej 15y agoMy back of the envelope estimate. These are based on the figures from last year and the fact that they currently have 240 EC2 instances, some are large (guessed 70), more are x-large (guessed 170). 8760 is the number of hours in a year. (8760 * $0.24 * 170) + (8760 * $0.12 * 70) = $430,992/yr in hourly fees ($1,820 * 170) + ($910 * 70) = $373,100/yr in reservation fees 373,100 + 430,992 = 804,092 / 12 months = $67,007.67/mo Reference for last years calculations: http://www.reddit.com/r/blog/comments/ctz7c/your_gold_dollars_at_work/c0v8yby&context=2 http://www.reddit.com/r/blog/comments/ctz7c/your_gold_dollar...
- cheald 15y agoNearly $1million/year in infrastructure costs so that I can laugh at GIFs of cats. The internet is truly a wonderous thing.
- gameshot911 15y agoHaving no experience with database/website administration myself, I'm struck by just how little I'm able to translate the works and concepts in this post into actual, manual labor. For each and every thing that Jason talked about...upgrading Cassandra, moving off EBS, embarking on self-heal and auto-scale projects...what took the reader a few seconds to read and cognise undoubtedly represented hours and hours of work on the part of the Reddit admins. I guess it's just the nature of the human mind. I don't think I could ever fully appreciate the amount of work that goes into any project unless I've been through it myself (and even then, the brain is awesome at minimizing the memory of pain). So Reddit admins, if you're reading this, while I certainly can't fully appreciate the amount of labor and life-force you've dedicated to the site, I honestly do appreciate it, and I wish you guys nothing but success in the future!
- deleted 15y ago[deleted]
- markerdmann 15y agoIt's interesting to see that they're sticking with Cassandra, and that they're having a much better experience with 0.8. I've been hearing so many fellow coders in SF hate on Cassandra that I had stopped considering it for projects. Has anybody worked with 0.8 or 1.0? Would you recommend Cassandra? I got to work with Riak a lot while I was at DotCloud, but the speed issue was pretty frustrating (it can be painfully slow).
- rbranson 15y agoThis is because people came to the table with unrealistic expectations. They were used to dealing with mature software based on decades old proven ideas and coming into very experimental territory expecting to get a smooth experience. Cassandra has enabled Reddit to manage a highly scalable distributed data store with a tiny staff. This is not to say it has been trouble free, but it has enabled them to do something that would have been infeasible without pioneers in this space (Cassandra, Riak, Voldemort, etc) making these tools available.
- onemoreact 15y agoI respect the Reddit team, but I don't think they need to use Cassandra at their scale. I mean they only have 2TB of data in total. They should easily be able to use a simple caching system to keep the last 2 weeks of data in RAM and basically never read from the database. That said, they may be freaked out based on their growth curve and simply thinking ahead.
- techscruggs 15y agoThey said that they had 2TB in postgres not 2TB of total data. I imagine all of their data is probably about an order of magnitude larger. Additionally, the challenges are not as much around how much data you have, but how you want to access that data (indexes).
- rbranson 15y agoIndeed. It boils down to their need for a durable cache. It's simply too expensive to try to cache every comment tree in RAM, and Cassandra's data model and disk storage layout is a really good fit for the structure of their data.
- thought_alarm 15y agoIt reminds me of Slashdot circa 1998/99, back when we watched those guys grow their then-new-found popularity out of a dorm-room Linux box; at a time when the web was a mere fraction of the size it is today. Godspeed, reddit. You're on the right track.
- joevandyk 15y agoThey say they moved off ebs and onto local storage for postgres and saw a big increase in reliability and performance. I did the same for my site last year and it was great. This is one of the reasons why I haven't moved my Postgres databases to enterprisedb or heroku: they use ebs.
- jarcoal 15y agoWhat a bummer. Amazon advertises EBS as being both faster and more reliable, but it sounds like they are delivering neither.
- cheald 15y agoEBS is fine until it isn't. The problem isn't general EBS suckage, it's unpredictable and sporadic suckage. When your DB server is blocking while it tries to write to a disk that isn't responding, things get really hairy really quickly.
- iand 15y agoBy reliable Amazon mean "won't lose your data" and they deliver on that. The issue in the articlew is around latency and Amazon aren't making any claims in that area. High throughput databases need steady latency guarantees so they're not a great fit for EBS. EBS is great for many other scenarios though.
- tibbon 15y agoBy local storage, do they mean they are running their DB disks on physical hardware, but the database servers still on EC2?
- rogerbinns 15y agoThe Amazon servers have local disks physically attached. They are wiped between customers, on machine failure etc hence "ephemeral". The EBS (elastic block storage) is accessed as a disk but is over the other end of a network connection. Amazon does more to ensure the contents are available and durable (eg replication, backup to S3). The problem with EBS is that performance especially latency is highly variable and unpredictable.
- sadfasdfasdf 15y agolol wtf they weren't using postgres 9 with streaming replication? they are upgrading now? I was using that shit for an insanely high write/avg read stuff and the sharding was fucking awesome 3 years ago.
- fleitz 15y agoRunning a DB on a single spindle, and they have performance problems? I couldn't imagine why. 2 TB OMG, thats almost a decent sized SQL Server instance. Yeah, it should take about an hour or two to replicate. I'm assuming they have a 10Gb enet on their DB server.
- ctekin 15y agoDoes anyone know what kind of hardware those 240 servers have? I wonder how much they cost.
- davej 15y agoThey're EC2 large and x-large instance.
- Ecio78 15y agoWhat about IndexTank? They dont talk about it in this blog post. Have they stopped using it?
- eco 15y agoThey still use it. Whenever you search you get the "Powered by IndexTank" logo in the corner.
- ypcx 15y agoWondering how much of that 2TB dataset is necessary for the common daily functionality of reddit, probably less than 1%, and the rest is historical data, accessed by almost no one, except perhaps by the submission-dupe- checking algorithms, and similar?
- rplnt 15y agoAre you suggesting moving that down the ladder and not having that data everywhere? I.e. if someone want to see an old post, there would be one extra step required to load the data (so cdn, cassandra, now subset of postgres with not so old data, "full" postgres). I think facebook does something similar, but they really have to, considering their size.
- nbm 15y agoIn terms of status updates (ie, stories which may mention check-ins or photos or similar, but not Facebook Messages), before Facebook's Timeline launch, there were multiple stores of data depending on age. With Timeline, all the different versions of data over all ages were put back together into a single (logical) store. More about that process at: https://www.facebook.com/notes/facebook-engineering/building-timeline-scaling-up-to-hold-your-life-story/10150468255628920
- zerostar07 15y agoThose are staggering numbers, glad i invested my time in reddit last year. We must be cautious of overheating though, signs of a bubble or a possible subreddit crisis.
- brador 15y agoCould we get a public backup of the database already? Make it a torrent if bandwidth is an issue, but lets back that amazing resource up.
- obtu 15y agoTerabytes is starting to be expensive to mirror, in terms of bandwidth and storage.