9 ms·

Scaling lessons learned at Dropbox, part 1

418 points by eranki 14y ago

deleted 14y ago
[deleted]
prayag 14y ago
Fabulous post. Thanks for writing. One point it misses though is to test your backup strategy often. When you scale fast things break very often and it's good to be in practice of restoring from backups every now and then.
- mirkules 14y ago
  Just started reading a book called "High Performance MySQL" and in one of the early pages, the following advice appears: "It's an excellent idea to run a realistic load simulation on a test server and then literally pull the power plug. The firsthand experience of recovering from a crash is priceless. It saves nasty surprises later." Same goes for testing network connectivity and failover. I can't tell you how many times I've heard things like "The automatic recovery _should_ have kicked in but..." Having a recovery procedure and backup strategy is completely different from having actually restored a backup and recovered from a failure.
  - RegEx 14y ago
    Reading High Performance MySQL as well. Loving it so far!
- eranki 14y ago
  Thanks! Good point. We actually repurposed our offsite database recovery to clone slaves off a master (after LVM was no longer performing), so that's a great way to get more testing in.
matt 14y ago
Nice, love the idea of running with extra load to predict breaking points.
ivankirigin 14y ago
Rajiv is awesome, you should listen to him
- akent 14y ago
  Says an ex "Product Manager at Dropbox". Edit: Thanks for the downvotes. My point is, just make it unambiguous to everyone in your comment so we don't have to click through your profile. Context matters. e.g.: "I was Product Manager at Dropbox and worked with Rajiv (the OP). He's awesome, you should listen to him." Much better.
  - sriramk 14y ago
    Even more reason to listen to him.
  - to3m 14y ago
    I've found working with people to be a reasonable way of finding out whether their opinions are worth listening to...
  - stratos2 14y ago
    which means his opinion counts at least 100 times more than yours does.
    - carb 14y ago
      What he's saying is that ivankirigin should have said that himself. I don't know that he has any credibility to his statement and wasn't going to give it any merit until akent made me realize that ivankirigin had first-hand experience.
acslater00 14y ago
For the record, I use sqlalchemy 0.6.6 regularly under fairly heavy load, and have never had a problem with it. Any 'sqlalchemy bugs' are inevitably coding mistakes on my part.
- kennywinker 14y ago
  Yeah, I found that bit quite vague. Are they using SQLAlchemy's object layer, but just not the high level query stuff? Or are they using only the low-level query stuff and nothing else? I'd love to know more about how their system works, if they are indeed not using an ORM. Every time I've tried to build something without an ORM, I just end up writing my own shitty one accidentally.
- zzzeek 14y ago
  We got a dozen or so email list requests for support from people who I know to be from Dropbox in late 2008. At that time, we were at version 0.4.8. That is an extremely old version and the codebase was quite immature at that time - I personally didn't use SQLAlchemy in production until 0.5 (which of course is because Python was hardly used at all in the early 2000's outside of the zope community, so I was still stuck with Java/Perl gigs). However I am still quite skeptical of the claim that it returned the wrong results. You're expected to watch the SQL you're telling it to generate during development. It will always be true that pushing an ORM will not always generate the SQL you want - which is why you have to make sure those queries are how they should be, before pushing to production. The ORM will of course stick to the plan you've given it - it isn't "deciding" anything, and at worst it can only misinterpret your intent - just like any library. My strategy with SQLAlchemy has always been to under-promote it. If you have lots of big players early adopting you and hitting all the pointy edges, it can damage your rep. There's a group of major folks out there who will never use my library due to old experiences. Others like Reddit and Yelp have hung on, and apparently dropbox is still using the core, hooray ! That's why I'm always amazed at how aggressively MongoDB is promoted, when it seems like they're still going through a lot of growing pains. I guess they sort of have to, given that they're a business and all.
philfreo 14y ago
Can you explain the nginx/HAproxy config a little more?
- misiti3780 14y ago
  agree - i see a lot of start ups putting haproxy behind nginx for load balancing but i have never figured out why they wouldnt just stick with nginx. does anyone have an example of how the configuration looks on github?
  - tszming 14y ago
    there are features that haproxy support out of the box, e.g. sticky session load balancing, http 1.1 to upstream (newer version of nginx also support btw); on the other hand, people use nginx for SSL termination.
    - misiti3780 14y ago
      any chance could either explain this a bit more or post a link to a blog? this is very interesting to me and i do not know much about it
- emmett 14y ago
  HAproxy is great at exactly one thing: load balancing. It's better than nginx for that one use, because it's more flexible, has better controls for flapping, is smarter about queuing, gives you cool stats pages, etc. Nginx is great for...pretty much everything else.
  - deleted 14y ago
    [deleted]
jgannonjr 14y ago
Great post, but this part scares me a bit... I think a lot of services (even banks!) have serious security problems and seem to be able to weather a small PR storm. So figure it out if it really is important to you (are you worth hacking? do you actually care if you’re hacked? is it worth the engineering or product cost?) before you go and lock down everything. Just because you can "afford" to be hacked, doesn't mean you shouldn't take all the steps necessary to proactively protect your data. In the end, security is not about you, it is about your users. This is exactly the type of attitude that leads to all the massive breaches we have been seeing recently. Sure your company is "hurt" with bad PR, but really your users are the ones who are the real victims. You should consider their risk (especially with something as sensitive as people's files!) before you consider your own company's well being. Edit: formatting
- cma 14y ago
  That's an externality; shareholders can sue the board of directors if they find out it is company policy to waste resources on it beyond its potential costs to the company: PR + legal liability + probability of causing new regulation * the cost of adhering to said regulation. Sucks, but that's capitalism. However, there are a few states now which allow you to have some charitable clauses in your corporate charter.
- jandrewrogers 14y ago
  Yeah, that point significantly underestimates the cost of cleaning up once your systems have been penetrated. By the time you notice that one system has been compromised, there is no guarantee that every system at your company is not compromised, particularly if so little effort is put into a robust security architecture. I've seen companies that took the attitude the author does and ended up paying for it down the road. Systems get compromised, it happens. Organizations with weak security architectures can become so compromised that cleanup becomes a nightmare because it is difficult to isolate the threat(s) without serious disruption in services. A strong security architecture is not so much to ensure breaches never happen but to limit the amount of damage likely to occur when breaches do happen. And yes, this happens even to organizations that think they have nothing worth hacking.
brc 14y ago
The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you leave for the train a couple of minutes later because you know your watch is actually running fast.
- joshma 14y ago
  It's not exactly the same, though. If you're late 5 minutes by your watch you're on time naturally, without any action. If you're "overloaded" on your servers, you at least have to consciously decrease the extra load or risk having real consequences.
- apu 14y ago
  I actually purposely used to set all the various clocks at home ahead by anywhere from 0 - 15 minutes. At first, I could remember which ones were ahead by how much, but then soon I started to forgot and had to just assume they were running at the right time. It worked great. After a few years of this, I set them all back to right time and found that I had trained myself to just leave at the right time, with no more trickery needed.
  - inerte 14y ago
    I only have one alarm. If it fails I am late. I found out that depending on complex systems work against you. Once I had three wake up alarms, at different points at the bedroom. Didn't work. Being late is lame. Suffering its consequences is the best teacher one can have.
    - chipsy 14y ago
      Were you the college roommate I had who suspended the alarm on a string over the bed so that standing was required to turn it off?
akent 14y ago
I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have Why not take the extra half a second to make those random strings meaningful and hidden behind a DEBUG log level?
- ephemeralgomi 14y ago
  Probably most of their logging _is_ meaningful, but deciding how to professionally phrase each and every log message will eventually get you to decision fatigue. The point that he was making with this was that over-logging is a good thing - this probably wasn't something the initial author thought was going to be terribly informative, hence the random string. And yet it ended up diagnosing a real world problem. In a perfect world, by all means properly write out your messages - but if you're stalling on a log message because you're not sure how to phrase it, you may get concrete benefit from just dropping a FUUUCCKKKKKasdjkfnff and moving on.
  - christoph 14y ago
    So, so true. When the problem occurs, it's pretty quick for the guy who needs to fix it at 2am, to find where it exploded in the code base, while the original developer is (maybe) passed out in a bar somewhere. Not much else matters. He could of just done :( x 10 and had the same result. The main thing is, it's easily traceable!
    - akent 14y ago
      Sure, for fatal errors, random (greppable) strings aren't so bad, but the OP made it sound like his FFFFFFFFFFFUUUUUU message was expected all the time rather than an exception. If you're going to print something all the time in normal operation, make it meaningful.
      - aboodman 14y ago
        That isn't how I took it. I read it as: there was an error scenario that was clearly happening, which should have resulted in a particular log message. Except it wasn't, which meant that something was racing.
nl 14y ago
I wish he'd left the security advice out. The whole post was excellent, but all the useful points will now be overshadowed by the armchair quarterbacking about security by people who mostly don't understand that ALL security is a compromise, and it is as important to understand and make deliberate decisions about your security as it is to try to make a secure system in the first place.
- bestes 14y ago
  I'm glad he put the security notes in. It is so hard to get true facts about how things are actually done.
  - mturmon 14y ago
    Looking again at the post, I think the author was in fact rather careful to not give away anything about security practices at Dropbox when he was there, for obvious reasons. He keeps many comments at a high level (security/convenience) and refers to a few non-Dropbox examples.
  - deleted 14y ago
    [deleted]
- eranki 14y ago
  I thought about it, but honestly, I think it's important to try to fight against all the sanctimony and handwringing that surrounds security. People should feel comfortable talking about security as a tradeoff without diluting the argument with gratuitous qualifications and apologies.
stratos2 14y ago
all security is a balancing act which is the point he is making. there is always a tradeoff
opminion 14y ago
A topic usually left out in scaling discussions is: how much can one predict? Or is it mostly trial and error? Is it mostly about good "reactive" engineering, would it have benefited from good mathematical modeling?
deleted 14y ago
[deleted]
gallerytungsten 14y ago
Great article. Rajiv made it easy to understand the conceptual framework. The lesson is: always strive to be robust. Test your failure points deliberately. Applicable to more than just server scaling.
misiti3780 14y ago
Great advice: "pick lightweight things that are known to work and see a lot of use outside your company, or else be prepared to become the “primary contributor” to the project."
kevinburke 14y ago
MySQL has a huge network of support and we were pretty sure if we had a problem, Google, Yahoo, or Facebook would have to deal with it and patch it before we did. :) I am fairly certain Google is running its own (patched) version that's fairly different than the off-the-shelf MySQL.
- nl 14y ago
  You mean using the Google Mysql5patches[1]? [1] http://code.google.com/p/google-mysql-tools/wiki/Mysql5Patches http://code.google.com/p/google-mysql-tools/wiki/Mysql5Patch...
- wizard_2 14y ago
  And hopefully they're pushing important stuff upstream, it wouldn't make sense to not leverage the community.
anamax 14y ago
There's a talk about Dropbox scaling at http://www.stanford.edu/class/ee380/winter-schedule-20112012.html http://www.stanford.edu/class/ee380/winter-schedule-20112012... .
dools 14y ago
but I really hate ORM’s and this was just a giant nuisance to deal with I like object relational mapping as a theory (ie. I have an object of type Author which has 1 or more books I can loop over), but I hate ActiveRecord implementations. Eventually, they just end up implementing almost all of SQL but in some arcane bullshit syntax or sequence of method calls that you have to spend a bunch of time learning. I also seriously doubt that anyone has ever written a production system of any reasonable complexity and been able to use the exact same ORM code with absolutely any backend (if you have an example please correct me on this). This barely even works with something like PDO in PHP which is a bare bones abstraction across multiple SQL backends. When it comes down to it, the benefits of ActiveRecord are all but dead on about the third day of development. The data mapper pattern adopted by SQLAlchemy (et. al.) takes all of the shitness of ActiveRecord and adds mind bending complexity to it. SQL is easy to learn and very expressive. Why try and abstract it? I spent years working with an ActiveRecord ORM I wrote myself in my feckless youth and thought that it was the answer to the world's problems. I didn't really understand why it was so terrible until I did a large project in Django and had to use someone else's ORM. When I really analysed it, there were only three things that I really wanted out of an ORM: 1) Make the task of writing complex join statements a bit less tedious 2) Make the task of writing a sub-set of very basic where clauses slightly less tedious 3) Obviate the need for me to detect primary key changes when iterating over a joined result set to detect changes in an object (for example, looping over a list of Authors and their Books) To that end, I wrote this: https://github.com/iaindooley/PluSQL https://github.com/iaindooley/PluSQL It's written in PHP because I like and use PHP but it's a very simple pattern that I would like to see elaborated upon/taken to other languages as I think it provides just the bare minimum amount of functionality to give some real productivity gains without creating a steep learning curve, performance trade-off or any barrier to just writing out SQL statements if that's the fastest way to solve the problem at hand.
- thezilch 14y ago
  All I want from an ORM is to manage caching intelligently. I'll learn some arcane bits to assist the ORM's pathfinding, but I simply can't imagine ORM's strong suits being in writing less tedious queries. Granted, the less-strokes and less-verbose nature of an ORM query is still a nice benefit. There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton
JohnGB 14y ago
I believe that the section on "The security-convenience tradeoff" is fundamentally flawed. A username and password represent a pair. Neither one has meaning in terms of authentication without the other. Take the example where I have forgotten my username (JohnGB), but try with what I think it is (Say JohnB), and enter the correct password for my actual username. The system would then tell me that my username is fine, but that my password isn't. From then on, I would be trying to reset the password for a different user as the system has already told me that my username was correct. Please, for the sake of sane UX, don't do this!
- dudeguy 14y ago
  No way, sir. Saying 'you entered the wrong password' in that case is not any more confusing than the ambiguous error that says 'you got one of them wrong but I'm not gonna tell you which.' most reset password systems are keyed to your email address anyway.
elefont2 14y ago
'Even memcached, which is the conceptually simplest of these technologies and used by so many other companies, had some REALLY nasty memory corruption bugs we had to deal with, so I shudder to think about using stuff that’s newer and more complicated' Does anyone know what memory corruption bugs they are referring to?
wulczer 14y ago
Great article! Small nitpick from someone who just tried this on his server logs :) * on my machine xargs -I implies -L1, so you can drop that * use gnuplot -p or the graphic will disappear immediately after rendering
- ralph 14y ago
  I agree, good article. A sort -n is also required before the uniq since server logs have the time of the request but are printed when the response is complete so they're not necessarily increasing.
crazygringo 14y ago
> I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have :) I've never seen a shorter description of real-world software development. That's it in a nutshell!
mistercow 14y ago
Running with extra load seems inefficient in terms of energy consumption. Would it be possible to achieve the same thing by inserting delays or something that can be turned off?
lobster_johnson 14y ago
I'm surprised that Dropbox actually uses S3 internally to store data. All along I had assumed, wrongly, that Dropbox had built their own distributed storage cluster.

418 points by eranki 14y ago

deleted 14y ago

[deleted]

Fabulous post. Thanks for writing. One point it misses though is to test your backup strategy often. When you scale fast things break very often and it's good to be in practice of restoring from backups every now and then.

mirkules 14y ago
Just started reading a book called "High Performance MySQL" and in one of the early pages, the following advice appears: "It's an excellent idea to run a realistic load simulation on a test server and then literally pull the power plug. The firsthand experience of recovering from a crash is priceless. It saves nasty surprises later." Same goes for testing network connectivity and failover. I can't tell you how many times I've heard things like "The automatic recovery _should_ have kicked in but..." Having a recovery procedure and backup strategy is completely different from having actually restored a backup and recovered from a failure.
- RegEx 14y ago
  Reading High Performance MySQL as well. Loving it so far!
eranki 14y ago
Thanks! Good point. We actually repurposed our offsite database recovery to clone slaves off a master (after LVM was no longer performing), so that's a great way to get more testing in.

matt 14y ago

Nice, love the idea of running with extra load to predict breaking points.

ivankirigin 14y ago

Rajiv is awesome, you should listen to him

akent 14y ago
Says an ex "Product Manager at Dropbox". Edit: Thanks for the downvotes. My point is, just make it unambiguous to everyone in your comment so we don't have to click through your profile. Context matters. e.g.: "I was Product Manager at Dropbox and worked with Rajiv (the OP). He's awesome, you should listen to him." Much better.
- sriramk 14y ago
  Even more reason to listen to him.
- to3m 14y ago
  I've found working with people to be a reasonable way of finding out whether their opinions are worth listening to...
- stratos2 14y ago
  which means his opinion counts at least 100 times more than yours does.
  - carb 14y ago
    What he's saying is that ivankirigin should have said that himself. I don't know that he has any credibility to his statement and wasn't going to give it any merit until akent made me realize that ivankirigin had first-hand experience.

acslater00 14y ago

For the record, I use sqlalchemy 0.6.6 regularly under fairly heavy load, and have never had a problem with it. Any 'sqlalchemy bugs' are inevitably coding mistakes on my part.

kennywinker 14y ago
Yeah, I found that bit quite vague. Are they using SQLAlchemy's object layer, but just not the high level query stuff? Or are they using only the low-level query stuff and nothing else? I'd love to know more about how their system works, if they are indeed not using an ORM. Every time I've tried to build something without an ORM, I just end up writing my own shitty one accidentally.
zzzeek 14y ago
We got a dozen or so email list requests for support from people who I know to be from Dropbox in late 2008. At that time, we were at version 0.4.8. That is an extremely old version and the codebase was quite immature at that time - I personally didn't use SQLAlchemy in production until 0.5 (which of course is because Python was hardly used at all in the early 2000's outside of the zope community, so I was still stuck with Java/Perl gigs). However I am still quite skeptical of the claim that it returned the wrong results. You're expected to watch the SQL you're telling it to generate during development. It will always be true that pushing an ORM will not always generate the SQL you want - which is why you have to make sure those queries are how they should be, before pushing to production. The ORM will of course stick to the plan you've given it - it isn't "deciding" anything, and at worst it can only misinterpret your intent - just like any library. My strategy with SQLAlchemy has always been to under-promote it. If you have lots of big players early adopting you and hitting all the pointy edges, it can damage your rep. There's a group of major folks out there who will never use my library due to old experiences. Others like Reddit and Yelp have hung on, and apparently dropbox is still using the core, hooray ! That's why I'm always amazed at how aggressively MongoDB is promoted, when it seems like they're still going through a lot of growing pains. I guess they sort of have to, given that they're a business and all.

philfreo 14y ago

Can you explain the nginx/HAproxy config a little more?

misiti3780 14y ago
agree - i see a lot of start ups putting haproxy behind nginx for load balancing but i have never figured out why they wouldnt just stick with nginx. does anyone have an example of how the configuration looks on github?
- tszming 14y ago
  there are features that haproxy support out of the box, e.g. sticky session load balancing, http 1.1 to upstream (newer version of nginx also support btw); on the other hand, people use nginx for SSL termination.
  - misiti3780 14y ago
    any chance could either explain this a bit more or post a link to a blog? this is very interesting to me and i do not know much about it
emmett 14y ago
HAproxy is great at exactly one thing: load balancing. It's better than nginx for that one use, because it's more flexible, has better controls for flapping, is smarter about queuing, gives you cool stats pages, etc. Nginx is great for...pretty much everything else.
- deleted 14y ago
  [deleted]

jgannonjr 14y ago

Great post, but this part scares me a bit... I think a lot of services (even banks!) have serious security problems and seem to be able to weather a small PR storm. So figure it out if it really is important to you (are you worth hacking? do you actually care if you’re hacked? is it worth the engineering or product cost?) before you go and lock down everything. Just because you can "afford" to be hacked, doesn't mean you shouldn't take all the steps necessary to proactively protect your data. In the end, security is not about you, it is about your users. This is exactly the type of attitude that leads to all the massive breaches we have been seeing recently. Sure your company is "hurt" with bad PR, but really your users are the ones who are the real victims. You should consider their risk (especially with something as sensitive as people's files!) before you consider your own company's well being. Edit: formatting

cma 14y ago
That's an externality; shareholders can sue the board of directors if they find out it is company policy to waste resources on it beyond its potential costs to the company: PR + legal liability + probability of causing new regulation * the cost of adhering to said regulation. Sucks, but that's capitalism. However, there are a few states now which allow you to have some charitable clauses in your corporate charter.
jandrewrogers 14y ago
Yeah, that point significantly underestimates the cost of cleaning up once your systems have been penetrated. By the time you notice that one system has been compromised, there is no guarantee that every system at your company is not compromised, particularly if so little effort is put into a robust security architecture. I've seen companies that took the attitude the author does and ended up paying for it down the road. Systems get compromised, it happens. Organizations with weak security architectures can become so compromised that cleanup becomes a nightmare because it is difficult to isolate the threat(s) without serious disruption in services. A strong security architecture is not so much to ensure breaches never happen but to limit the amount of damage likely to occur when breaches do happen. And yes, this happens even to organizations that think they have nothing worth hacking.

brc 14y ago

The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you leave for the train a couple of minutes later because you know your watch is actually running fast.

joshma 14y ago
It's not exactly the same, though. If you're late 5 minutes by your watch you're on time naturally, without any action. If you're "overloaded" on your servers, you at least have to consciously decrease the extra load or risk having real consequences.
apu 14y ago
I actually purposely used to set all the various clocks at home ahead by anywhere from 0 - 15 minutes. At first, I could remember which ones were ahead by how much, but then soon I started to forgot and had to just assume they were running at the right time. It worked great. After a few years of this, I set them all back to right time and found that I had trained myself to just leave at the right time, with no more trickery needed.
- inerte 14y ago
  I only have one alarm. If it fails I am late. I found out that depending on complex systems work against you. Once I had three wake up alarms, at different points at the bedroom. Didn't work. Being late is lame. Suffering its consequences is the best teacher one can have.
  - chipsy 14y ago
    Were you the college roommate I had who suspended the alarm on a string over the bed so that standing was required to turn it off?

akent 14y ago

I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have Why not take the extra half a second to make those random strings meaningful and hidden behind a DEBUG log level?

ephemeralgomi 14y ago
Probably most of their logging _is_ meaningful, but deciding how to professionally phrase each and every log message will eventually get you to decision fatigue. The point that he was making with this was that over-logging is a good thing - this probably wasn't something the initial author thought was going to be terribly informative, hence the random string. And yet it ended up diagnosing a real world problem. In a perfect world, by all means properly write out your messages - but if you're stalling on a log message because you're not sure how to phrase it, you may get concrete benefit from just dropping a FUUUCCKKKKKasdjkfnff and moving on.
- christoph 14y ago
  So, so true. When the problem occurs, it's pretty quick for the guy who needs to fix it at 2am, to find where it exploded in the code base, while the original developer is (maybe) passed out in a bar somewhere. Not much else matters. He could of just done :( x 10 and had the same result. The main thing is, it's easily traceable!
  - akent 14y ago
    Sure, for fatal errors, random (greppable) strings aren't so bad, but the OP made it sound like his FFFFFFFFFFFUUUUUU message was expected all the time rather than an exception. If you're going to print something all the time in normal operation, make it meaningful.
    - aboodman 14y ago
      That isn't how I took it. I read it as: there was an error scenario that was clearly happening, which should have resulted in a particular log message. Except it wasn't, which meant that something was racing.

nl 14y ago

I wish he'd left the security advice out. The whole post was excellent, but all the useful points will now be overshadowed by the armchair quarterbacking about security by people who mostly don't understand that ALL security is a compromise, and it is as important to understand and make deliberate decisions about your security as it is to try to make a secure system in the first place.

bestes 14y ago
I'm glad he put the security notes in. It is so hard to get true facts about how things are actually done.
- mturmon 14y ago
  Looking again at the post, I think the author was in fact rather careful to not give away anything about security practices at Dropbox when he was there, for obvious reasons. He keeps many comments at a high level (security/convenience) and refers to a few non-Dropbox examples.
- deleted 14y ago
  [deleted]
eranki 14y ago
I thought about it, but honestly, I think it's important to try to fight against all the sanctimony and handwringing that surrounds security. People should feel comfortable talking about security as a tradeoff without diluting the argument with gratuitous qualifications and apologies.

stratos2 14y ago

all security is a balancing act which is the point he is making. there is always a tradeoff

opminion 14y ago

A topic usually left out in scaling discussions is: how much can one predict? Or is it mostly trial and error? Is it mostly about good "reactive" engineering, would it have benefited from good mathematical modeling?

deleted 14y ago

[deleted]

gallerytungsten 14y ago

Great article. Rajiv made it easy to understand the conceptual framework. The lesson is: always strive to be robust. Test your failure points deliberately. Applicable to more than just server scaling.

misiti3780 14y ago

Great advice: "pick lightweight things that are known to work and see a lot of use outside your company, or else be prepared to become the “primary contributor” to the project."

kevinburke 14y ago

MySQL has a huge network of support and we were pretty sure if we had a problem, Google, Yahoo, or Facebook would have to deal with it and patch it before we did. :) I am fairly certain Google is running its own (patched) version that's fairly different than the off-the-shelf MySQL.

nl 14y ago
You mean using the Google Mysql5patches[1]? [1] http://code.google.com/p/google-mysql-tools/wiki/Mysql5Patches http://code.google.com/p/google-mysql-tools/wiki/Mysql5Patch...
wizard_2 14y ago
And hopefully they're pushing important stuff upstream, it wouldn't make sense to not leverage the community.

anamax 14y ago

There's a talk about Dropbox scaling at http://www.stanford.edu/class/ee380/winter-schedule-20112012.html http://www.stanford.edu/class/ee380/winter-schedule-20112012... .

dools 14y ago

but I really hate ORM’s and this was just a giant nuisance to deal with I like object relational mapping as a theory (ie. I have an object of type Author which has 1 or more books I can loop over), but I hate ActiveRecord implementations. Eventually, they just end up implementing almost all of SQL but in some arcane bullshit syntax or sequence of method calls that you have to spend a bunch of time learning. I also seriously doubt that anyone has ever written a production system of any reasonable complexity and been able to use the exact same ORM code with absolutely any backend (if you have an example please correct me on this). This barely even works with something like PDO in PHP which is a bare bones abstraction across multiple SQL backends. When it comes down to it, the benefits of ActiveRecord are all but dead on about the third day of development. The data mapper pattern adopted by SQLAlchemy (et. al.) takes all of the shitness of ActiveRecord and adds mind bending complexity to it. SQL is easy to learn and very expressive. Why try and abstract it? I spent years working with an ActiveRecord ORM I wrote myself in my feckless youth and thought that it was the answer to the world's problems. I didn't really understand why it was so terrible until I did a large project in Django and had to use someone else's ORM. When I really analysed it, there were only three things that I really wanted out of an ORM: 1) Make the task of writing complex join statements a bit less tedious 2) Make the task of writing a sub-set of very basic where clauses slightly less tedious 3) Obviate the need for me to detect primary key changes when iterating over a joined result set to detect changes in an object (for example, looping over a list of Authors and their Books) To that end, I wrote this: https://github.com/iaindooley/PluSQL https://github.com/iaindooley/PluSQL It's written in PHP because I like and use PHP but it's a very simple pattern that I would like to see elaborated upon/taken to other languages as I think it provides just the bare minimum amount of functionality to give some real productivity gains without creating a steep learning curve, performance trade-off or any barrier to just writing out SQL statements if that's the fastest way to solve the problem at hand.

thezilch 14y ago
All I want from an ORM is to manage caching intelligently. I'll learn some arcane bits to assist the ORM's pathfinding, but I simply can't imagine ORM's strong suits being in writing less tedious queries. Granted, the less-strokes and less-verbose nature of an ORM query is still a nice benefit. There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton

JohnGB 14y ago

I believe that the section on "The security-convenience tradeoff" is fundamentally flawed. A username and password represent a pair. Neither one has meaning in terms of authentication without the other. Take the example where I have forgotten my username (JohnGB), but try with what I think it is (Say JohnB), and enter the correct password for my actual username. The system would then tell me that my username is fine, but that my password isn't. From then on, I would be trying to reset the password for a different user as the system has already told me that my username was correct. Please, for the sake of sane UX, don't do this!

dudeguy 14y ago
No way, sir. Saying 'you entered the wrong password' in that case is not any more confusing than the ambiguous error that says 'you got one of them wrong but I'm not gonna tell you which.' most reset password systems are keyed to your email address anyway.

elefont2 14y ago

'Even memcached, which is the conceptually simplest of these technologies and used by so many other companies, had some REALLY nasty memory corruption bugs we had to deal with, so I shudder to think about using stuff that’s newer and more complicated' Does anyone know what memory corruption bugs they are referring to?

wulczer 14y ago

Great article! Small nitpick from someone who just tried this on his server logs :) * on my machine xargs -I implies -L1, so you can drop that * use gnuplot -p or the graphic will disappear immediately after rendering

ralph 14y ago
I agree, good article. A sort -n is also required before the uniq since server logs have the time of the request but are printed when the response is complete so they're not necessarily increasing.

crazygringo 14y ago

> I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have :) I've never seen a shorter description of real-world software development. That's it in a nutshell!

mistercow 14y ago

Running with extra load seems inefficient in terms of energy consumption. Would it be possible to achieve the same thing by inserting delays or something that can be turned off?

lobster_johnson 14y ago

I'm surprised that Dropbox actually uses S3 internally to store data. All along I had assumed, wrongly, that Dropbox had built their own distributed storage cluster.