5 ms·

A newbie’s guide to scaling AWS

97 points by maxticket 15y ago

christianreed 15y ago
Oh man, Fake Grimlock is the man! Er... Giant bird robot.
njames029 15y ago
Revisu is pretty sweet, great blog as well.
mrmagooey 15y ago
I think I may have had less idea about what your product does after clicking "See Revisu in Action", three cat pictures does not a demo make.
- maxticket 15y ago
  Yeah, we're working on that. Hopefully the upcoming video helps a bit more. It's just one designer and one developer, so we gotta triage some things while we build the product.
prib 15y ago
I can attest to the unreliability of AWS Micro instances for production environments. Two examples: Ex 1: We had a request triggering an asynchronous job that would take 3-4 seconds of processor time. Several times, this job was being throttled, so even in a low traffic situation each time one of those requests came, a whole micro instance was consumed on it making it unavailable for anything else. Switching to small instances solved the problem. Ex 2: When launching a new load balancer, due to some sort of wrong DNS/elastic-ip configuration, it's not uncommon to be hit hard by traffic coming for the last owner of the given ip. We've seen this happen several times in the past. Again, switching from 10+ micro instances to 2 small instances cleared the problem. There's an interesting post by Greg Wilson addressing the throttling problem better than I could: http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instance-cpu-steal/ http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instan...
- Terretta 15y ago
  Thank you for the Greg Wilson link.
jaredstenquist 15y ago
Yes RDS, is great for the money considering the ease of backups and replicas, which normally are a pain in the ass for any sysadmin. Overall this article has little to do with scaling on AWS. Running any sort of production website on a micro instance is pretty foolish, besides a blog or static website perhaps.
blantonl 15y ago
As a large scale AWS user, I'm not sure I can agree with any of these recommendations. Don’t use micro instances for real work There are some use cases where micro instances work in production environments (such as MongoDb arbitrators etc) but running anything in production on AWS on anything less than a m1.large instance is setting you up for failure. Micro and small instances can be used for a personal Web site or tiny blog, but nothing else. When I visually think of a micro or small instance, I envision a PC sitting under a desk performing operations that a server in a rack should be hosting. Have servers on standby. Configure a large instance, stick it in an Elastic Load Balancer with your other web servers, and just stop it.... () I really don't consider one server stopped, on standby, as scaling AWS. In the cases of Web servers, you might need to provision 5-10 Web servers, and having a single stopped instance is not going to allow you to accomplish that. The better approach would be to have a server image prepared that can auto-provision itself (multiple times when needed) into the role defined through user-data passed to the image. Scaling Web servers is an n+1 operation, not a +1 operation. Seriously, use RDS Seriously, if you are doing any real MySQL work on AWS, DO NOT use RDS. The reality is that AWS doesn't nearly do the EBS tuning that you yourself can do (RAID-0, kernel tuning etc) on your own EBS based MySQL implementation. See: Getting Good IO from Amazon's EBS: http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/ http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs... Running MySQL on AWS: http://aws.amazon.com/articles/1663 http://aws.amazon.com/articles/1663 Finally, for MySQL slaves, use RAID-0 over the ephemeral disks which have dramatically better performance than EBS.
- typicalrunt 15y ago
  Configure a large instance, stick it in an Elastic Load Balancer with your other web servers, and just stop it. I agree with allowing AWS to auto-provision N new instances when needed. But from the article and with my financial hat on, when the large instance is stopped and left in a standby role, are you still paying for it?
  - deleted 15y ago
    [deleted]
  - bradly 15y ago
k33n 15y ago
This should be called "A guide to scaling AWS written by a newbie".
- bradhe 15y ago
  zing!
Brajeshwar 15y ago
I've been using a Micro Instance for almost a year (the free tier) to run http://nsfw.in/ http://nsfw.in/ successfully. And the traffic on the site is also not bad at all, with 100,000+ pageviews a month on an average. You've to know the right thing you need for your requirement and tweak it.
yummybear 15y ago
One of our problems in scaling is not starting new instances, but rather to ensure that new instances have the latest version of the platform. Also, we are running a fantasy soccer game with trading windows, so the server load near the trading window closure time (the last hour or so) is orders of magnitude higher than average load. Ideally we would scale up when the load demands it, but with the platform versioning issue unresolved we need to run our platform at "full scale" almost constantly (or scale manually which is a really bad idea when someone forgets to scale back up again for the weekend). We're running in a WISC setup.
- mnutt 15y ago
  What if every time you did a deploy, afterwards you kicked off a job that wrote a snapshot and brought up instances based off that. Then at any point you'd only ever be at most a deploy behind.
ultimatewarrior 15y ago
Your scaling AWS and you're a newb? You must be terrified!

97 points by maxticket 15y ago

christianreed 15y ago

Oh man, Fake Grimlock is the man! Er... Giant bird robot.

njames029 15y ago

Revisu is pretty sweet, great blog as well.

mrmagooey 15y ago

I think I may have had less idea about what your product does after clicking "See Revisu in Action", three cat pictures does not a demo make.

maxticket 15y ago
Yeah, we're working on that. Hopefully the upcoming video helps a bit more. It's just one designer and one developer, so we gotta triage some things while we build the product.

prib 15y ago

I can attest to the unreliability of AWS Micro instances for production environments. Two examples: Ex 1: We had a request triggering an asynchronous job that would take 3-4 seconds of processor time. Several times, this job was being throttled, so even in a low traffic situation each time one of those requests came, a whole micro instance was consumed on it making it unavailable for anything else. Switching to small instances solved the problem. Ex 2: When launching a new load balancer, due to some sort of wrong DNS/elastic-ip configuration, it's not uncommon to be hit hard by traffic coming for the last owner of the given ip. We've seen this happen several times in the past. Again, switching from 10+ micro instances to 2 small instances cleared the problem. There's an interesting post by Greg Wilson addressing the throttling problem better than I could: http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instance-cpu-steal/ http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instan...

Terretta 15y ago
Thank you for the Greg Wilson link.

jaredstenquist 15y ago

Yes RDS, is great for the money considering the ease of backups and replicas, which normally are a pain in the ass for any sysadmin. Overall this article has little to do with scaling on AWS. Running any sort of production website on a micro instance is pretty foolish, besides a blog or static website perhaps.

blantonl 15y ago

As a large scale AWS user, I'm not sure I can agree with any of these recommendations. Don’t use micro instances for real work There are some use cases where micro instances work in production environments (such as MongoDb arbitrators etc) but running anything in production on AWS on anything less than a m1.large instance is setting you up for failure. Micro and small instances can be used for a personal Web site or tiny blog, but nothing else. When I visually think of a micro or small instance, I envision a PC sitting under a desk performing operations that a server in a rack should be hosting. Have servers on standby. Configure a large instance, stick it in an Elastic Load Balancer with your other web servers, and just stop it.... () I really don't consider one server stopped, on standby, as scaling AWS. In the cases of Web servers, you might need to provision 5-10 Web servers, and having a single stopped instance is not going to allow you to accomplish that. The better approach would be to have a server image prepared that can auto-provision itself (multiple times when needed) into the role defined through user-data passed to the image. Scaling Web servers is an n+1 operation, not a +1 operation. Seriously, use RDS Seriously, if you are doing any real MySQL work on AWS, DO NOT use RDS. The reality is that AWS doesn't nearly do the EBS tuning that you yourself can do (RAID-0, kernel tuning etc) on your own EBS based MySQL implementation. See: Getting Good IO from Amazon's EBS: http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/ http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs... Running MySQL on AWS: http://aws.amazon.com/articles/1663 http://aws.amazon.com/articles/1663 Finally, for MySQL slaves, use RAID-0 over the ephemeral disks which have dramatically better performance than EBS.

typicalrunt 15y ago
Configure a large instance, stick it in an Elastic Load Balancer with your other web servers, and just stop it. I agree with allowing AWS to auto-provision N new instances when needed. But from the article and with my financial hat on, when the large instance is stopped and left in a standby role, are you still paying for it?
- deleted 15y ago
  [deleted]
- bradly 15y ago

k33n 15y ago

This should be called "A guide to scaling AWS written by a newbie".

bradhe 15y ago
zing!

Brajeshwar 15y ago

I've been using a Micro Instance for almost a year (the free tier) to run http://nsfw.in/ http://nsfw.in/ successfully. And the traffic on the site is also not bad at all, with 100,000+ pageviews a month on an average. You've to know the right thing you need for your requirement and tweak it.

yummybear 15y ago

One of our problems in scaling is not starting new instances, but rather to ensure that new instances have the latest version of the platform. Also, we are running a fantasy soccer game with trading windows, so the server load near the trading window closure time (the last hour or so) is orders of magnitude higher than average load. Ideally we would scale up when the load demands it, but with the platform versioning issue unresolved we need to run our platform at "full scale" almost constantly (or scale manually which is a really bad idea when someone forgets to scale back up again for the weekend). We're running in a WISC setup.

mnutt 15y ago
What if every time you did a deploy, afterwards you kicked off a job that wrote a snapshot and brought up instances based off that. Then at any point you'd only ever be at most a deploy behind.

ultimatewarrior 15y ago

Your scaling AWS and you're a newb? You must be terrified!