17 ms·

Comparing Amazon EC2 to Joyent SmartOS

48 points by xachen 14y ago

twotwotwo 14y ago
- No one pick a host based on a single microbenchmark, please. Game out what your app would need to achieve your target performance level with what each host offers. Amazon still might lose (sounds like that happened with TripAdvisor) but at least you've made your decision a sane way. - For the best price, you'd want to use a reserved instance if this is a webserver, and perhaps a spot instance if it's a batch/HPC task. You'd want to use high-CPU instances if you really are all about quickly sorting 5M numbers. If your app is or might be come database-bound, DB perf's super key. - I'd love to hear more broadly about the practical differences between the clouds--whether anyone seems to be better at managing noisy neighbors than the next guy, network and I/O performance, etc. One CPU-bound task can't really be the end of it.
- twotwotwo 14y ago
  Did the math on how to get cheaper ECU-hours from Amazon. Prices below are per ECU-hour, not per instance-hour; '1yr' is the one-year reserved instance price with the upfront cost amortized; 'spot' is the spot price in the US West (Oregon) region when I looked just now. Small: 6.5 cents/ecu-hr Small, 1yr: 4.7 cents/ecu-hr Small, spot: 1 cent/ecu-hr High-CPU: 3.3 cents/ecu-hr High-CPU, 1yr: 1.9 cents/ecu-hr High-CPU, spot: 0.56 cents/ecu-hr If RAM matters, high-CPU (7G, or 3.5/core) > Small (1.7G) > EC2 Micro and Joyent extra-small (~0.5G). Again, don't rely too much on figures like these; look at all your needs (not just performance, everything) and try to figure out what'll work for you.
  - zimbatm 14y ago
    cc2.8xlarge is the best in terms of price/ECU-hours. You get a whopping 60GB of RAM, 10Gbps network interface. Downsides are: EBS only, us-east-1 only. cc2.8xlarge: 0.15 cents/ecu-hr cc2.8xlarge 1yr: 0.058 cents/ecu-hr
    - twotwotwo 14y ago
      Ah, it is indeed cheaper, but I'm getting different specific numbers: 240 cents/88 ecu-hr == 2.7 cents/ecu-hr on demand; less if reserved or spot. cc2.8xl is also in the Oregon and Ireland regions, but not others as far as I can tell. 1) Funny that scaling up to one of their largest instance types can improve bang-for-buck, on this metric at least. 2) I wish Amazon made it simpler to sort all this out.
      - zimbatm 14y ago
        Thanks for the check. I don't remember how I calculated that.. Another downside of cc2.8xl is that the spot market is crazy, with spikes at $6/h or more.
- nikcub 14y ago
  > sounds like that happened with TripAdvisor is this a reference to another post or story? I Googled and couldn't find it
  - twotwotwo 14y ago
    Yep: http://highscalability.com/blog/2012/10/2/an-epic-tripadvisor-update-why-not-run-on-the-cloud-the-gran.html http://highscalability.com/blog/2012/10/2/an-epic-tripadviso...
nickporter 14y ago
Anybody else want to share their experience with Joyent versus EC2? I'm considering doing the switch as my EC2 bill is getting a little expensive.
- imroot 14y ago
  At my last job, I did some evaluation between EC2 and Joyent with a node.js based application. Hands down, Joyent beat EC2 up -- there wasn't an explicit 1:1 correlation anywhere close, as I was seeing the performance with a m1.large go in line with what I was seeing for a small instance at joyent. When you got right down to it, what killed that plan was that you were still using Amazon S3 for storage; the money that we would have saved in CPU/processor time we would be eaten alive with transferring that data back to S3.
- james33 14y ago
  We've been using Joyent in production for close to a year (coming from dedicated servers and EC2), and we've had nothing but an outstanding experience.
- Loic 14y ago
  If you serve the US market Joyent is maybe good, if you serve the EU market you have way better deals with EU providers. Joyent have some servers in EU but they are just collocating. EC2 is available worldwide for good latency. Joyent love micro benchmarks because they allow one customer to grab a lot of CPU in burst, but if you are one of the other customers on the same hardware node, it is better to have a bit slower all the time than high variability like Joyent. Predictability when you run your infrastructure is better. Also, Joyent will kick you out of their infrastructure if they consider at their sole discretion that they are not making enough money out of you, but this is my personal experience and may not be yours. So basically, they broke the trust I placed in them, I am migrating out at the moment, so take my comment as coming from someone with a bad experience with them.
  - porker 14y ago
    Loic, any pointers you can give (here or off-list) for the EU providers with way better deals? Thanks!
sonier 14y ago
After seeing this, I thought I would spin up a server with Joyent, just to give it a try. After 10 minutes, the server is still not marked as "provisioning," which seems like an excessive amount of time.
- coenhyde 14y ago
  I've had quiet a bit of experience provisioning machines on Joyent. 95% things work flawlessly, with 20-40 second boot times. However occasionally I'll get 10m+ boot times or machines that just fail to boot. Sounds like you got unlucky. Delete that machine and try again.
ChuckMcM 14y ago
From the article (its conclusion) : "Amazon need to lift their game in terms of CPU performance. They offer a great service that obviously extends far beyond a simple CPU benchmark. But when you can get the same work done in Joyent significantly faster for the comparable price, you’ll get far more mileage per instance, which is ultimately going to save the dollars." Or, "Wow, Amazon sure makes bank on this EC2 stuff, they are selling kilo-core-seconds for a bunch more than Joyent is and they are still the most widely used provider." There have been a couple of good benchmarks around which instances get you the most 'bang' for the buck, but its pretty clear that Amazon is making a ton of money here. A good comparison is a rack of 40 SuperMicro 1U dual socket servers with 192G of ram in them. You can create 480 dedicated instances with 16 GB of RAM each, and run the whole thing in a colo facility for $5K/month (power, networking, and floor space). That is 0.2 cents per second, or a bit more than $10/month running them 100% of the time. Not a bad business to be in.
- ghshephard 14y ago
  That, $0.2cents/second of course, is on the margin. The real challenge to doing this is, of course, not the hardware/hvac/colo costs, but the several millions (tens of millions?) of dollars associated with building the software infrastructure required to scale/deploy the CPU/Storage/Services dynamically. And then there is the well staffed devops/NOC environment required. And the single-minded dedication to building these services so reliable, that you can run your primary business on. Oh, and course having a massively scaled business to both offset all those devops/noc/engineering efforts, and beta-test your scaling environment/server/storage infrastructure. Other than Amazon, I can only think of Google being in a position in which such an enterprise can take off. We tried this at Loudcloud, and it turns out you need more than a bunch of smart people, lots of money, and single-minded focus towards building on-demand infrastructure.
  - EwanToo 14y ago
    Of course today you can get over 90% of the required software off the shelf, see cloudstack and openstack, both used in clusters of 1000s of production servers running 10,000s of VMs.
malkia 14y ago
Sorting 5 million random numbers takes ~20secs - and this is the fastest benchmark? Wait there buddy.... First of all, why would OS choice be of so much significance here, after all this is all executed mainly on the CPU (not much IO). The only difference would be that the random() function might return different things on different systems (libc difference?)... or maybe not (I'm not familiar with ruby's internals). So I'm not sure what this measures at all... At least you can run the benchmark shootout (CPU), or something that can test the I/O.
kdsudac 14y ago
I haven't tried out Joyent, but I've tested out Linode and Storm-on-Demand and they easily beat Amazon EC2. My benchmark was running our production image processing algorithm that utilizes OpenCV via python on Ubuntu Server (definitely CPU-limited). FWIW, here are my benchmarks, along with pricing for Northern California regions: Linode 512: 15.2 seconds, 65 cents per day Storm-on-demand 1 GB: 13.05 seconds, 5 cents per hour EC2-micro: 168 seconds, 2.5 cents per hour EC2-small: 63 seconds, 9 cents per hour EC2-medium (high CPU): 12.13 seconds, 18.6 cents per hour
- feniv 14y ago
  That's very interesting! I converted the prices to be in the same units and did some calculations for better comparison. Linode Cost per day: $0.65 Time per image: 15.2 Images per day: 5684.2105263158 Cost per image: $0.0001143519 Images per dollar : 8744.9392712551 Storm-on-demand Cost per day: $1.2 Time per image: 13.05 Images per day: 6620.6896551724 Cost per image: $0.00018125 Images per dollar : 5517.2413793104 EC2-micro Cost per day: $0.84 Time per image: 168 Images per day: 514.2857142857 Cost per image: 0.0016333333 Images per dollar : 612.2448979592 EC2-small Cost per day: 2.16 Time per image: 63 Images per day: 1371.4285714286 Cost per image: 0.001575 Images per dollar : 634.9206349206 EC2-medium Cost per day: $4.464 Time per image: 12.13 Images per day: 7122.8359439407 Cost per image: 0.0006267167 Images per dollar : 1595.6173709545
  - kdsudac 14y ago
    that's a lot of significant digits :)
ComputerGuru 14y ago
For Windows servers, Joyent is prohibitively more expensive than EC2... though their Linux prices trounce AWS' significantly. I wonder why that is. They also don't offer anything smaller than "medium". Does anyone know a good cloud-based Windows host as an alternative to AWS?
- facorreia 14y ago
  I presume you've tried Windows Azure?
  - rplnt 14y ago
    Isn't azure more of a PaaS than a IaaS?
    - icebraining 14y ago
      It has both: http://www.windowsazure.com/en-us/pricing/details/#virtual-machines http://www.windowsazure.com/en-us/pricing/details/#virtual-m...
  - ComputerGuru 14y ago
    Azure IaaS is even more expensive on a small scale (though it does get cheaper as you scale up). Azure PaaS would fit the bill very nicely, but someone at MS completely dropped the ball: you have no option of using SSL + your own domain! You can have either your own domain name or SSL w/ their subdomain. Who the bleep would settle for that? What kind of "enterprise solution" is this?
    - facorreia 14y ago
      You probably mean the Azure Websites service. It doesn't support SSL on custom domains yet, but it's a preview version. That feature is expected before this service is released. The PaaS offer per se is called Hosted Services and it does support SSL on custom domains.[1] The price for the IaaS offer for a small Windows VM is $0.08/h ($57.60/month).[2] EC2 pricing for a small Windows VM is $0.115/h ($82.80/month).[3] [1] https://www.windowsazure.com/en-us/develop/net/common-tasks/enable-ssl/ https://www.windowsazure.com/en-us/develop/net/common-tasks/... [2] https://www.windowsazure.com/pt-br/pricing/calculator/?scenario=virtual-machines https://www.windowsazure.com/pt-br/pricing/calculator/?scena... [3] http://aws.amazon.com/ec2/pricing/ http://aws.amazon.com/ec2/pricing/
latchkey 14y ago
I recently tried out Joyent SmartOS for a NodeJS consulting gig I'm doing. After about 30 seconds, I killed the vm and switched to Ubuntu 12 server. I don't care if SmartOS is 'faster' or has dtrace. The fact that it is missing basic things like top (no prstat isn't anything close to top) and 'standard' things like upstart made me want to run for the hills. The Joyent wiki documentation on SmartOS is abysmal at best. Even the pkgsrc repository system feels ancient and pales in comparison to what apt is. I have no desire to spend time re-inventing basic tools that linux has had for ages. The image version of SmartOS w/ NodeJS isn't up-to-date with the latest Node release and it starts up MongoDB by default. While this may be good for just getting things running quickly, what happens if I don't want Mongo running at all? Generate my own images? No thanks. I know, lots of hate... but it isn't all bad. Joyent as a cloud platform is great. Basic ab testing against NodeJS had me serving 9900 concurrent connections without any issues on their least expensive vm (which runs on a blazing fast network at 10gb/s). We were on Rackspace before and with a network limited to 20Mbps/s (at the lowest tier) we just couldn't get the throughput we needed. tl;dr: SmartOS is a train wreck. Don't waste your time. Joyent as a platform is great. Why bother with EC2 when Joyent is faster and less expensive?
- james33 14y ago
  You can very easily install whatever version of Node.js you want using this bash script: https://gist.github.com/a0d40f1ff1679ed582a5 https://gist.github.com/a0d40f1ff1679ed582a5. Also, MongoDB isn't even installed by default on the Node.js instance.
  - latchkey 14y ago
    http://wiki.joyent.com/wiki/display/jpc2/SmartMachine+Node.js http://wiki.joyent.com/wiki/display/jpc2/SmartMachine+Node.j... v1.4.0 "This version now runs MongoDB by default" As for your bash script... I stopped compiling binaries ages ago, why would I want to start again when I can just use Ubuntu and the official NodeJS PPA?
- bcantrill 14y ago
  For whatever it's worth, we are very actively working to make SmartOS a more hospitable environment to those coming from other systems. I'm not sure if you're interested in the details here (after all, it sounds like you might have spent longer on the HN comment than on the "30 seconds" with SmartOS), but if you provision the default base smartos64 image (currently version 1.6.3), "top" is available via pkgsrc ("pkgin in top"). Not only that, but due to popular demand, we've also ported "htop" to SmartOS ("pkgin in htop"). In terms of node.js version, I think what you're referring to is the node found in the base platform (/usr/bin/node), not the node available via pkgsrc (which will appear as /opt/local/bin/node). This is definitely a point of confusion, and we've rectified it in the next version of our software by not having the node version that we use in the base platform be visible to SmartMachines (that is, there isn't a /usr/bin/node). More generally, we're definitely committed to making SmartOS the best environment for the modern cloud; I'm sorry that your experience was frustrating, and if you have any additional concrete feedback on what you'd like to see improved, we're all ears. And finally, I know you said that you "don't care" about DTrace, but I just can't help my (biased) self; if you're developing a node.js app, you really should check out some of the things that SmartOS can uniquely do with respect to node: http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/ http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/ http://dtrace.org/blogs/bmc/2012/05/05/debugging-node-js-memory-leaks/ http://dtrace.org/blogs/bmc/2012/05/05/debugging-node-js-mem...
feniv 14y ago
Running a benchmark on services like this can be really tricky. It really depends on which cluster you were allocated and what other applications were running at that time. If someone else was running a heavy process at the same time, then that could explain some of the sluggishness.. Even still, the numbers are so drastically different that this question does deserve a second look. Someone from Joyent or EC2 should really put together a solid benchmark to prove once and for that their service is indeed better in a wide range of compute and networking categories.
kristianp 14y ago
I tried out Joyent back when they did a year's free VM promotion with Facebook. I found Solaris to be very nice, but it required a little work to get software built for linux to build on it. Different C compiler for one thing. Usually for common pieces of software, like ruby there is a guide on the Joyent website for compiling.
nodesocket 14y ago
The `ruby` benchmark on my iMac i3 3GHZ took `14.5` seconds. Inside of a CentOS 5.3 x64 virtual machine on Softlayer with 2 cpus and 1GB of memory got `15.6` seconds. Not sure how Joyent and especially AWS performed so poorly on that benchmark.
- tbe 14y ago
  CPU quota
4ad 14y ago
If you think this benchmark is impressive... just do some I/O benchmarking. It blows EC2 away much harder than it does for CPU. ZFS on local disks with SSDs for L2ARC, and no virtualization overhead (this obviously applies to SmartOS). Oh, and the performance is predictable and doesn't vary in time. And you have DTrace to understand your application.
rplnt 14y ago
I never used any cloud computing but hell that's slow. Just tried the "sort five million numbers" on my desktop and it took under 5 seconds (something above 5 in VM). I tried it in python though.. but I imagine ruby's performance would be about the same in this regard.
adamlindsay 14y ago
An AWS micro is designed for burstable CPU loads, i.e. Websites. Performing a CPU intensive task will never do well. Amazon even says this in their own documentation. The second fault I have is that articles like this, is that they only every look at the OnDemand price, which most people using AWS seriously don't pay. AWS has reserved pricing and spot both of which will dramatically drop the pricing. Maybe not as inexpensively as Joyent, but much much closer.
moe 14y ago
I don't understand why Joyent clings to a dead operating system. Are they planning to single-handedly keep solaris alive? Cloud computing is a commodity. Many (most?) customers use multiple clouds in parallel. Learning and maintaining the intricacies of a niche OS in this environment is a nonsensical proposition. And for what? For a few percentage points on a CPU benchmark? Seriously?
- MatthewPhillips 14y ago
  You do know they employ many of Solaris' core developers, right?
  - moe 14y ago
    And how is that relevant?
ksec 14y ago
I have been asking, and wondering why would anyone want to use AWS. It is insanely expensive. It is Slow on CPU, Slow on I/O, Low on Bandwidth, Low on memory, and Zero Support. Compared to many other alternatives that gives you 3 – 4 times the on all the above plus better support for the same price. Even their High CPU and Memory plan is still on the expensive side, only the High I/O SSD offers some value. AWS works great when you are in the mid range of expanding, then you can calculate your usage and book their server in advance using Reserved Instances. Instagram and Netflix as example. But that is hassle you have to put though, there are alternatives that are priced and perform just as well without using Reserved Instances. And once you got past the mid range, a managed dedicated server and some instant cloud severs should be much more cost effective. Apart from being forced by VC ( because they are heavily invested in Amazon ) i could not understand why startup are choosing AWS at all.
- statictype 14y ago
  I have been asking, and wondering why would anyone want to use AWS It's dead-simple to get started. It's widely known. I can convince my CTO to use Amazon for cloud-hosting. He's never heard of Rackspace or Joylent - people who are serious about running their infrastructure on the cloud will research and know about them but for quick experimenting and companies just getting their feet wet, Amazon has a big edge here.
- marceldegraaf 14y ago
  We use AWS (and with great satisfaction) because it provides a full offering. We don't just need a bunch of servers, we also need databases, one or more queues, backup storage, caching and so forth. AWS provides all of these services, offers one (reasonably clear) dashboard for them and makes it insanely easy to get them all running and tied together. As for the Zero Support thing: our experience is that it is quite easy to get support from people at Amazon. I've been emailing extensively with one of their platform architects which was actually a very nice experience.
- sojacques 14y ago
  You're generalizing too much if you believe that the whole AWS stack is bad. S3 is very reliable, and I'm not aware of anything as reliable (we're talking eleven nines)/simple/cheap/distributed. EC2 gives you the ability to manage instances in several continents with a dead simple API, which is unheard of anywhere else at this price. ELB is very reliable, simple to use, and saves you hours of setup. My point is that if you really wonder why anyone would want to use AWS, you might not be considering AWS, but only EC2, and in a very non-international way.
tszming 14y ago
For a more detailed comparison, see: Is Joyent Really 14X Faster than EC2 and Azure the "Fastest Cloud"? (tl;dr - EC2 isn't that bad) http://blog.cloudharmony.com/2011/11/many-are-skeptical-of-claims-that.html http://blog.cloudharmony.com/2011/11/many-are-skeptical-of-c... To me, the weakest part of AWS is always on the technical support - you have no right to create a ticket to them unless you pay for the support fee (as contrast to Joyent /Rackspace/Linode), even you have discovered a bug in their system! On the other hand, I also agree with @latchkey that SmartOS /Ubuntu is an important factor when we are choosing the platform - for a lot of people, ease of use and larger community is more important than cutting edge features like dtrace and zfs.

48 points by xachen 14y ago

- No one pick a host based on a single microbenchmark, please. Game out what your app would need to achieve your target performance level with what each host offers. Amazon still might lose (sounds like that happened with TripAdvisor) but at least you've made your decision a sane way. - For the best price, you'd want to use a reserved instance if this is a webserver, and perhaps a spot instance if it's a batch/HPC task. You'd want to use high-CPU instances if you really are all about quickly sorting 5M numbers. If your app is or might be come database-bound, DB perf's super key. - I'd love to hear more broadly about the practical differences between the clouds--whether anyone seems to be better at managing noisy neighbors than the next guy, network and I/O performance, etc. One CPU-bound task can't really be the end of it.

twotwotwo 14y ago
Did the math on how to get cheaper ECU-hours from Amazon. Prices below are per ECU-hour, not per instance-hour; '1yr' is the one-year reserved instance price with the upfront cost amortized; 'spot' is the spot price in the US West (Oregon) region when I looked just now. Small: 6.5 cents/ecu-hr Small, 1yr: 4.7 cents/ecu-hr Small, spot: 1 cent/ecu-hr High-CPU: 3.3 cents/ecu-hr High-CPU, 1yr: 1.9 cents/ecu-hr High-CPU, spot: 0.56 cents/ecu-hr If RAM matters, high-CPU (7G, or 3.5/core) > Small (1.7G) > EC2 Micro and Joyent extra-small (~0.5G). Again, don't rely too much on figures like these; look at all your needs (not just performance, everything) and try to figure out what'll work for you.
- zimbatm 14y ago
  cc2.8xlarge is the best in terms of price/ECU-hours. You get a whopping 60GB of RAM, 10Gbps network interface. Downsides are: EBS only, us-east-1 only. cc2.8xlarge: 0.15 cents/ecu-hr cc2.8xlarge 1yr: 0.058 cents/ecu-hr
  - twotwotwo 14y ago
    Ah, it is indeed cheaper, but I'm getting different specific numbers: 240 cents/88 ecu-hr == 2.7 cents/ecu-hr on demand; less if reserved or spot. cc2.8xl is also in the Oregon and Ireland regions, but not others as far as I can tell. 1) Funny that scaling up to one of their largest instance types can improve bang-for-buck, on this metric at least. 2) I wish Amazon made it simpler to sort all this out.
    - zimbatm 14y ago
      Thanks for the check. I don't remember how I calculated that.. Another downside of cc2.8xl is that the spot market is crazy, with spikes at $6/h or more.
nikcub 14y ago
> sounds like that happened with TripAdvisor is this a reference to another post or story? I Googled and couldn't find it
- twotwotwo 14y ago
  Yep: http://highscalability.com/blog/2012/10/2/an-epic-tripadvisor-update-why-not-run-on-the-cloud-the-gran.html http://highscalability.com/blog/2012/10/2/an-epic-tripadviso...

nickporter 14y ago

Anybody else want to share their experience with Joyent versus EC2? I'm considering doing the switch as my EC2 bill is getting a little expensive.

imroot 14y ago
At my last job, I did some evaluation between EC2 and Joyent with a node.js based application. Hands down, Joyent beat EC2 up -- there wasn't an explicit 1:1 correlation anywhere close, as I was seeing the performance with a m1.large go in line with what I was seeing for a small instance at joyent. When you got right down to it, what killed that plan was that you were still using Amazon S3 for storage; the money that we would have saved in CPU/processor time we would be eaten alive with transferring that data back to S3.
james33 14y ago
We've been using Joyent in production for close to a year (coming from dedicated servers and EC2), and we've had nothing but an outstanding experience.
Loic 14y ago
If you serve the US market Joyent is maybe good, if you serve the EU market you have way better deals with EU providers. Joyent have some servers in EU but they are just collocating. EC2 is available worldwide for good latency. Joyent love micro benchmarks because they allow one customer to grab a lot of CPU in burst, but if you are one of the other customers on the same hardware node, it is better to have a bit slower all the time than high variability like Joyent. Predictability when you run your infrastructure is better. Also, Joyent will kick you out of their infrastructure if they consider at their sole discretion that they are not making enough money out of you, but this is my personal experience and may not be yours. So basically, they broke the trust I placed in them, I am migrating out at the moment, so take my comment as coming from someone with a bad experience with them.
- porker 14y ago
  Loic, any pointers you can give (here or off-list) for the EU providers with way better deals? Thanks!

sonier 14y ago

After seeing this, I thought I would spin up a server with Joyent, just to give it a try. After 10 minutes, the server is still not marked as "provisioning," which seems like an excessive amount of time.

coenhyde 14y ago
I've had quiet a bit of experience provisioning machines on Joyent. 95% things work flawlessly, with 20-40 second boot times. However occasionally I'll get 10m+ boot times or machines that just fail to boot. Sounds like you got unlucky. Delete that machine and try again.

ChuckMcM 14y ago

From the article (its conclusion) : "Amazon need to lift their game in terms of CPU performance. They offer a great service that obviously extends far beyond a simple CPU benchmark. But when you can get the same work done in Joyent significantly faster for the comparable price, you’ll get far more mileage per instance, which is ultimately going to save the dollars." Or, "Wow, Amazon sure makes bank on this EC2 stuff, they are selling kilo-core-seconds for a bunch more than Joyent is and they are still the most widely used provider." There have been a couple of good benchmarks around which instances get you the most 'bang' for the buck, but its pretty clear that Amazon is making a ton of money here. A good comparison is a rack of 40 SuperMicro 1U dual socket servers with 192G of ram in them. You can create 480 dedicated instances with 16 GB of RAM each, and run the whole thing in a colo facility for $5K/month (power, networking, and floor space). That is 0.2 cents per second, or a bit more than $10/month running them 100% of the time. Not a bad business to be in.

ghshephard 14y ago
That, $0.2cents/second of course, is on the margin. The real challenge to doing this is, of course, not the hardware/hvac/colo costs, but the several millions (tens of millions?) of dollars associated with building the software infrastructure required to scale/deploy the CPU/Storage/Services dynamically. And then there is the well staffed devops/NOC environment required. And the single-minded dedication to building these services so reliable, that you can run your primary business on. Oh, and course having a massively scaled business to both offset all those devops/noc/engineering efforts, and beta-test your scaling environment/server/storage infrastructure. Other than Amazon, I can only think of Google being in a position in which such an enterprise can take off. We tried this at Loudcloud, and it turns out you need more than a bunch of smart people, lots of money, and single-minded focus towards building on-demand infrastructure.
- EwanToo 14y ago
  Of course today you can get over 90% of the required software off the shelf, see cloudstack and openstack, both used in clusters of 1000s of production servers running 10,000s of VMs.

malkia 14y ago

Sorting 5 million random numbers takes ~20secs - and this is the fastest benchmark? Wait there buddy.... First of all, why would OS choice be of so much significance here, after all this is all executed mainly on the CPU (not much IO). The only difference would be that the random() function might return different things on different systems (libc difference?)... or maybe not (I'm not familiar with ruby's internals). So I'm not sure what this measures at all... At least you can run the benchmark shootout (CPU), or something that can test the I/O.

kdsudac 14y ago

I haven't tried out Joyent, but I've tested out Linode and Storm-on-Demand and they easily beat Amazon EC2. My benchmark was running our production image processing algorithm that utilizes OpenCV via python on Ubuntu Server (definitely CPU-limited). FWIW, here are my benchmarks, along with pricing for Northern California regions: Linode 512: 15.2 seconds, 65 cents per day Storm-on-demand 1 GB: 13.05 seconds, 5 cents per hour EC2-micro: 168 seconds, 2.5 cents per hour EC2-small: 63 seconds, 9 cents per hour EC2-medium (high CPU): 12.13 seconds, 18.6 cents per hour

feniv 14y ago
That's very interesting! I converted the prices to be in the same units and did some calculations for better comparison. Linode Cost per day: $0.65 Time per image: 15.2 Images per day: 5684.2105263158 Cost per image: $0.0001143519 Images per dollar : 8744.9392712551 Storm-on-demand Cost per day: $1.2 Time per image: 13.05 Images per day: 6620.6896551724 Cost per image: $0.00018125 Images per dollar : 5517.2413793104 EC2-micro Cost per day: $0.84 Time per image: 168 Images per day: 514.2857142857 Cost per image: 0.0016333333 Images per dollar : 612.2448979592 EC2-small Cost per day: 2.16 Time per image: 63 Images per day: 1371.4285714286 Cost per image: 0.001575 Images per dollar : 634.9206349206 EC2-medium Cost per day: $4.464 Time per image: 12.13 Images per day: 7122.8359439407 Cost per image: 0.0006267167 Images per dollar : 1595.6173709545
- kdsudac 14y ago
  that's a lot of significant digits :)

ComputerGuru 14y ago

For Windows servers, Joyent is prohibitively more expensive than EC2... though their Linux prices trounce AWS' significantly. I wonder why that is. They also don't offer anything smaller than "medium". Does anyone know a good cloud-based Windows host as an alternative to AWS?

facorreia 14y ago
I presume you've tried Windows Azure?
- rplnt 14y ago
  Isn't azure more of a PaaS than a IaaS?
  - icebraining 14y ago
    It has both: http://www.windowsazure.com/en-us/pricing/details/#virtual-machines http://www.windowsazure.com/en-us/pricing/details/#virtual-m...
- ComputerGuru 14y ago
  Azure IaaS is even more expensive on a small scale (though it does get cheaper as you scale up). Azure PaaS would fit the bill very nicely, but someone at MS completely dropped the ball: you have no option of using SSL + your own domain! You can have either your own domain name or SSL w/ their subdomain. Who the bleep would settle for that? What kind of "enterprise solution" is this?
  - facorreia 14y ago
    You probably mean the Azure Websites service. It doesn't support SSL on custom domains yet, but it's a preview version. That feature is expected before this service is released. The PaaS offer per se is called Hosted Services and it does support SSL on custom domains.[1] The price for the IaaS offer for a small Windows VM is $0.08/h ($57.60/month).[2] EC2 pricing for a small Windows VM is $0.115/h ($82.80/month).[3] [1] https://www.windowsazure.com/en-us/develop/net/common-tasks/enable-ssl/ https://www.windowsazure.com/en-us/develop/net/common-tasks/... [2] https://www.windowsazure.com/pt-br/pricing/calculator/?scenario=virtual-machines https://www.windowsazure.com/pt-br/pricing/calculator/?scena... [3] http://aws.amazon.com/ec2/pricing/ http://aws.amazon.com/ec2/pricing/

latchkey 14y ago

I recently tried out Joyent SmartOS for a NodeJS consulting gig I'm doing. After about 30 seconds, I killed the vm and switched to Ubuntu 12 server. I don't care if SmartOS is 'faster' or has dtrace. The fact that it is missing basic things like top (no prstat isn't anything close to top) and 'standard' things like upstart made me want to run for the hills. The Joyent wiki documentation on SmartOS is abysmal at best. Even the pkgsrc repository system feels ancient and pales in comparison to what apt is. I have no desire to spend time re-inventing basic tools that linux has had for ages. The image version of SmartOS w/ NodeJS isn't up-to-date with the latest Node release and it starts up MongoDB by default. While this may be good for just getting things running quickly, what happens if I don't want Mongo running at all? Generate my own images? No thanks. I know, lots of hate... but it isn't all bad. Joyent as a cloud platform is great. Basic ab testing against NodeJS had me serving 9900 concurrent connections without any issues on their least expensive vm (which runs on a blazing fast network at 10gb/s). We were on Rackspace before and with a network limited to 20Mbps/s (at the lowest tier) we just couldn't get the throughput we needed. tl;dr: SmartOS is a train wreck. Don't waste your time. Joyent as a platform is great. Why bother with EC2 when Joyent is faster and less expensive?

james33 14y ago
You can very easily install whatever version of Node.js you want using this bash script: https://gist.github.com/a0d40f1ff1679ed582a5 https://gist.github.com/a0d40f1ff1679ed582a5. Also, MongoDB isn't even installed by default on the Node.js instance.
- latchkey 14y ago
  http://wiki.joyent.com/wiki/display/jpc2/SmartMachine+Node.js http://wiki.joyent.com/wiki/display/jpc2/SmartMachine+Node.j... v1.4.0 "This version now runs MongoDB by default" As for your bash script... I stopped compiling binaries ages ago, why would I want to start again when I can just use Ubuntu and the official NodeJS PPA?
bcantrill 14y ago
For whatever it's worth, we are very actively working to make SmartOS a more hospitable environment to those coming from other systems. I'm not sure if you're interested in the details here (after all, it sounds like you might have spent longer on the HN comment than on the "30 seconds" with SmartOS), but if you provision the default base smartos64 image (currently version 1.6.3), "top" is available via pkgsrc ("pkgin in top"). Not only that, but due to popular demand, we've also ported "htop" to SmartOS ("pkgin in htop"). In terms of node.js version, I think what you're referring to is the node found in the base platform (/usr/bin/node), not the node available via pkgsrc (which will appear as /opt/local/bin/node). This is definitely a point of confusion, and we've rectified it in the next version of our software by not having the node version that we use in the base platform be visible to SmartMachines (that is, there isn't a /usr/bin/node). More generally, we're definitely committed to making SmartOS the best environment for the modern cloud; I'm sorry that your experience was frustrating, and if you have any additional concrete feedback on what you'd like to see improved, we're all ears. And finally, I know you said that you "don't care" about DTrace, but I just can't help my (biased) self; if you're developing a node.js app, you really should check out some of the things that SmartOS can uniquely do with respect to node: http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/ http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/ http://dtrace.org/blogs/bmc/2012/05/05/debugging-node-js-memory-leaks/ http://dtrace.org/blogs/bmc/2012/05/05/debugging-node-js-mem...

feniv 14y ago

Running a benchmark on services like this can be really tricky. It really depends on which cluster you were allocated and what other applications were running at that time. If someone else was running a heavy process at the same time, then that could explain some of the sluggishness.. Even still, the numbers are so drastically different that this question does deserve a second look. Someone from Joyent or EC2 should really put together a solid benchmark to prove once and for that their service is indeed better in a wide range of compute and networking categories.

kristianp 14y ago

I tried out Joyent back when they did a year's free VM promotion with Facebook. I found Solaris to be very nice, but it required a little work to get software built for linux to build on it. Different C compiler for one thing. Usually for common pieces of software, like ruby there is a guide on the Joyent website for compiling.

nodesocket 14y ago

The `ruby` benchmark on my iMac i3 3GHZ took `14.5` seconds. Inside of a CentOS 5.3 x64 virtual machine on Softlayer with 2 cpus and 1GB of memory got `15.6` seconds. Not sure how Joyent and especially AWS performed so poorly on that benchmark.

tbe 14y ago
CPU quota

4ad 14y ago

If you think this benchmark is impressive... just do some I/O benchmarking. It blows EC2 away much harder than it does for CPU. ZFS on local disks with SSDs for L2ARC, and no virtualization overhead (this obviously applies to SmartOS). Oh, and the performance is predictable and doesn't vary in time. And you have DTrace to understand your application.

rplnt 14y ago

I never used any cloud computing but hell that's slow. Just tried the "sort five million numbers" on my desktop and it took under 5 seconds (something above 5 in VM). I tried it in python though.. but I imagine ruby's performance would be about the same in this regard.

adamlindsay 14y ago

An AWS micro is designed for burstable CPU loads, i.e. Websites. Performing a CPU intensive task will never do well. Amazon even says this in their own documentation. The second fault I have is that articles like this, is that they only every look at the OnDemand price, which most people using AWS seriously don't pay. AWS has reserved pricing and spot both of which will dramatically drop the pricing. Maybe not as inexpensively as Joyent, but much much closer.

moe 14y ago

I don't understand why Joyent clings to a dead operating system. Are they planning to single-handedly keep solaris alive? Cloud computing is a commodity. Many (most?) customers use multiple clouds in parallel. Learning and maintaining the intricacies of a niche OS in this environment is a nonsensical proposition. And for what? For a few percentage points on a CPU benchmark? Seriously?

MatthewPhillips 14y ago
You do know they employ many of Solaris' core developers, right?
- moe 14y ago
  And how is that relevant?

ksec 14y ago

I have been asking, and wondering why would anyone want to use AWS. It is insanely expensive. It is Slow on CPU, Slow on I/O, Low on Bandwidth, Low on memory, and Zero Support. Compared to many other alternatives that gives you 3 – 4 times the on all the above plus better support for the same price. Even their High CPU and Memory plan is still on the expensive side, only the High I/O SSD offers some value. AWS works great when you are in the mid range of expanding, then you can calculate your usage and book their server in advance using Reserved Instances. Instagram and Netflix as example. But that is hassle you have to put though, there are alternatives that are priced and perform just as well without using Reserved Instances. And once you got past the mid range, a managed dedicated server and some instant cloud severs should be much more cost effective. Apart from being forced by VC ( because they are heavily invested in Amazon ) i could not understand why startup are choosing AWS at all.

statictype 14y ago
I have been asking, and wondering why would anyone want to use AWS It's dead-simple to get started. It's widely known. I can convince my CTO to use Amazon for cloud-hosting. He's never heard of Rackspace or Joylent - people who are serious about running their infrastructure on the cloud will research and know about them but for quick experimenting and companies just getting their feet wet, Amazon has a big edge here.
marceldegraaf 14y ago
We use AWS (and with great satisfaction) because it provides a full offering. We don't just need a bunch of servers, we also need databases, one or more queues, backup storage, caching and so forth. AWS provides all of these services, offers one (reasonably clear) dashboard for them and makes it insanely easy to get them all running and tied together. As for the Zero Support thing: our experience is that it is quite easy to get support from people at Amazon. I've been emailing extensively with one of their platform architects which was actually a very nice experience.
sojacques 14y ago
You're generalizing too much if you believe that the whole AWS stack is bad. S3 is very reliable, and I'm not aware of anything as reliable (we're talking eleven nines)/simple/cheap/distributed. EC2 gives you the ability to manage instances in several continents with a dead simple API, which is unheard of anywhere else at this price. ELB is very reliable, simple to use, and saves you hours of setup. My point is that if you really wonder why anyone would want to use AWS, you might not be considering AWS, but only EC2, and in a very non-international way.

tszming 14y ago

For a more detailed comparison, see: Is Joyent Really 14X Faster than EC2 and Azure the "Fastest Cloud"? (tl;dr - EC2 isn't that bad) http://blog.cloudharmony.com/2011/11/many-are-skeptical-of-claims-that.html http://blog.cloudharmony.com/2011/11/many-are-skeptical-of-c... To me, the weakest part of AWS is always on the technical support - you have no right to create a ticket to them unless you pay for the support fee (as contrast to Joyent /Rackspace/Linode), even you have discovered a bug in their system! On the other hand, I also agree with @latchkey that SmartOS /Ubuntu is an important factor when we are choosing the platform - for a lot of people, ease of use and larger community is more important than cutting edge features like dtrace and zfs.