7 ms·

Going Multi-Cloud with AWS and GCP: Lessons Learned at Scale

229 points by jbyers 9y ago

dswalter 9y ago
If AWS were to go to a per-minute billing cycle, they would be instantly more price-competitive with Google's offering. Or, to put it the other way around, those leftover minutes form a significant chunk of AWS's profit margin.
- boulos 9y ago
  Agreed, and I hope they do so! Though there would still be other things like the lower on-demand rates, custom shapes, networking that scales with shape (rather than being coarsely grouped), being able to attach SSD / GPUs semi-arbitrarily, and so on. For those that care, not having to pay up front for the best price is also a huge deal. You see the same thing in GCS vs S3 as well: Glacier and S3-IA have a few rounding up gotchas that catch many people out. All that said, I hope we all get to per-minute billing. Disclosure: I work on Google Cloud (but haven't talked to the Metamarkets folks)
- obulpathi 9y ago
  I don't think so. GCP's bill is usually about 50% of AWS's bill for same application, if you run it full hour (from my personal experiences and from several others as well: https://thehftguy.com/2016/11/18/google-cloud-is-50-cheaper-than-aws/ https://thehftguy.com/2016/11/18/google-cloud-is-50-cheaper-...). GCP has lot more cost saving features like seamless scalability, custom shapes, sustained discounts and so on. If you workloads span less than hour, GCP can offer more then 50% savings.
  - boulos 9y ago
    He said "more" price-competitive :). I think we're all saying the same thing.
  - ranman 9y ago
    I refuted some of the networking claims in that article previously (I work for AWS). Especially the bizarre claims that you have to get a C4.4xlarge for 1gpbs... The 220 mpbs network cap claim is just not true. Just run iperf3 on any aws instance to a GCE instance and you can see greater than 220mpbs.
    - kuschku 9y ago
      > Just run iperf3 on any aws instance to a GCE instance and you can see greater than 220mpbs. For how long is the question. Historically, it’s been considered common knowledge (might just be an urban legend) that AWS, even if you pay for more traffic, at some point just throttles you, the same way that they do with IO.
    - isatty 9y ago
      Honestly we all know that the small instances have terrible CPU that doesn't let you use the advertized 1Gbps anyway. Other than that, even if AWS let 1Gbps traffic go on for a while, you get throttled pretty quickly from my experience.
    - user5994461 9y ago
      Author of the quoted article here. The run of iperf refuted your refutation.
swozey 9y ago
Great thorough comparison and falls very into line with my experience. Definitely worth the read. Thanks!
throwaway0071 9y ago
Off Topic: it's frustrating that these companies spend quite a lot of time and money learning about the complexities of their infrastructure but when you're interviewing at such companies, you're expected to have answers for everything and a complete strategy for the cloud. /rant
mrg3_2013 9y ago
Nice post! I will be using it as a reference.
whatsmyhandle 9y ago
Very nice writeup! A nice, detailed read that was easy to understand. It seems to focus more on raw infrastructure (EC2 vs GCE) instead of each company's PaaS offerings. Obviously AWS has the front runner lead here, but would be super curious in a comparison of RDS vs. Cloud Spanner for instance. (pun unintentional, but then realized, and left in there)
- vira28 9y ago
  Did you mean to say AWS Aurora vs Cloud Spanner? Because, I don't think you can compare RDS vs Cloud Spanner. RDS is a managed for the most of the famous RDBMS out in the market (except Aurora). Cloud Spanner is a google proprietary db running only on GC.
- outworlder 9y ago
  This should be RDS vs Cloud SQL
nodesocket 9y ago
One of the biggest benefits of Google Cloud is networking. By default GCE instances in VPC's can communicate with all instances across zones and regions. This is a huge plus. On AWS, multi region involves setting up VPN and NAT instances. Not rocket science, but wasted brain cycles. Generally, with GCP setting up clusters that span three regions should provide ample high availability and most users don't need to deal with the multi cloud headaches. KISS. You can even get pretty good latency between regions if you setup North Carolina, South Carolina, and Iowa. Soon West Coast clusters will be possible between Oregon and Los Angels (region coming soon).
- sdrothrock 9y ago
  I may be completely off here, but isn't this due to their underlying architecture decisions? That is, AWS from the start has kept all regions completely separate, so that problems in one region do not influence another. But GCP has has issues with failure across regions IIRC.
  - outworlder 9y ago
    Having a software defined networking spanning across regions and failure cascades across regions are two different things. There's nothing preventing a vendor from presenting to you a single network, while they are actually distinct networks.
    - hossbeast 9y ago
      Having distinct networks in different regions encourages you to architect your application in a fault tolerant way.
      - kakwa_ 9y ago
        Or the contrary. In most cases there is something to synchronize between regions, like a replica of the data. With difficult interconnection of regions, it makes it somewhat harder to do, and it can easily end-up with "meh, AZs are good enough".
azurezyq 9y ago
One extra point for tracking VM bills: GCE bills are aggregated across instances. To get more detailed breakdown, you can apply labels to them and the bills will have label information attached in BQ. Alternatively, you can leverage GCE usage exports here: https://cloud.google.com/compute/docs/usage-export https://cloud.google.com/compute/docs/usage-export Which has per-instance per-day per-item usage data for GCE. Disclosure: I work for Google Cloud but not on GCE.
matt_wulfeck 9y ago
> As we investigated growth strategies outside of a single AZ, we realized a lot of the infrastructure changes we needed to make to accommodate multiple availability zones were the same changes we would need to make to accommodate multiple clouds. Maybe he author means multiple regions? Multi az is so easy. Everything works. Multi region is much harder.
user5994461 9y ago
>>> on AWS you have the option of getting dedicated machines which you can use to guarantee no two machines of yours run on the same underlying motherboard, or you can just use the largest instance type of its class (ex: r3.8xlarge) to probably have a whole motherboard to yourself. Not at all. Major mistake here. When you buy a dedicated instances on AWS, you reserve an entire server for yourself. All the VMs you buy subsequently will go to that same physical machine. In effect, your VMs are on the same motherboard and will all die together if the hardware experiences a failure. It's the exact opposite of what you wanted to do!
- e12e 9y ago
  I'm not sure what "dedicated machines" mean here - as far as I can tell from: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-hosts-overview.html#dedicated-hosts-dedicated-instances http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated... You can buy up to two of each type/location and schedule your vms to run on different physical hosts?
  - stephengillie 9y ago
    Effectively, Dedicated Instances are an Affinity policy. You're looking for an Anti-Affinity policy, which isn't common. Here's an article about Affinity and Anti-Affinity on OpenStack: https://techglimpse.com/affinity-anti-affinity-policies-openstack/ https://techglimpse.com/affinity-anti-affinity-policies-open...
- deleted 9y ago
  [deleted]
- ranman 9y ago
  I think two concepts are being conflated: Dedicated Instances: https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst... and Dedicated Hosts: https://aws.amazon.com/ec2/dedicated-hosts/ https://aws.amazon.com/ec2/dedicated-hosts/
  - 9y ago
vira28 9y ago
One thing that I liked with GCP is their recommendation for cost saving. I spun up a compute engine for a hobby project and within minutes they gave recommendations to reduce the instance size and how much i can save. I don't think AWS offers something like that. Correct me if I am wrong.
- mianosm 9y ago
  AWS has the trusted advisor, and it will offer to assist you in cost savings in terms of: * Idle Load Balancers * Underutilization of EBS volumes * Unassociated Elastic IP addresses * Idle RDS intsances * R53 latency resource record sets * etc...
  - BurritoAlPastor 9y ago
    Most of the Trusted Advisor checks are only available if you're on a Business or higher tier support plan. And those are now priced as a percentage of your monthly spend – not cheap.
    - illumin8 9y ago
      If you're running your business on any provider, wouldn't you want to make sure you had support?
      - user5994461 9y ago
        Not when the support is useless and not required.
- obulpathi 9y ago
  Even better are Google managed services (PubSub / Dataflow / Datastore), which scale up and down based on usage (cloud native products) and thus save money automatically compared to their equivalents in AWS (Kinesis / Kinesis Analytics / DynamoDB) which does not autoscale.
  - 9y ago
ad_hominem 9y ago
If any Google Cloud people are listening I wish you had an equivalent to AWS's Certificate Manager. Provisioning a TLS certificate which automatically renews for eternity (no out-of-band Let's Encrypt renewal process needed) and attaching it to a load balancer is so nice compared to Google Cloud's manual SslCertificate resource creation flow[1]. To a lesser extent, it's also nice registering domains within AWS and setting them to auto renew. Since Google Domains already exists, it would be neat to have this feature right inside Google Cloud. [1]: https://cloud.google.com/compute/docs/load-balancing/http/ssl-certificates https://cloud.google.com/compute/docs/load-balancing/http/ss...
- rmhrisk 9y ago
  We hear you, while I can't speak to future products and features I can say we understand there is room to improve the SSL provisioning and lifecycle management story in our products and we are making investments in that area.
- manigandham 9y ago
  It's in progress, star this issue to vote: https://issuetracker.google.com/issues/35900034 https://issuetracker.google.com/issues/35900034
manigandham 9y ago
When it comes to GCP: - They have Role Based Support plans which offer flat prices per subscribed user which is a much better model. [1] - Live migration for VMs mean host maintenance and failures are a minor issue, even if all your apps are running on the same machine. It's pretty much magical and when combined with persistent disks, effectively gives you a very reliable "machine" in the cloud. [2] 1. https://cloud.google.com/support/role-based/ https://cloud.google.com/support/role-based/ 2. https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati...
hobolord 9y ago
Great post! How difficult is it to switch from an AWS EC2 instance to the GCP version?

229 points by jbyers 9y ago

If AWS were to go to a per-minute billing cycle, they would be instantly more price-competitive with Google's offering. Or, to put it the other way around, those leftover minutes form a significant chunk of AWS's profit margin.

boulos 9y ago
Agreed, and I hope they do so! Though there would still be other things like the lower on-demand rates, custom shapes, networking that scales with shape (rather than being coarsely grouped), being able to attach SSD / GPUs semi-arbitrarily, and so on. For those that care, not having to pay up front for the best price is also a huge deal. You see the same thing in GCS vs S3 as well: Glacier and S3-IA have a few rounding up gotchas that catch many people out. All that said, I hope we all get to per-minute billing. Disclosure: I work on Google Cloud (but haven't talked to the Metamarkets folks)
obulpathi 9y ago
I don't think so. GCP's bill is usually about 50% of AWS's bill for same application, if you run it full hour (from my personal experiences and from several others as well: https://thehftguy.com/2016/11/18/google-cloud-is-50-cheaper-than-aws/ https://thehftguy.com/2016/11/18/google-cloud-is-50-cheaper-...). GCP has lot more cost saving features like seamless scalability, custom shapes, sustained discounts and so on. If you workloads span less than hour, GCP can offer more then 50% savings.
- boulos 9y ago
  He said "more" price-competitive :). I think we're all saying the same thing.
- ranman 9y ago
  I refuted some of the networking claims in that article previously (I work for AWS). Especially the bizarre claims that you have to get a C4.4xlarge for 1gpbs... The 220 mpbs network cap claim is just not true. Just run iperf3 on any aws instance to a GCE instance and you can see greater than 220mpbs.
  - kuschku 9y ago
    > Just run iperf3 on any aws instance to a GCE instance and you can see greater than 220mpbs. For how long is the question. Historically, it’s been considered common knowledge (might just be an urban legend) that AWS, even if you pay for more traffic, at some point just throttles you, the same way that they do with IO.
  - isatty 9y ago
    Honestly we all know that the small instances have terrible CPU that doesn't let you use the advertized 1Gbps anyway. Other than that, even if AWS let 1Gbps traffic go on for a while, you get throttled pretty quickly from my experience.
  - user5994461 9y ago
    Author of the quoted article here. The run of iperf refuted your refutation.

swozey 9y ago

Great thorough comparison and falls very into line with my experience. Definitely worth the read. Thanks!

throwaway0071 9y ago

Off Topic: it's frustrating that these companies spend quite a lot of time and money learning about the complexities of their infrastructure but when you're interviewing at such companies, you're expected to have answers for everything and a complete strategy for the cloud. /rant

mrg3_2013 9y ago

Nice post! I will be using it as a reference.

whatsmyhandle 9y ago

Very nice writeup! A nice, detailed read that was easy to understand. It seems to focus more on raw infrastructure (EC2 vs GCE) instead of each company's PaaS offerings. Obviously AWS has the front runner lead here, but would be super curious in a comparison of RDS vs. Cloud Spanner for instance. (pun unintentional, but then realized, and left in there)

vira28 9y ago
Did you mean to say AWS Aurora vs Cloud Spanner? Because, I don't think you can compare RDS vs Cloud Spanner. RDS is a managed for the most of the famous RDBMS out in the market (except Aurora). Cloud Spanner is a google proprietary db running only on GC.
outworlder 9y ago
This should be RDS vs Cloud SQL

nodesocket 9y ago

One of the biggest benefits of Google Cloud is networking. By default GCE instances in VPC's can communicate with all instances across zones and regions. This is a huge plus. On AWS, multi region involves setting up VPN and NAT instances. Not rocket science, but wasted brain cycles. Generally, with GCP setting up clusters that span three regions should provide ample high availability and most users don't need to deal with the multi cloud headaches. KISS. You can even get pretty good latency between regions if you setup North Carolina, South Carolina, and Iowa. Soon West Coast clusters will be possible between Oregon and Los Angels (region coming soon).

sdrothrock 9y ago
I may be completely off here, but isn't this due to their underlying architecture decisions? That is, AWS from the start has kept all regions completely separate, so that problems in one region do not influence another. But GCP has has issues with failure across regions IIRC.
- outworlder 9y ago
  Having a software defined networking spanning across regions and failure cascades across regions are two different things. There's nothing preventing a vendor from presenting to you a single network, while they are actually distinct networks.
  - hossbeast 9y ago
    Having distinct networks in different regions encourages you to architect your application in a fault tolerant way.
    - kakwa_ 9y ago
      Or the contrary. In most cases there is something to synchronize between regions, like a replica of the data. With difficult interconnection of regions, it makes it somewhat harder to do, and it can easily end-up with "meh, AZs are good enough".

azurezyq 9y ago

One extra point for tracking VM bills: GCE bills are aggregated across instances. To get more detailed breakdown, you can apply labels to them and the bills will have label information attached in BQ. Alternatively, you can leverage GCE usage exports here: https://cloud.google.com/compute/docs/usage-export https://cloud.google.com/compute/docs/usage-export Which has per-instance per-day per-item usage data for GCE. Disclosure: I work for Google Cloud but not on GCE.

matt_wulfeck 9y ago

> As we investigated growth strategies outside of a single AZ, we realized a lot of the infrastructure changes we needed to make to accommodate multiple availability zones were the same changes we would need to make to accommodate multiple clouds. Maybe he author means multiple regions? Multi az is so easy. Everything works. Multi region is much harder.

user5994461 9y ago

>>> on AWS you have the option of getting dedicated machines which you can use to guarantee no two machines of yours run on the same underlying motherboard, or you can just use the largest instance type of its class (ex: r3.8xlarge) to probably have a whole motherboard to yourself. Not at all. Major mistake here. When you buy a dedicated instances on AWS, you reserve an entire server for yourself. All the VMs you buy subsequently will go to that same physical machine. In effect, your VMs are on the same motherboard and will all die together if the hardware experiences a failure. It's the exact opposite of what you wanted to do!

e12e 9y ago
I'm not sure what "dedicated machines" mean here - as far as I can tell from: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-hosts-overview.html#dedicated-hosts-dedicated-instances http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated... You can buy up to two of each type/location and schedule your vms to run on different physical hosts?
- stephengillie 9y ago
  Effectively, Dedicated Instances are an Affinity policy. You're looking for an Anti-Affinity policy, which isn't common. Here's an article about Affinity and Anti-Affinity on OpenStack: https://techglimpse.com/affinity-anti-affinity-policies-openstack/ https://techglimpse.com/affinity-anti-affinity-policies-open...
deleted 9y ago
[deleted]
ranman 9y ago
I think two concepts are being conflated: Dedicated Instances: https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst... and Dedicated Hosts: https://aws.amazon.com/ec2/dedicated-hosts/ https://aws.amazon.com/ec2/dedicated-hosts/
- 9y ago

vira28 9y ago

One thing that I liked with GCP is their recommendation for cost saving. I spun up a compute engine for a hobby project and within minutes they gave recommendations to reduce the instance size and how much i can save. I don't think AWS offers something like that. Correct me if I am wrong.

mianosm 9y ago
AWS has the trusted advisor, and it will offer to assist you in cost savings in terms of: * Idle Load Balancers * Underutilization of EBS volumes * Unassociated Elastic IP addresses * Idle RDS intsances * R53 latency resource record sets * etc...
- BurritoAlPastor 9y ago
  Most of the Trusted Advisor checks are only available if you're on a Business or higher tier support plan. And those are now priced as a percentage of your monthly spend – not cheap.
  - illumin8 9y ago
    If you're running your business on any provider, wouldn't you want to make sure you had support?
    - user5994461 9y ago
      Not when the support is useless and not required.
obulpathi 9y ago
Even better are Google managed services (PubSub / Dataflow / Datastore), which scale up and down based on usage (cloud native products) and thus save money automatically compared to their equivalents in AWS (Kinesis / Kinesis Analytics / DynamoDB) which does not autoscale.
- 9y ago

ad_hominem 9y ago

If any Google Cloud people are listening I wish you had an equivalent to AWS's Certificate Manager. Provisioning a TLS certificate which automatically renews for eternity (no out-of-band Let's Encrypt renewal process needed) and attaching it to a load balancer is so nice compared to Google Cloud's manual SslCertificate resource creation flow[1]. To a lesser extent, it's also nice registering domains within AWS and setting them to auto renew. Since Google Domains already exists, it would be neat to have this feature right inside Google Cloud. [1]: https://cloud.google.com/compute/docs/load-balancing/http/ssl-certificates https://cloud.google.com/compute/docs/load-balancing/http/ss...

rmhrisk 9y ago
We hear you, while I can't speak to future products and features I can say we understand there is room to improve the SSL provisioning and lifecycle management story in our products and we are making investments in that area.
manigandham 9y ago
It's in progress, star this issue to vote: https://issuetracker.google.com/issues/35900034 https://issuetracker.google.com/issues/35900034

manigandham 9y ago

When it comes to GCP: - They have Role Based Support plans which offer flat prices per subscribed user which is a much better model. [1] - Live migration for VMs mean host maintenance and failures are a minor issue, even if all your apps are running on the same machine. It's pretty much magical and when combined with persistent disks, effectively gives you a very reliable "machine" in the cloud. [2] 1. https://cloud.google.com/support/role-based/ https://cloud.google.com/support/role-based/ 2. https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati...

hobolord 9y ago

Great post! How difficult is it to switch from an AWS EC2 instance to the GCP version?