6 ms·

Under the Hood of Amazon EC2 Container Service

136 points by werner 11y ago

justinsb 11y ago
I think it is interesting that AWS seems to be moving to consistent data-stores. Previously they were championing eventual consistency everywhere, even when it made for painful products (SimpleDB) or painful APIs (retry loops when using EC2 APIs).
- csears 11y ago
  My impression is that they have several different backend data stores, each with different trade-offs and consistency models, and they choose the one that makes sense for each app/service. The way Werner describes the ECS data store sounds very similar to Google's Megastore: To achieve concurrency control, we implemented Amazon ECS using one of Amazon’s core distributed systems primitives: a Paxos-based transactional journal based data store that keeps a record of every change made to a data entry. Any write to the data store is committed as a transaction in the journal with a specific order-based ID. The current value in a data store is the sum of all transactions made as recorded by the journal. Any read from the data store is only a snapshot in time of the journal. For a write to succeed, the write proposed must be the latest transaction since the last read.
- larsmak 11y ago
  I don't know, S3 and DynamoDB are both eventual consistent. And keeping in mind the CAP-theorem it makes sense. And I for one love SimpleDB - it's just that, simple. And great for prototyping (really cheap) and small production-loads. Often you just need a place to stick your data, scalability can be achieved to adding a caching layer.
  - justinsb 11y ago
    They have actually been making S3 _more_ consistent over time: in the newer regions you get e.g. read-your-writes for object creation. DynamoDB also supports consistency, though still defaults to eventual consistency if you prefer. In my mind, there's definitely a trend towards consistency here. I'd love to see an AWS blog post about the reasons behind this!
    - zorked 11y ago
      We should be glad. Eventual consistency is hard to reason about, particularly when its tradeoffs have to do with other people's systems...
    - dbarlett 11y ago
      US Standard now provides read-after-write consistency when accessed through the Northern Virginia endpoint [1]. [1] http://aws.amazon.com/s3/faqs/#What_data_consistency_model_does_Amazon_S3_employ http://aws.amazon.com/s3/faqs/#What_data_consistency_model_d...
  - x5n1 11y ago
    i think you can force consistency on DynamoDB for a price.
  - werner 11y ago
    You can choose eventual or fully consistent in DynamoDB. Given that full consistency comes at a higher cost (read from a quorum of replicas) we expose that cost to you. BTW nobody wants eventual consistency, it is a fact of live among many trade-offs. I would rather not expose it but it comes with other advantages ...
- rdtsc 11y ago
  What are some other examples. In general consistent vs available (or neither, that's possible too of course) is a trade-off and you'd want to pick one vs other depending on your business case.
  - justinsb 11y ago
    I know I'm bound to be proved wrong here, but I think _every_ product after the original set (EC2, S3, SimpleDB) has been consistent or now has a consistency option: the major ones are RDS, EBS, EFS, DynamoDB, RedShift, Elasticache, Route53, Kinesis, SES. Some of those APIs are sort of odd, admittedly, and could be covering up eventual consistency under the covers (Route53 in particular springs to mind there!) Edit: And S3 and SimpleDB now expose more consistency than they did at launch.
    - ranman 11y ago
      DynamoDB encourages eventual consistency by charging .5 for each read capacity unit consumed on an eventually consistent read.
      - justinsb 11y ago
        I consider consistency a bargain at twice the price :-)
- noelwelsh 11y ago
  I believe raw eventual consistency has failed as a programming API. I believe CRDTs in their many incarnations provide a great alternative but i) CRDTs are quite new and ii) they require a more complex API.
kylemathews 11y ago
How does this compare to Kubernetes?
- carterehsmith 11y ago
  I guess the better question would be "how does Kubernetes compares to this"? It's not like Kubernetes is a market leader or anything.
deleted 11y ago
[deleted]
2461001642 11y ago
I've heard that it just runs Mesos under the hood, how does this differ than me running Mesos on my own? Can I run Chronos or Marathon against it?
samstave 11y ago
The biggest issue that I had with ECS is that you need to initially create EC2 instances to put into your ECS cluster, using the AMI such that they have the ECS agent on them... BUT you have to prescale that cluster manually thereafter. In the task definition - it would be MUCH better if you could select the EC2 instance type you want, collect them in an ASG and have the task slicing scale the ASG accordingly. Right now - you have to manually determine the slice size for each container-to-EC2 and manually scale the ASG. Further, it was noted that ECS is actually NOT AZ aware and it will spread load over EC2 instances in the pool -- but it wont also balance the tasks across AZs.... So, its a fantastic version one... but these are some resiliency and scaling features that should have already been included.
- osipov 11y ago
  I've been using IBM Container Service. They have Docker containers running on bare metal servers with a free tier and a trial account for 30 days. With respect to clusters, you can set up a scalable group with min/max sizes and they'll take care of routing across the containers in the group.
  - nogox 11y ago
    How do they handle the isolation in a multi-tenant environment?
- Pyxl101 11y ago
  > Right now - you have to manually [...] scale the ASG. You have to manually scale the AutoScalingGroup, you say? :-) More seriously, what's the barrier to configuring the ASG to grow and shrink automatically?
  - samstave 11y ago
    Heh, yeah basically you can create an asg, but ecs has no trigger to scale it based on the number of task containers you attempt to launch on the cluster.... There is a workaround, apparently, which is to create a custom metric but aws says this has not been tested to their knowledge... So any asg will be "static" implicitly...
Rapzid 11y ago
A glaring gap currently is security. Per-container IAM roles would go a long way IMHO, but that still leaves "other" secret management which is a PITA. Other options such as kubernetes lack the AWS/ELB integration; all seem to be lacking a good security management model.
- TheIronYuppie 11y ago
  FWIW, Kubernetes provides its own load balancer, which you can put behind ELB. Other than that, Kubernetes works on AWS out of the box, with a one line setup. Full disclosure: I work at Google, on Kubernetes.
  - Rapzid 11y ago
    I'm not sure that's what I would call an integration... AWS provides for easy host management and elastic scaling traditionally through the integration of the ELB with autoscale groups, and now with life-cycle hooks. I'm not aware that kubernetes integrates with this stuff in any way or provides a sufficient alternative. Reading through the documentation I was not able to find information about connection draining on rolling updates, taking hosts out of service for maintenance/scaling/replacement, and so on. I am aware that kubernetes will run on AWS now and there is a guide for setting it up. However this really wasn't the point of my comment, which is that security for application secrets(and AWS API access) is currently a sore spot. It would be nice if kubernetes would adopt some of hashicorps stuff like consul, templates, and vault. Maybe that's too far up the container stack though and a popular bundling of technologies will appear.

136 points by werner 11y ago

I think it is interesting that AWS seems to be moving to consistent data-stores. Previously they were championing eventual consistency everywhere, even when it made for painful products (SimpleDB) or painful APIs (retry loops when using EC2 APIs).

csears 11y ago
My impression is that they have several different backend data stores, each with different trade-offs and consistency models, and they choose the one that makes sense for each app/service. The way Werner describes the ECS data store sounds very similar to Google's Megastore: To achieve concurrency control, we implemented Amazon ECS using one of Amazon’s core distributed systems primitives: a Paxos-based transactional journal based data store that keeps a record of every change made to a data entry. Any write to the data store is committed as a transaction in the journal with a specific order-based ID. The current value in a data store is the sum of all transactions made as recorded by the journal. Any read from the data store is only a snapshot in time of the journal. For a write to succeed, the write proposed must be the latest transaction since the last read.
larsmak 11y ago
I don't know, S3 and DynamoDB are both eventual consistent. And keeping in mind the CAP-theorem it makes sense. And I for one love SimpleDB - it's just that, simple. And great for prototyping (really cheap) and small production-loads. Often you just need a place to stick your data, scalability can be achieved to adding a caching layer.
- justinsb 11y ago
  They have actually been making S3 _more_ consistent over time: in the newer regions you get e.g. read-your-writes for object creation. DynamoDB also supports consistency, though still defaults to eventual consistency if you prefer. In my mind, there's definitely a trend towards consistency here. I'd love to see an AWS blog post about the reasons behind this!
  - zorked 11y ago
    We should be glad. Eventual consistency is hard to reason about, particularly when its tradeoffs have to do with other people's systems...
  - dbarlett 11y ago
    US Standard now provides read-after-write consistency when accessed through the Northern Virginia endpoint [1]. [1] http://aws.amazon.com/s3/faqs/#What_data_consistency_model_does_Amazon_S3_employ http://aws.amazon.com/s3/faqs/#What_data_consistency_model_d...
- x5n1 11y ago
  i think you can force consistency on DynamoDB for a price.
- werner 11y ago
  You can choose eventual or fully consistent in DynamoDB. Given that full consistency comes at a higher cost (read from a quorum of replicas) we expose that cost to you. BTW nobody wants eventual consistency, it is a fact of live among many trade-offs. I would rather not expose it but it comes with other advantages ...
rdtsc 11y ago
What are some other examples. In general consistent vs available (or neither, that's possible too of course) is a trade-off and you'd want to pick one vs other depending on your business case.
- justinsb 11y ago
  I know I'm bound to be proved wrong here, but I think _every_ product after the original set (EC2, S3, SimpleDB) has been consistent or now has a consistency option: the major ones are RDS, EBS, EFS, DynamoDB, RedShift, Elasticache, Route53, Kinesis, SES. Some of those APIs are sort of odd, admittedly, and could be covering up eventual consistency under the covers (Route53 in particular springs to mind there!) Edit: And S3 and SimpleDB now expose more consistency than they did at launch.
  - ranman 11y ago
    DynamoDB encourages eventual consistency by charging .5 for each read capacity unit consumed on an eventually consistent read.
    - justinsb 11y ago
      I consider consistency a bargain at twice the price :-)
noelwelsh 11y ago
I believe raw eventual consistency has failed as a programming API. I believe CRDTs in their many incarnations provide a great alternative but i) CRDTs are quite new and ii) they require a more complex API.

kylemathews 11y ago

How does this compare to Kubernetes?

carterehsmith 11y ago
I guess the better question would be "how does Kubernetes compares to this"? It's not like Kubernetes is a market leader or anything.

deleted 11y ago

[deleted]

2461001642 11y ago

I've heard that it just runs Mesos under the hood, how does this differ than me running Mesos on my own? Can I run Chronos or Marathon against it?

samstave 11y ago

The biggest issue that I had with ECS is that you need to initially create EC2 instances to put into your ECS cluster, using the AMI such that they have the ECS agent on them... BUT you have to prescale that cluster manually thereafter. In the task definition - it would be MUCH better if you could select the EC2 instance type you want, collect them in an ASG and have the task slicing scale the ASG accordingly. Right now - you have to manually determine the slice size for each container-to-EC2 and manually scale the ASG. Further, it was noted that ECS is actually NOT AZ aware and it will spread load over EC2 instances in the pool -- but it wont also balance the tasks across AZs.... So, its a fantastic version one... but these are some resiliency and scaling features that should have already been included.

osipov 11y ago
I've been using IBM Container Service. They have Docker containers running on bare metal servers with a free tier and a trial account for 30 days. With respect to clusters, you can set up a scalable group with min/max sizes and they'll take care of routing across the containers in the group.
- nogox 11y ago
  How do they handle the isolation in a multi-tenant environment?
Pyxl101 11y ago
> Right now - you have to manually [...] scale the ASG. You have to manually scale the AutoScalingGroup, you say? :-) More seriously, what's the barrier to configuring the ASG to grow and shrink automatically?
- samstave 11y ago
  Heh, yeah basically you can create an asg, but ecs has no trigger to scale it based on the number of task containers you attempt to launch on the cluster.... There is a workaround, apparently, which is to create a custom metric but aws says this has not been tested to their knowledge... So any asg will be "static" implicitly...

Rapzid 11y ago

A glaring gap currently is security. Per-container IAM roles would go a long way IMHO, but that still leaves "other" secret management which is a PITA. Other options such as kubernetes lack the AWS/ELB integration; all seem to be lacking a good security management model.

TheIronYuppie 11y ago
FWIW, Kubernetes provides its own load balancer, which you can put behind ELB. Other than that, Kubernetes works on AWS out of the box, with a one line setup. Full disclosure: I work at Google, on Kubernetes.
- Rapzid 11y ago
  I'm not sure that's what I would call an integration... AWS provides for easy host management and elastic scaling traditionally through the integration of the ELB with autoscale groups, and now with life-cycle hooks. I'm not aware that kubernetes integrates with this stuff in any way or provides a sufficient alternative. Reading through the documentation I was not able to find information about connection draining on rolling updates, taking hosts out of service for maintenance/scaling/replacement, and so on. I am aware that kubernetes will run on AWS now and there is a guide for setting it up. However this really wasn't the point of my comment, which is that security for application secrets(and AWS API access) is currently a sore spot. It would be nice if kubernetes would adopt some of hashicorps stuff like consul, templates, and vault. Maybe that's too far up the container stack though and a popular bundling of technologies will appear.