7 ms·

AWS Cloud Control API, a Uniform API to Access AWS and Third-Party Services

178 points by lukehoban 5y ago

Dunedan 5y ago
Luc van Donkersgoed wrote a blog post about inconsistent AWS APIs just a week ago [1]. Funny coincidence. [1]: https://www.lastweekinaws.com/blog/how-aws-dumps-the-mental-burden-of-inconsistent-apis-on-developers/ https://www.lastweekinaws.com/blog/how-aws-dumps-the-mental-...
k__ 5y ago
Pretty awesome. Now IaC tools like Pulumi will be always up-to-date.
- lukehoban 5y ago
  Indeed! And we just released a new AWS Native provider which builds on top of this new AWS Cloud Control API: https://www.pulumi.com/blog/announcing-aws-native/ https://www.pulumi.com/blog/announcing-aws-native/. I'm really excited about what Cloud Control API enables more broadly as well. I see this as a new high bar for cloud provider resource provisioning APIs, and it's an approach I hope to see more of across the industry going forward. Previously, to provision and manage a resource, you would need to find the REST APIs for that specific resource, and define your own model for what set of REST API calls constitute a logical Create, Read, Update or Delete operation. This work was done and re-done in dozens or hundreds of cloud provisioning tools. But AWS themselves had already built a model for doing this, and service teams at AWS were already defining their own well-defined resource model - the CloudFormation resource model for their services/features - it was just locked up behind CloudFormation templates as the only way to author and deploy resources using this resource model. With Cloud Control API, all of the value of that resource model, and the work by teams at AWS to cleanly define that resource model, is now unlocked to be used by any sort of provisioning tool or service - not just CloudFormation templates. At Pulumi, that has enabled us to build our new AWS Native provider on top of Cloud Control API, but it opens up tons of additional scenarios as well. Kudos to the AWS team for their work on this!
  - k__ 5y ago
    Is a native provider planned for Cloudflare?
  - mwcampbell 5y ago
    Now if we could just use cfn-signal (or something like it) when setting up an EC2 auto-scaling group without having to create a Cloudformation stack behind the scenes as the Pulumi awsx package does.
    - mdaniel 5y ago
      Pardon my density, but isn't that the function of the health checks built into the ASG, which offers both EC2 and ELB health checking to take a bogus instance out of service? Or do you mean "operation A --> ASG --> (wait $foo time for 'healthy') --> operation B" type thing?
- benatkin 5y ago
  It will be super easy to write alternatives to Pulumi.
  - fishnchips 5y ago
    Will it? Given how Pulumi can use Terraform providers, I'd say the hard work there isn't the connectors (which this change makes easy) but the framework and the orchestration bits. These aren't made any easier by AWS standardizing their APIs.
  - k__ 5y ago
    I wouldn't say super easy, but at least much easier than before.
ablekh 5y ago
I'm really excited that Pulumi supports this from day one: https://www.pulumi.com/blog/announcing-aws-native https://www.pulumi.com/blog/announcing-aws-native.
OJFord 5y ago
Why: aws cloudcontrol delete-resource \ --type-name AWS::Lambda::Function \ --identifier ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 and not: aws cloudcontrol delete-resource \ AWS::Lambda::Function \ ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 (and why 'cloudcontrol' for that matter, not top-level CRUD) Awscli loves its arguments. Several people are going to point me at autocompletions, tell me zsh has this really great feature where .. all fine. But why have this 'problem' to solve in the first place? I don't get it. For non-interactive use, sure, I don't care how verbose it is, it's even a sort of self-documenting feature. Is that it? They just really want all interaction via the web console?
- justicezyx 5y ago
  Other than the flag name, what's other significance of this difference? The flag seems trivial to me. Warp it in a shell script seems sufficient.
  - OJFord 5y ago
    What do you mean other than? I removed it, that is the (entire) difference. > Warp it in a shell script As I said, for non-interactive use of course I don't care. But trying to actually do/check/test anything with awscli interactively/'manually' is pretty painful IMO.
    - cheeze 5y ago
      I disagree here. The first example is much more self-describing and follows the same paradigm that the rest of the AWS CLI already follows. I'd prefer erbosity here rather than the AWS CLI trying to figure out what I'm doing.
      - jen20 5y ago
        I'm of the same opinion - I'd rather not have to consider the argument order, and instead just be able to append flags to an interactive command in any order I see fit as I'm refining queries.
arpinum 5y ago
Since this is from CloudFormation team, what can we expect about the performance vs calling the APIs directly? CloudFormation is not known for its speed.
- jfcllotr 5y ago
  ^ This. Surprised no one has enquired about the overhead of this abstraction layer.
cperciva 5y ago
There's an xkcd for that: https://xkcd.com/927/ https://xkcd.com/927/
- mooreds 5y ago
  Not sure why you are getting downvoted, this was the first thing that sprang to mind for me. "Oh, yes, another option." On top of terraform, pulumi, CloudFormation, the AWS SDKs, the CLI, and the CDK. I mean, I appreciate them trying to push things forward and realize all options aren't all equivalent, but where's the deprecation to make things simpler?
nprateem 5y ago
I don't know why hashicorp and pulumi are all smug about this. In one go it's destroyed half their moats.
- neximo64 5y ago
  People use Hashicorp and Pulumi not to use AWS by api but to provide options to move to other providers easily.
  - rad_gruchalski 5y ago
    If it only was so easy to move! Sure, one tool, one configuration format but it’s not so easy to simply change some configs and move. Even going for the provider lowest common denominator implies maintaining different configurations.
  - VWWHFSfQ 5y ago
    I don't think the (theoretical) ability to move to another infrastructure provider is the main motivation to use those tools. It might be a supporting motivation. But definitely not the real reason why they're using it. Hardly anybody actually changes their infrastructure provider anyway
  - dfinninger 5y ago
    In most places I've been, Terraform was used more for documentation than mobility. Sure, you don't need to unlearn Cloudformation if you switch to GCP, but you have a ton of AWS-specific stuff in your TF config that's going to need to be ported to the appropriate counterparts in your next cloud. But having a documented change history in Git for (almost) everything was much more useful than theoretical mobility.
  - streetcat1 5y ago
    I think that the main usage pattern for a tool like terraform is be able to version control your config, and , on the way, apply the git flow/git ops to infra changes (infra as code).
  - mdeeks 5y ago
m0zg 5y ago
Just buy Pulumi already and have them do it the way it's supposed to be. It's ridiculous that AWS documentation usually contains dozens of error prone manual steps.
- sebsto 5y ago
  Sorry to hear you found errors in the doc. Please report error to AWS using the button on the top of the page. https://aws.amazon.com/premiumsupport/knowledge-center/send-feedback-aws/ https://aws.amazon.com/premiumsupport/knowledge-center/send-...
nprateem 5y ago
I wonder how long before this becomes a standard like swagger. People will create a spec for a well-known endpoint and probably call it CURDL that allows APIs to be self documenting and automatically compatible with a general universal API client. No one will have to maintain their own clients any more and everyone will wonder why it took so long to think of this...
- robxu9 5y ago
  There's Smithy, which tries to help design API operations around a resource in a CRUDL-like fashion - I think that's what you're getting at? https://awslabs.github.io/smithy/ https://awslabs.github.io/smithy/
spullara 5y ago
It is super interesting to me that they now have essentially a declarative API in CloudFormation and now this corresponding imperative version with the same format. I'm not quite sure why you would need the imperative version. Anyone have any ideas? Couldn't Pulumi have just compiled their code into CloudFormation and deployed it?
- rhooke 5y ago
  Imperative version can fill more dynamic use cases, like spinning up entire servers in response to something (eg user wants to create a game server at the click of a button). Not something you can easily do with the declarative version.
  - spullara 5y ago
    That new game server could just be another cloudformation stack but I get your point.
  - dragonwriter 5y ago
    > Not something you can easily do with the declarative version. Its something that you can do easily with the imperative API for CloudFormation plus the declarative description of the stacks in CloudFormation.
- lukehoban 5y ago
  CloudFormation is really three separate things rolled up into one: (1) a resource model for AWS, (2) a deployment orchestration engine and (3) a syntax for specifying desired state as CloudFormation templates. Tools that compile to CloudFormation templates offer a way to access (2) directly - to use alternative front-end syntaxes but to still deploy via the CloudFormation orchestration engine. Tools like Pulumi (and Terraform) have their own deployment orchestration engines, which we believe offer many benefits - performance, secrets, components, transformations, aliases/refactoring, multi-cloud provisioning, and a lot more. Cloud Control API lets us (and others like us) access (1) directly, without having to use (2) or (3), and thus being able to still offer the full set of benefits of provisioning via Pulumi, along with the full set of benefits of a well-defined AWS resource model.
shaicoleman 5y ago
They only released support for it for the legacy AWS CLI v1, but not for AWS CLI v2.
- shaicoleman 5y ago
  Released for AWS CLI v2 as well now
Aeolun 5y ago
I don’t understand how this changes something? Now the type of resource you want to create and many different bodies depending on that will be inside the CreateResource instead of whatever function the specific AWS API had.
- jen20 5y ago
  The important difference is that the wait semantics are built in, meaning that each operation no longer needs custom code. This is a boon for things like Pulumi and Terraform providers.
  - kesor 5y ago
    I highly doubt that tools like Terraform/Pulumi will not have custom code for each type of resource. Each type still needs to have its own custom Body with specific data types, some mandatory, some optional. Even if TF supports Cloud Control as yet another resource, using it is going to be a huge mess of untyped/unchecked/undocumented bodies.
    - kall 5y ago
      Pulumi promises same-day support on their "Native" providers like this one, so I think they really don‘t.
    - jen20 5y ago
      The bodies are specified in a schematised way, so all of this code can be generated. This is not hypothetical - Pulumi’s Kubernetes support, Azure (“nextgen”) support and GCP Native providers are all built like this. It’s quite possible that the provided schema may need some augmentation for some resources (for example, semantic diff for IAM policies) to get the best plan-time experience, but this is a different world from having to hand write schema and CRUD logic for every resource type per the last 7 years of Terraform AWS support.
carlsborg 5y ago
Where is the sdk?
deleted 5y ago
[deleted]
renewiltord 5y ago
Interesting. One could see this being a way to operate cloud resources through a Kube API.
- kesor 5y ago
  You should take a look at CrossPlane for that kind of thing. They have been doing it for a couple of years.
joe200 5y ago
What does this change compared to the current status ? Terraform/Pulumi/AWS CDK already use AWS API. Is it just about having consistent API calls naming ?
- bbgm 5y ago
  lukehoban says it best in https://news.ycombinator.com/item?id=28711903 https://news.ycombinator.com/item?id=28711903 where he talks about the three things CloudFormation provides. The Cloud Control API is an API to the resource model. It unlocks the ability for AWS teams to contribute to the resource model without having to worry about the rest of what Cloudformation does. As a service owner interested in contributing to other systems like TF and Pulumi, now I just have to contribute to one thing. In the medium/long term it's going to be a lot more efficient way to evolve how we all use AWS resources in our favorite Infrastructure-as-code systems. Disclosure: Work at AWS (but not on the CloudFormation team)
dmarinus 5y ago
Hmm this would also be nice for ansible.
kesor 5y ago
Each API still needs to have its own custom body, with specific mandatory keys and values. The name of the verb is just another key and value ... so how is this different from existing APIs? Not different much. The only addition introduced here is the support for third-party CloudFormation resources. So the same AWS APIs can now be used to directly manipulate third party resources without using CloudFormation at all.
danw1979 5y ago
I feel like the more complex corners of the AWS resource model (e.g. offer-accept stuff between accounts, in TGW for example) are going to get over simplified behind this CRUDL API.
rawoke083600 5y ago
Yo more complexity ! They(AWS) should hire the UX team from "Hetzner Cloud Services" - sure it doesn't have the same features, but dammit it's super easy to navigate.
CSDude 5y ago
My use case is for mostly read-calls. I just tried it for DynamoDB Table, SQS Queue, SNS Topic, and EC2 Instance, Kinesis Stream. With these resources, only Kinesis supports List call. SQS Queue & SNS Topic does not even support READ. It might be useful for resource creation, but I would go nuts to discover what resource and what APIs are supported.

178 points by lukehoban 5y ago

Luc van Donkersgoed wrote a blog post about inconsistent AWS APIs just a week ago [1]. Funny coincidence. [1]: https://www.lastweekinaws.com/blog/how-aws-dumps-the-mental-burden-of-inconsistent-apis-on-developers/ https://www.lastweekinaws.com/blog/how-aws-dumps-the-mental-...

k__ 5y ago

Pretty awesome. Now IaC tools like Pulumi will be always up-to-date.

lukehoban 5y ago
Indeed! And we just released a new AWS Native provider which builds on top of this new AWS Cloud Control API: https://www.pulumi.com/blog/announcing-aws-native/ https://www.pulumi.com/blog/announcing-aws-native/. I'm really excited about what Cloud Control API enables more broadly as well. I see this as a new high bar for cloud provider resource provisioning APIs, and it's an approach I hope to see more of across the industry going forward. Previously, to provision and manage a resource, you would need to find the REST APIs for that specific resource, and define your own model for what set of REST API calls constitute a logical Create, Read, Update or Delete operation. This work was done and re-done in dozens or hundreds of cloud provisioning tools. But AWS themselves had already built a model for doing this, and service teams at AWS were already defining their own well-defined resource model - the CloudFormation resource model for their services/features - it was just locked up behind CloudFormation templates as the only way to author and deploy resources using this resource model. With Cloud Control API, all of the value of that resource model, and the work by teams at AWS to cleanly define that resource model, is now unlocked to be used by any sort of provisioning tool or service - not just CloudFormation templates. At Pulumi, that has enabled us to build our new AWS Native provider on top of Cloud Control API, but it opens up tons of additional scenarios as well. Kudos to the AWS team for their work on this!
- k__ 5y ago
  Is a native provider planned for Cloudflare?
- mwcampbell 5y ago
  Now if we could just use cfn-signal (or something like it) when setting up an EC2 auto-scaling group without having to create a Cloudformation stack behind the scenes as the Pulumi awsx package does.
  - mdaniel 5y ago
    Pardon my density, but isn't that the function of the health checks built into the ASG, which offers both EC2 and ELB health checking to take a bogus instance out of service? Or do you mean "operation A --> ASG --> (wait $foo time for 'healthy') --> operation B" type thing?
benatkin 5y ago
It will be super easy to write alternatives to Pulumi.
- fishnchips 5y ago
  Will it? Given how Pulumi can use Terraform providers, I'd say the hard work there isn't the connectors (which this change makes easy) but the framework and the orchestration bits. These aren't made any easier by AWS standardizing their APIs.
- k__ 5y ago
  I wouldn't say super easy, but at least much easier than before.

ablekh 5y ago

I'm really excited that Pulumi supports this from day one: https://www.pulumi.com/blog/announcing-aws-native https://www.pulumi.com/blog/announcing-aws-native.

OJFord 5y ago

Why: aws cloudcontrol delete-resource \ --type-name AWS::Lambda::Function \ --identifier ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 and not: aws cloudcontrol delete-resource \ AWS::Lambda::Function \ ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 (and why 'cloudcontrol' for that matter, not top-level CRUD) Awscli loves its arguments. Several people are going to point me at autocompletions, tell me zsh has this really great feature where .. all fine. But why have this 'problem' to solve in the first place? I don't get it. For non-interactive use, sure, I don't care how verbose it is, it's even a sort of self-documenting feature. Is that it? They just really want all interaction via the web console?

justicezyx 5y ago
Other than the flag name, what's other significance of this difference? The flag seems trivial to me. Warp it in a shell script seems sufficient.
- OJFord 5y ago
  What do you mean other than? I removed it, that is the (entire) difference. > Warp it in a shell script As I said, for non-interactive use of course I don't care. But trying to actually do/check/test anything with awscli interactively/'manually' is pretty painful IMO.
  - cheeze 5y ago
    I disagree here. The first example is much more self-describing and follows the same paradigm that the rest of the AWS CLI already follows. I'd prefer erbosity here rather than the AWS CLI trying to figure out what I'm doing.
    - jen20 5y ago
      I'm of the same opinion - I'd rather not have to consider the argument order, and instead just be able to append flags to an interactive command in any order I see fit as I'm refining queries.

arpinum 5y ago

Since this is from CloudFormation team, what can we expect about the performance vs calling the APIs directly? CloudFormation is not known for its speed.

jfcllotr 5y ago
^ This. Surprised no one has enquired about the overhead of this abstraction layer.

cperciva 5y ago

There's an xkcd for that: https://xkcd.com/927/ https://xkcd.com/927/

mooreds 5y ago
Not sure why you are getting downvoted, this was the first thing that sprang to mind for me. "Oh, yes, another option." On top of terraform, pulumi, CloudFormation, the AWS SDKs, the CLI, and the CDK. I mean, I appreciate them trying to push things forward and realize all options aren't all equivalent, but where's the deprecation to make things simpler?

nprateem 5y ago

I don't know why hashicorp and pulumi are all smug about this. In one go it's destroyed half their moats.

neximo64 5y ago
People use Hashicorp and Pulumi not to use AWS by api but to provide options to move to other providers easily.
- rad_gruchalski 5y ago
  If it only was so easy to move! Sure, one tool, one configuration format but it’s not so easy to simply change some configs and move. Even going for the provider lowest common denominator implies maintaining different configurations.
- VWWHFSfQ 5y ago
  I don't think the (theoretical) ability to move to another infrastructure provider is the main motivation to use those tools. It might be a supporting motivation. But definitely not the real reason why they're using it. Hardly anybody actually changes their infrastructure provider anyway
- dfinninger 5y ago
  In most places I've been, Terraform was used more for documentation than mobility. Sure, you don't need to unlearn Cloudformation if you switch to GCP, but you have a ton of AWS-specific stuff in your TF config that's going to need to be ported to the appropriate counterparts in your next cloud. But having a documented change history in Git for (almost) everything was much more useful than theoretical mobility.
- streetcat1 5y ago
  I think that the main usage pattern for a tool like terraform is be able to version control your config, and , on the way, apply the git flow/git ops to infra changes (infra as code).
- mdeeks 5y ago

m0zg 5y ago

Just buy Pulumi already and have them do it the way it's supposed to be. It's ridiculous that AWS documentation usually contains dozens of error prone manual steps.

sebsto 5y ago
Sorry to hear you found errors in the doc. Please report error to AWS using the button on the top of the page. https://aws.amazon.com/premiumsupport/knowledge-center/send-feedback-aws/ https://aws.amazon.com/premiumsupport/knowledge-center/send-...

nprateem 5y ago

I wonder how long before this becomes a standard like swagger. People will create a spec for a well-known endpoint and probably call it CURDL that allows APIs to be self documenting and automatically compatible with a general universal API client. No one will have to maintain their own clients any more and everyone will wonder why it took so long to think of this...

robxu9 5y ago
There's Smithy, which tries to help design API operations around a resource in a CRUDL-like fashion - I think that's what you're getting at? https://awslabs.github.io/smithy/ https://awslabs.github.io/smithy/

spullara 5y ago

It is super interesting to me that they now have essentially a declarative API in CloudFormation and now this corresponding imperative version with the same format. I'm not quite sure why you would need the imperative version. Anyone have any ideas? Couldn't Pulumi have just compiled their code into CloudFormation and deployed it?

rhooke 5y ago
Imperative version can fill more dynamic use cases, like spinning up entire servers in response to something (eg user wants to create a game server at the click of a button). Not something you can easily do with the declarative version.
- spullara 5y ago
  That new game server could just be another cloudformation stack but I get your point.
- dragonwriter 5y ago
  > Not something you can easily do with the declarative version. Its something that you can do easily with the imperative API for CloudFormation plus the declarative description of the stacks in CloudFormation.
lukehoban 5y ago
CloudFormation is really three separate things rolled up into one: (1) a resource model for AWS, (2) a deployment orchestration engine and (3) a syntax for specifying desired state as CloudFormation templates. Tools that compile to CloudFormation templates offer a way to access (2) directly - to use alternative front-end syntaxes but to still deploy via the CloudFormation orchestration engine. Tools like Pulumi (and Terraform) have their own deployment orchestration engines, which we believe offer many benefits - performance, secrets, components, transformations, aliases/refactoring, multi-cloud provisioning, and a lot more. Cloud Control API lets us (and others like us) access (1) directly, without having to use (2) or (3), and thus being able to still offer the full set of benefits of provisioning via Pulumi, along with the full set of benefits of a well-defined AWS resource model.

shaicoleman 5y ago

They only released support for it for the legacy AWS CLI v1, but not for AWS CLI v2.

shaicoleman 5y ago
Released for AWS CLI v2 as well now

Aeolun 5y ago

I don’t understand how this changes something? Now the type of resource you want to create and many different bodies depending on that will be inside the CreateResource instead of whatever function the specific AWS API had.

jen20 5y ago
The important difference is that the wait semantics are built in, meaning that each operation no longer needs custom code. This is a boon for things like Pulumi and Terraform providers.
- kesor 5y ago
  I highly doubt that tools like Terraform/Pulumi will not have custom code for each type of resource. Each type still needs to have its own custom Body with specific data types, some mandatory, some optional. Even if TF supports Cloud Control as yet another resource, using it is going to be a huge mess of untyped/unchecked/undocumented bodies.
  - kall 5y ago
    Pulumi promises same-day support on their "Native" providers like this one, so I think they really don‘t.
  - jen20 5y ago
    The bodies are specified in a schematised way, so all of this code can be generated. This is not hypothetical - Pulumi’s Kubernetes support, Azure (“nextgen”) support and GCP Native providers are all built like this. It’s quite possible that the provided schema may need some augmentation for some resources (for example, semantic diff for IAM policies) to get the best plan-time experience, but this is a different world from having to hand write schema and CRUD logic for every resource type per the last 7 years of Terraform AWS support.

carlsborg 5y ago

Where is the sdk?

deleted 5y ago

[deleted]

renewiltord 5y ago

Interesting. One could see this being a way to operate cloud resources through a Kube API.

kesor 5y ago
You should take a look at CrossPlane for that kind of thing. They have been doing it for a couple of years.

joe200 5y ago

What does this change compared to the current status ? Terraform/Pulumi/AWS CDK already use AWS API. Is it just about having consistent API calls naming ?

bbgm 5y ago
lukehoban says it best in https://news.ycombinator.com/item?id=28711903 https://news.ycombinator.com/item?id=28711903 where he talks about the three things CloudFormation provides. The Cloud Control API is an API to the resource model. It unlocks the ability for AWS teams to contribute to the resource model without having to worry about the rest of what Cloudformation does. As a service owner interested in contributing to other systems like TF and Pulumi, now I just have to contribute to one thing. In the medium/long term it's going to be a lot more efficient way to evolve how we all use AWS resources in our favorite Infrastructure-as-code systems. Disclosure: Work at AWS (but not on the CloudFormation team)

dmarinus 5y ago

Hmm this would also be nice for ansible.

kesor 5y ago

Each API still needs to have its own custom body, with specific mandatory keys and values. The name of the verb is just another key and value ... so how is this different from existing APIs? Not different much. The only addition introduced here is the support for third-party CloudFormation resources. So the same AWS APIs can now be used to directly manipulate third party resources without using CloudFormation at all.

danw1979 5y ago

I feel like the more complex corners of the AWS resource model (e.g. offer-accept stuff between accounts, in TGW for example) are going to get over simplified behind this CRUDL API.

rawoke083600 5y ago

Yo more complexity ! They(AWS) should hire the UX team from "Hetzner Cloud Services" - sure it doesn't have the same features, but dammit it's super easy to navigate.

CSDude 5y ago

My use case is for mostly read-calls. I just tried it for DynamoDB Table, SQS Queue, SNS Topic, and EC2 Instance, Kinesis Stream. With these resources, only Kinesis supports List call. SQS Queue & SNS Topic does not even support READ. It might be useful for resource creation, but I would go nuts to discover what resource and what APIs are supported.