6 ms·

Don't be so quick to dismiss this one. This is notable for several reasons, including (1) it wasn't his mistake, but rather that of a Visual Studio bug not foll

Don't be so quick to dismiss this one. This is notable for several reasons, including (1) it wasn't his mistake, but rather that of a Visual Studio bug not following his instructions to make a private repo on GitHub; (2) the speed in which this happened (minutes), and (3) it has useful analysis into some sorely lacking functionality in AWS that lets this continue to happen.

nathancahill 11y ago
I don't know. Even for private repos, it's bad practice to commit private keys to source control.
- zeveb 11y ago
  > Even for private repos, it's bad practice to commit private keys to source control. Is it, though? Committing them to the same repository as one's code lives in, sure, but committing them to a separate production-deploy repo seems okay to me (although I'd much prefer that private repo never to hit a centralised service like GitHub).
  - curun1r 11y ago
    > Is it, though? Yes, it is. Period. Full Stop. Don't ever check access keys into any repo, public, private or even self-hosted. AWS needs to do a better job of making you realize that access keys are like 100-year-old sticks of dynamite and should be handled with an equal amount of care considering they can cause a similar amount of damage. To their credit, they basically noted this when they changed the way access keys are handled in the command line tool[1]. Quoting: An important point is that the default location for the credentials file is a user directory. It's no longer part of a project file structure, such as an app.config file (.NET) or .properties file (Java). This can enhance security by allowing you to keep the credentials in a location that's accessible only to you, and it makes it less likely that you'll inadvertently upload credentials if you upload a project to a developer sharing site like GitHub. [1] http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-...
    - zeveb 11y ago
      > Don't ever check access keys into any repo, public, private or even self-hosted. So, how do you propose to handle version control of your exact production configuration? Do you think that recording one's production configuration is a valuable (or even useful) technique? Do you propose to have a single file with production configuration relying only on point-in-time backups for history? That gets us back to the bad old days… A local or self-hosted repo is 'a local location that's accessible only to youation that's accessible only to you.'
      - nathancahill 11y ago
        I use Ansible Vault if it's imperative to store keys in a repo, and I've been really happy with it. StackExchange has something similar called Blackbox[0]. I remember reading about a Ruby gem too, but I can't find it now. [0] https://github.com/StackExchange/blackbox https://github.com/StackExchange/blackbox
        zeveb 11y ago
        > https://github.com/StackExchange/blackbox https://github.com/StackExchange/blackbox That looks pretty cool, at least at first. I'll have to take a deeper dive into it. I like that it looks to be all shell scripts. Wish it were sh rather than bash, but c'est la vie.
      - curun1r 11y ago
        > So, how do you propose to handle version control of your exact production configuration? Version control is perfectly acceptable for that, but the key point is that access keys aren’t part of your production configuration. I personally don’t allow my team to have long-lived AWS access keys that have access to production. Production is only changed via an assumed role requiring MFA or an IAM instance role. If this kind of setup is too complex to create on your own, solutions like Vault (https://vaultproject.io https://vaultproject.io) can make it much easier to get right.
        zeveb 11y ago
gregmac 11y ago
Well, it is partially his mistake, which he acknowledges: > I am certainly not innocent here and some mistakes were made on my part. [...] To this end, having encrypted access keys or excluding configuration settings from GitHub would have prevented the AWS charges - and this is certainly the approach I would take from now on.
cptnbob 11y ago
AWS SDK for .Net supports named profiles for VS so your source check in only contains a profile name. If you hard coded your IAM or account keys or stuck them in the app.config you're simply doing it wrong. There is no excuse. It's all here: http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/DeveloperGuide/net-dg-config-creds.html http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/Developer... Also if you use a proper IAM profile locked down to specific resources then you wouldn't expose your entire account. The author stated he didn't use EC2 so why wasn't the key/secret pair an IAM account with a policy set for minimal access? What I'd worry more about is that the VS bug exposed private source code, data and proprietary intellectual property. We use github and I worry every day someone will public fork one of our repos by accident. That would be a grave fuck up but it's waiting to happen. We should have stuck with centrally controlled active-directory integrated SVN from a security perspective (even if it is a pain in the ass).
morder 11y ago
In this case it wasn't Visual studio but an extension created by Github that caused the problem.

nathancahill 11y ago

I don't know. Even for private repos, it's bad practice to commit private keys to source control.

zeveb 11y ago
> Even for private repos, it's bad practice to commit private keys to source control. Is it, though? Committing them to the same repository as one's code lives in, sure, but committing them to a separate production-deploy repo seems okay to me (although I'd much prefer that private repo never to hit a centralised service like GitHub).
- curun1r 11y ago
  > Is it, though? Yes, it is. Period. Full Stop. Don't ever check access keys into any repo, public, private or even self-hosted. AWS needs to do a better job of making you realize that access keys are like 100-year-old sticks of dynamite and should be handled with an equal amount of care considering they can cause a similar amount of damage. To their credit, they basically noted this when they changed the way access keys are handled in the command line tool[1]. Quoting: An important point is that the default location for the credentials file is a user directory. It's no longer part of a project file structure, such as an app.config file (.NET) or .properties file (Java). This can enhance security by allowing you to keep the credentials in a location that's accessible only to you, and it makes it less likely that you'll inadvertently upload credentials if you upload a project to a developer sharing site like GitHub. [1] http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-...
  - zeveb 11y ago
    > Don't ever check access keys into any repo, public, private or even self-hosted. So, how do you propose to handle version control of your exact production configuration? Do you think that recording one's production configuration is a valuable (or even useful) technique? Do you propose to have a single file with production configuration relying only on point-in-time backups for history? That gets us back to the bad old days… A local or self-hosted repo is 'a local location that's accessible only to youation that's accessible only to you.'
    - nathancahill 11y ago
      I use Ansible Vault if it's imperative to store keys in a repo, and I've been really happy with it. StackExchange has something similar called Blackbox[0]. I remember reading about a Ruby gem too, but I can't find it now. [0] https://github.com/StackExchange/blackbox https://github.com/StackExchange/blackbox
      - zeveb 11y ago
        > https://github.com/StackExchange/blackbox https://github.com/StackExchange/blackbox That looks pretty cool, at least at first. I'll have to take a deeper dive into it. I like that it looks to be all shell scripts. Wish it were sh rather than bash, but c'est la vie.
    - curun1r 11y ago
      > So, how do you propose to handle version control of your exact production configuration? Version control is perfectly acceptable for that, but the key point is that access keys aren’t part of your production configuration. I personally don’t allow my team to have long-lived AWS access keys that have access to production. Production is only changed via an assumed role requiring MFA or an IAM instance role. If this kind of setup is too complex to create on your own, solutions like Vault (https://vaultproject.io https://vaultproject.io) can make it much easier to get right.
      - zeveb 11y ago

gregmac 11y ago

Well, it is partially his mistake, which he acknowledges: > I am certainly not innocent here and some mistakes were made on my part. [...] To this end, having encrypted access keys or excluding configuration settings from GitHub would have prevented the AWS charges - and this is certainly the approach I would take from now on.

cptnbob 11y ago

AWS SDK for .Net supports named profiles for VS so your source check in only contains a profile name. If you hard coded your IAM or account keys or stuck them in the app.config you're simply doing it wrong. There is no excuse. It's all here: http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/DeveloperGuide/net-dg-config-creds.html http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/Developer... Also if you use a proper IAM profile locked down to specific resources then you wouldn't expose your entire account. The author stated he didn't use EC2 so why wasn't the key/secret pair an IAM account with a policy set for minimal access? What I'd worry more about is that the VS bug exposed private source code, data and proprietary intellectual property. We use github and I worry every day someone will public fork one of our repos by accident. That would be a grave fuck up but it's waiting to happen. We should have stuck with centrally controlled active-directory integrated SVN from a security perspective (even if it is a pain in the ass).

morder 11y ago

In this case it wasn't Visual studio but an extension created by Github that caused the problem.