14 ms·

> Don't you feel some anxiety given we've normalized committing encrypted secrets to git repos? Maybe I haven't worked at enough places, but... when has this e

> Don't you feel some anxiety given we've normalized committing encrypted secrets to git repos? Maybe I haven't worked at enough places, but... when has this ever been allowed/encouraged/normalized?

apopapo 1y ago
What's wrong with committing encrypted secrets? That's how I use `sops`.
- nodesocket 1y ago
  I would venture to guess the main concern is accidental commit of decrypted secrets.
  - thomasingalls 1y ago
    If a key gets compromised, the encrypted secrets are compromised forever, since you can't be sure all the git clones everywhere can be updated with a new encryption key. Not to mention how fiddly it is to edit git history.
    - NewJazz 1y ago
      But you can and should be rotating those secrets on some schedule regardless, and if you find out a key has been compromised you can immediately rotate the secrets.
    - dvtkrlbs 1y ago
      I would assume if you are committing encrypted secrets you would make sure they are rotatable
  - NewJazz 1y ago
    That can happen regardless.
- JeffMcCune 1y ago
  You can’t revoke, rotate, or audit access to them.
  - NewJazz 1y ago
    You can rotate them, although admittedly it can be more or less complicated depending on how your tooling compares to that of the secrets management system you are comparing against. You can't necessarily revoke a secret just because it is in Hashicorp Vault or AWS Secrets Manager. Revocation is a function of the system that provisions and/or uses the secret for authentication, not the system that stores the secret. E.g. if you generate a certificate and store the private key with vault or sops, the revocation procedure is identical and has nothing to do with the secrets storage system. Auditing access can be done coarsely by auditing access to the encryption key. Admittedly, this is an area where a more sophisticated system offers benefits. Although it isn't exactly iron clad -- a service may access a secret then leak or transfer the secret without that being visible in the audit log.
    - deleted 1y ago
      [deleted]
jasonthorsness 1y ago
Indeed, the only time I saw this was a decade ago for a temporary POC... not doing this is a good defense-in-depth practice even if the encryption is solid.
JohnMakin 1y ago
You'd be surprised. In the past I was on a big project at company with multi-billion $ revenue. They got caught with their pants down on an audit once because people would not only commit credentials into internal repositories, they were usually not encrypted at all, among other deeper issues. It sparked a multi-year long project of incorporating a secrets management service into the 1000+ repositories and services the company used. Found a loooooot of dead bodies, tons of people got fired during the process. After that experience I imagine this practice is fairly common - people, even smart developers, don't always seem to be able to comprehend the blast radius of some of these things. One of my favorite incidents during this clean-up effort was, the security team + my team had discovered a lot of DB credentials were just sitting on developer's local machines and basically nowhere else that made any kind of sense, and they'd hand them around as needed via email or message. So, we made tickets everywhere we found instances of this to migrate to the secret management platform. One lead developer with a privileged DB credential wrote a ticket that was basically: "Migrate secret to secret management platform" and in the info section, wrote the plaintext value of the key, inadvertently giving anyone with Jira read access to a sensitive production database. Even when it was explained to him I could tell he didn't really understand fully why that was silly. Why did he have it in the first place is a natural followup question, but these situations don't happen in a vacuum, there's usually a lot of other dumb stuff happening to even allow such a situation to unfold.
- NewJazz 1y ago
  Okay, but that sounds like a very different situation than a small shop where encrypted secrets are committed to one file per-repo, and keys and secrets are rotated regularly.
  - JohnMakin 1y ago
    Okay, in case it was missed, my salient point was that this behavior is very common and provided a ridiculous example as my evidence. I'm making no commentary on the practice itself (although I do think committing configs like secrets is really silly and anti-productive)
anbotero 1y ago
Wait, why are there so many skeptics in this thread? I have setup AWS + SOPS in several projects now, and the developers do not have access to the secrets themselves nor the encryption key (which is stored in AWS). Only once did we ever require to rollback a secret and that happened at AWS level, not the code’s. Also it happened within the key rotation period, so it was easy. For us it’s easier to track changes (not the value, but when it changes), easier to associate it with incidents.

> Don't you feel some anxiety given we've normalized committing encrypted secrets to git repos? Maybe I haven't worked at enough places, but... when has this ever been allowed/encouraged/normalized?

apopapo 1y ago

What's wrong with committing encrypted secrets? That's how I use `sops`.

nodesocket 1y ago
I would venture to guess the main concern is accidental commit of decrypted secrets.
- thomasingalls 1y ago
  If a key gets compromised, the encrypted secrets are compromised forever, since you can't be sure all the git clones everywhere can be updated with a new encryption key. Not to mention how fiddly it is to edit git history.
  - NewJazz 1y ago
    But you can and should be rotating those secrets on some schedule regardless, and if you find out a key has been compromised you can immediately rotate the secrets.
  - dvtkrlbs 1y ago
    I would assume if you are committing encrypted secrets you would make sure they are rotatable
- NewJazz 1y ago
  That can happen regardless.
JeffMcCune 1y ago
You can’t revoke, rotate, or audit access to them.
- NewJazz 1y ago
  You can rotate them, although admittedly it can be more or less complicated depending on how your tooling compares to that of the secrets management system you are comparing against. You can't necessarily revoke a secret just because it is in Hashicorp Vault or AWS Secrets Manager. Revocation is a function of the system that provisions and/or uses the secret for authentication, not the system that stores the secret. E.g. if you generate a certificate and store the private key with vault or sops, the revocation procedure is identical and has nothing to do with the secrets storage system. Auditing access can be done coarsely by auditing access to the encryption key. Admittedly, this is an area where a more sophisticated system offers benefits. Although it isn't exactly iron clad -- a service may access a secret then leak or transfer the secret without that being visible in the audit log.
  - deleted 1y ago
    [deleted]

jasonthorsness 1y ago

Indeed, the only time I saw this was a decade ago for a temporary POC... not doing this is a good defense-in-depth practice even if the encryption is solid.

JohnMakin 1y ago

You'd be surprised. In the past I was on a big project at company with multi-billion $ revenue. They got caught with their pants down on an audit once because people would not only commit credentials into internal repositories, they were usually not encrypted at all, among other deeper issues. It sparked a multi-year long project of incorporating a secrets management service into the 1000+ repositories and services the company used. Found a loooooot of dead bodies, tons of people got fired during the process. After that experience I imagine this practice is fairly common - people, even smart developers, don't always seem to be able to comprehend the blast radius of some of these things. One of my favorite incidents during this clean-up effort was, the security team + my team had discovered a lot of DB credentials were just sitting on developer's local machines and basically nowhere else that made any kind of sense, and they'd hand them around as needed via email or message. So, we made tickets everywhere we found instances of this to migrate to the secret management platform. One lead developer with a privileged DB credential wrote a ticket that was basically: "Migrate secret to secret management platform" and in the info section, wrote the plaintext value of the key, inadvertently giving anyone with Jira read access to a sensitive production database. Even when it was explained to him I could tell he didn't really understand fully why that was silly. Why did he have it in the first place is a natural followup question, but these situations don't happen in a vacuum, there's usually a lot of other dumb stuff happening to even allow such a situation to unfold.

NewJazz 1y ago
Okay, but that sounds like a very different situation than a small shop where encrypted secrets are committed to one file per-repo, and keys and secrets are rotated regularly.
- JohnMakin 1y ago
  Okay, in case it was missed, my salient point was that this behavior is very common and provided a ridiculous example as my evidence. I'm making no commentary on the practice itself (although I do think committing configs like secrets is really silly and anti-productive)

anbotero 1y ago

Wait, why are there so many skeptics in this thread? I have setup AWS + SOPS in several projects now, and the developers do not have access to the secrets themselves nor the encryption key (which is stored in AWS). Only once did we ever require to rollback a secret and that happened at AWS level, not the code’s. Also it happened within the key rotation period, so it was easy. For us it’s easier to track changes (not the value, but when it changes), easier to associate it with incidents.