6 ms·

I think it's good advice to not pass secrets through environment variables. Env vars leak a lot. Think php_info, Sentry, java vm dumps, etc. Also, env vars leak

I think it's good advice to not pass secrets through environment variables. Env vars leak a lot. Think php_info, Sentry, java vm dumps, etc. Also, env vars leak into sub-processes if you don't pay extra attention. Instead, read secrets from a vault or from a file-system from _inside_ your process. See also [1] (or [2] which discusses [1]). Dotnet does this pretty good with user secrets [3]. [1] https://blog.diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/ https://blog.diogomonica.com/2017/03/27/why-you-shouldnt-use... [2] https://security.stackexchange.com/questions/197784/is-it-unsafe-to-use-environmental-variables-for-secret-data https://security.stackexchange.com/questions/197784/is-it-un... [3] https://learn.microsoft.com/en-us/aspnet/core/security/app-secrets https://learn.microsoft.com/en-us/aspnet/core/security/app-s...

skissane 2y ago
> Instead, read secrets from a vault or from a file-system from _inside_ your process. I’ve never liked making secrets available on the filesystem. Lots of security vulnerabilities have turned up over the years that let an attacker read an arbitrary file. If retrieving secrets is a completely different API from normal file IO (e.g. inject a Unix domain socket into each container, and the software running on that container sends a request to that socket to get secrets), that is much less likely to happen.
- itchyouch 2y ago
  I used to store secrets in the FS, and was told the best practice was env vars. Now it's not env vars. What is it then?
  - andrewmcwatters 2y ago
    The filesystem is fine, but we really shouldn't be using .env files that get loaded into environment variables due to them leaking in a few different ways.
  - skissane 2y ago
    Well, I’d be the first to admit that we have a gap here, the solution that I personally would consider ideal doesn’t seem to actually exist, at least on the server-side. If we are running under something like K8S or Docker, then I think there should be some component that runs on the host, that provides access to secrets over a Unix domain secret, and then we mount that socket into each container. (The reason I say a Unix domain socket, is so then the component can use SCM_CREDENTIALS/SO_PEERCRED/etc to authenticate the containers). I’d also suggest not using HTTP, to reduce the potential impact of any SSRF vulnerabilities (although maybe that’s less of a risk given many HTTP clients don’t work with Unix domain sockets-or at least not without special config). (Can we pass memfd_secret using SCM_RIGHTS?) For desktop and native mobile, I think the best practice is to use the platform secret store (Keychain on macOS/iOS, Freedesktop Secret Service for desktop Linux, Android Keystore, Windows Credential Manager API, etc). But for server-side apps, those APIs generally aren’t available (Windows excepted). Server-side Linux often lacks desktop Linux components such as Freedesktop APIs (and even when they’re present, they aren’t the best fit for server-side use cases)
  - airtonix 2y ago
    [dead]
  - qweqwe14 2y ago
    There isn't a right answer. It's just that people don't understand that one doesn't provide any meaningful benefit over the other (in the context of storing secrets), but the security "experts" are always eager to claim "X is insecure, do Y instead, it's best practice btw" Unless I'm missing something, there are three scenarios where this comes up: 1. You are using a .env file to store secrets that will then be passed to the program through env vars. There's literally no difference in this case, you end up storing secrets in the FS anyway. 2. You are manually setting an env var with the secret when launching a program, e.g. SECRET=foo ./bar. The secret can still be easily obtained by inspecting /proc/PID/environ. It can't be read by other users, but so are the files in your user's directory (.env/secrets.json/whatever) 3. A program obtains the secret via some other means (network, user input, etc). You can still access /proc/PID/mem and extract the secret from process memory. So I'm assuming that what people really want is passing the secret to a program and having that secret not be readable by anything other than that program. The proper way to do this is using some OS-provided mechanism, like memfd_secret in Linux. The program can ask for the secret on startup via stdin, then store that secret in the special memory region designed for storing secrets.
    - delusional 2y ago
      You don't even need to roll your own solution with memfd. Linux already has keyrings[1] as a kernel concept. [1]: https://man7.org/linux/man-pages/man7/keyrings.7.html https://man7.org/linux/man-pages/man7/keyrings.7.html
    - merb 2y ago
      Also for 3. you just added another supply chain issue and another thing to maintain and keeping up to date, which is probably worse
    - macNchz 2y ago
      I’d wager that–in the context of web apps–over time there have been many more (or more readily exploitable) arbitrary file read/directory traversal/file inclusion vulnerabilities than remote code execution ones, so the preference for having secrets in memory as env vars may stem from that. You’re also probably not reading from /proc/self/mem without code execution either.
  - deleted 2y ago
    [deleted]
  - graemep 2y ago
    The problem with .env files is that you are doing both. You have a .env file that is in the same directory as your code and you just copy to to env vars at some point. This does not even meet the security principles that dotenv is supposed to implement! I think people are blindly following the advice "put secrets in env vars" without understanding that the point of it is to keep secrets outside files your app can read - because if you do a vulnerability or misconfiguration that lets people read those files leaks the secrets. What you can do is have environment vars set outside your code, preferably by another user. You do it in your init system or process supervisor. Someone mentioned passing them in from outside a docker container in another comment.
    - skissane 2y ago
      > people are blindly following the advice "put secrets in env vars" without understanding that the point of it is to keep secrets outside files your app can read - because if you do a vulnerability or misconfiguration that lets people read those files leaks the secrets. The problem with this is that, on Linux, the environment is a file, /proc/self/environ And yes, as has been mentioned in some other comments, the process memory is also a file /proc/self/mem - but it is a special file that can only be read using special procedures, whereas /proc/self/environ behaves much more like a normal file, so a lot of vulnerabilities that enable reading /proc/self/environ wouldn’t enable reading /proc/self/mem Technically one workaround on Linux is to not mount /proc (or at least not in your app’s container) - but doing that breaks a lot of things
    - anentropic 2y ago
      Yes... surely the point of .env file is as a shortcut to provide env vars for local development .env file shouldn't be used in production, env vars should be injected directly
      - graemep 2y ago
        I agree that is what should happen. That is not what dotenv does though: https://www.npmjs.com/package/dotenv#-documentation https://www.npmjs.com/package/dotenv#-documentation
  - csomar 2y ago
    Not using env vars is security through obscurity. If someone has ssh access to your container, it doesn't matter whether the secrets are on a file or on memory. The attacker has as much access as the app itself. On the other hand, using .env vars can leak in different ways like a developer mistakenly committing secrets to git or making this file available to the world wide web.
  - justinclift 2y ago
    Pull them from a remote host over ssh? :D
- deleted 2y ago
  [deleted]
- andrewmcwatters 2y ago
  God this is such a prime example of how we just don't do security well enough industry wide, and then you end up with weird stupid stuff like encryption being an enterprise paid feature. Secrets have to be somewhere. Environment variables are not a good place for them, but if you can't trust your filesystem to be secure, you're already screwed. There's no where else to go. The only remaining place is memory, and it's the same story. If you can't trust memory isolation, you're screwed. As a counterintuitive example from a former insider: virtually no one is storing secrets for financial software on an HSM. Almost no one does it, period.
  - TristanBall 2y ago
    Did they say why? It strikes me that those envs might be particularly prone to corporate inertia, ieg "the current way passed security audit, don't change it or we need to requalify" It's possibly also harder to rely on a HSM when your software is in a container? ( I'm guessing here tho )
    - nprateem 2y ago
      It's a useless, unproveable generalisation from a supposedly omniscient "insider". I know of at least one finance organisation using HSM as you'd expect.
      - skissane 2y ago
        And I know non-finance orgs using HSM to protect encryption keys used to encrypt PII
      - andrewmcwatters 2y ago
        Yeah, you don't have to trust me, there are plenty of software engineers working in finance who can tell you the same. Or they're using outdated ciphers, or they're storing information in plaintext or in logs, or they have no security playbooks. It's irrelevant to me whether you believe it, it's happening today, and it happens with some of the top financial institutions and their subsidiaries and it's the same bureaucratic nonsense to move those teams to do something about it like it is anywhere else.
bhawks 2y ago
Secrets are nasty and there are tradeoffs in every direction. Environment vars propagate from process to process _by design_ and generally last the entire process(es) lifetime. They are observable from many os tools unless you've hardened your config and they will appear in core files etc. Secrets imply scope and lifetime - so env variables feel very at odds. Alternatively Env variables are nearly perfect for config for the same reasons that they are concerning for secrets. Tl/Dr; in low stakes environments the fact that secrets are a special type of config means you will see it being used with env vars which are great for most configs but are poor for secrets. And frankly if you can stomach the risks, it is not that bad. Storing secrets on the filesystem - you immediately need to answer where on the filesystem and how to restrict access (and are your rules being followed). Is your media encrypted at rest? Do you have se Linux configured? Are you sure the secrets are cleaned after you no longer need them? Retrieving secrets or elevated perms via sockets / local ipc have very similar problems (but perhaps at this point your compartmentalizing all the secrets into a centralized, local point). A step beyond this are secrets that are locked behind cloud key management APIs or systems like spiffe/spire. At this point you still have to tackle workload identity, which is also a very nuanced problem. With secrets every solution has challenges and there the only clear answer is to have a threat model and design an architecture and appropriate mitigations that let you feel comfortable while acknowledging the cost, user, developer and operator experience balancing act.
jillesvangurp 2y ago
Disagree here. Basically if you use docker (which for most of the stuff you mention, you should), environment variables are pretty much how you configure your docker containers and a lot of sever software packaged up as docker containers expects to be configured this way. Building a lot of assumptions into your containers about where and how they are being deployed kind of defeats the point of using containers. You should inject configuration, including secrets, from the outside. The right time to access secret stores is just before you start the container as part of the deploy process or vm startup in cloud environments. And then you use environment variables to pass the information onto the container. Of course that does make some assumptions about the environment where you run your containers not being compromised. But then if that assumption breaks you are in big trouble anyway. Of course this tool is designed for developer machines and for that it seems useful. But I hope to never find this in a production environment.
- vitalysh 2y ago
  inject those variables via mount?
  - wegfawefgawefg 2y ago
    When you launch the docker containers you can pass in process env vars or do it via file. Nowadays people do this via kubernetes config yamls, which passes env to docker. Or rather they used to. Most people now use Helm charts which pass in the env to k8 yaml which pass them to docker. But then they feel its not secure enough... so a lot of people have the env split halfway between github actions secrets, and amazon secrets. The yaml for your github action config sends aws secret uri to the runner, which runs cdk which grabs the aws secret, and passes that to helm which makes k8 yamls, which passes the env to docker, which passes it to the process. Then I killed myself and was reborn. Now I just use an env file.
- jen20 2y ago
  > The right time to access secret stores is just before you start the container as part of the deploy process or vm startup in cloud environments. So how do you rotate secrets without bouncing app servers..?!
darkwater 2y ago
Remember we are mainly talking about dev envs here. If you put the secret key in a file...where do you put the file? In a common location for all the dotenv instances? One per dotenv instance? What if people start putting it as a dotfile in the same project directory?

skissane 2y ago

> Instead, read secrets from a vault or from a file-system from _inside_ your process. I’ve never liked making secrets available on the filesystem. Lots of security vulnerabilities have turned up over the years that let an attacker read an arbitrary file. If retrieving secrets is a completely different API from normal file IO (e.g. inject a Unix domain socket into each container, and the software running on that container sends a request to that socket to get secrets), that is much less likely to happen.

itchyouch 2y ago
I used to store secrets in the FS, and was told the best practice was env vars. Now it's not env vars. What is it then?
- andrewmcwatters 2y ago
  The filesystem is fine, but we really shouldn't be using .env files that get loaded into environment variables due to them leaking in a few different ways.
- skissane 2y ago
  Well, I’d be the first to admit that we have a gap here, the solution that I personally would consider ideal doesn’t seem to actually exist, at least on the server-side. If we are running under something like K8S or Docker, then I think there should be some component that runs on the host, that provides access to secrets over a Unix domain secret, and then we mount that socket into each container. (The reason I say a Unix domain socket, is so then the component can use SCM_CREDENTIALS/SO_PEERCRED/etc to authenticate the containers). I’d also suggest not using HTTP, to reduce the potential impact of any SSRF vulnerabilities (although maybe that’s less of a risk given many HTTP clients don’t work with Unix domain sockets-or at least not without special config). (Can we pass memfd_secret using SCM_RIGHTS?) For desktop and native mobile, I think the best practice is to use the platform secret store (Keychain on macOS/iOS, Freedesktop Secret Service for desktop Linux, Android Keystore, Windows Credential Manager API, etc). But for server-side apps, those APIs generally aren’t available (Windows excepted). Server-side Linux often lacks desktop Linux components such as Freedesktop APIs (and even when they’re present, they aren’t the best fit for server-side use cases)
- airtonix 2y ago
  [dead]
- qweqwe14 2y ago
  There isn't a right answer. It's just that people don't understand that one doesn't provide any meaningful benefit over the other (in the context of storing secrets), but the security "experts" are always eager to claim "X is insecure, do Y instead, it's best practice btw" Unless I'm missing something, there are three scenarios where this comes up: 1. You are using a .env file to store secrets that will then be passed to the program through env vars. There's literally no difference in this case, you end up storing secrets in the FS anyway. 2. You are manually setting an env var with the secret when launching a program, e.g. SECRET=foo ./bar. The secret can still be easily obtained by inspecting /proc/PID/environ. It can't be read by other users, but so are the files in your user's directory (.env/secrets.json/whatever) 3. A program obtains the secret via some other means (network, user input, etc). You can still access /proc/PID/mem and extract the secret from process memory. So I'm assuming that what people really want is passing the secret to a program and having that secret not be readable by anything other than that program. The proper way to do this is using some OS-provided mechanism, like memfd_secret in Linux. The program can ask for the secret on startup via stdin, then store that secret in the special memory region designed for storing secrets.
  - delusional 2y ago
    You don't even need to roll your own solution with memfd. Linux already has keyrings[1] as a kernel concept. [1]: https://man7.org/linux/man-pages/man7/keyrings.7.html https://man7.org/linux/man-pages/man7/keyrings.7.html
  - merb 2y ago
    Also for 3. you just added another supply chain issue and another thing to maintain and keeping up to date, which is probably worse
  - macNchz 2y ago
    I’d wager that–in the context of web apps–over time there have been many more (or more readily exploitable) arbitrary file read/directory traversal/file inclusion vulnerabilities than remote code execution ones, so the preference for having secrets in memory as env vars may stem from that. You’re also probably not reading from /proc/self/mem without code execution either.
- deleted 2y ago
  [deleted]
- graemep 2y ago
  The problem with .env files is that you are doing both. You have a .env file that is in the same directory as your code and you just copy to to env vars at some point. This does not even meet the security principles that dotenv is supposed to implement! I think people are blindly following the advice "put secrets in env vars" without understanding that the point of it is to keep secrets outside files your app can read - because if you do a vulnerability or misconfiguration that lets people read those files leaks the secrets. What you can do is have environment vars set outside your code, preferably by another user. You do it in your init system or process supervisor. Someone mentioned passing them in from outside a docker container in another comment.
  - skissane 2y ago
    > people are blindly following the advice "put secrets in env vars" without understanding that the point of it is to keep secrets outside files your app can read - because if you do a vulnerability or misconfiguration that lets people read those files leaks the secrets. The problem with this is that, on Linux, the environment is a file, /proc/self/environ And yes, as has been mentioned in some other comments, the process memory is also a file /proc/self/mem - but it is a special file that can only be read using special procedures, whereas /proc/self/environ behaves much more like a normal file, so a lot of vulnerabilities that enable reading /proc/self/environ wouldn’t enable reading /proc/self/mem Technically one workaround on Linux is to not mount /proc (or at least not in your app’s container) - but doing that breaks a lot of things
  - anentropic 2y ago
    Yes... surely the point of .env file is as a shortcut to provide env vars for local development .env file shouldn't be used in production, env vars should be injected directly
    - graemep 2y ago
      I agree that is what should happen. That is not what dotenv does though: https://www.npmjs.com/package/dotenv#-documentation https://www.npmjs.com/package/dotenv#-documentation
- csomar 2y ago
  Not using env vars is security through obscurity. If someone has ssh access to your container, it doesn't matter whether the secrets are on a file or on memory. The attacker has as much access as the app itself. On the other hand, using .env vars can leak in different ways like a developer mistakenly committing secrets to git or making this file available to the world wide web.
- justinclift 2y ago
  Pull them from a remote host over ssh? :D
deleted 2y ago
[deleted]
andrewmcwatters 2y ago
God this is such a prime example of how we just don't do security well enough industry wide, and then you end up with weird stupid stuff like encryption being an enterprise paid feature. Secrets have to be somewhere. Environment variables are not a good place for them, but if you can't trust your filesystem to be secure, you're already screwed. There's no where else to go. The only remaining place is memory, and it's the same story. If you can't trust memory isolation, you're screwed. As a counterintuitive example from a former insider: virtually no one is storing secrets for financial software on an HSM. Almost no one does it, period.
- TristanBall 2y ago
  Did they say why? It strikes me that those envs might be particularly prone to corporate inertia, ieg "the current way passed security audit, don't change it or we need to requalify" It's possibly also harder to rely on a HSM when your software is in a container? ( I'm guessing here tho )
  - nprateem 2y ago
    It's a useless, unproveable generalisation from a supposedly omniscient "insider". I know of at least one finance organisation using HSM as you'd expect.
    - skissane 2y ago
      And I know non-finance orgs using HSM to protect encryption keys used to encrypt PII
    - andrewmcwatters 2y ago
      Yeah, you don't have to trust me, there are plenty of software engineers working in finance who can tell you the same. Or they're using outdated ciphers, or they're storing information in plaintext or in logs, or they have no security playbooks. It's irrelevant to me whether you believe it, it's happening today, and it happens with some of the top financial institutions and their subsidiaries and it's the same bureaucratic nonsense to move those teams to do something about it like it is anywhere else.

bhawks 2y ago

Secrets are nasty and there are tradeoffs in every direction. Environment vars propagate from process to process _by design_ and generally last the entire process(es) lifetime. They are observable from many os tools unless you've hardened your config and they will appear in core files etc. Secrets imply scope and lifetime - so env variables feel very at odds. Alternatively Env variables are nearly perfect for config for the same reasons that they are concerning for secrets. Tl/Dr; in low stakes environments the fact that secrets are a special type of config means you will see it being used with env vars which are great for most configs but are poor for secrets. And frankly if you can stomach the risks, it is not that bad. Storing secrets on the filesystem - you immediately need to answer where on the filesystem and how to restrict access (and are your rules being followed). Is your media encrypted at rest? Do you have se Linux configured? Are you sure the secrets are cleaned after you no longer need them? Retrieving secrets or elevated perms via sockets / local ipc have very similar problems (but perhaps at this point your compartmentalizing all the secrets into a centralized, local point). A step beyond this are secrets that are locked behind cloud key management APIs or systems like spiffe/spire. At this point you still have to tackle workload identity, which is also a very nuanced problem. With secrets every solution has challenges and there the only clear answer is to have a threat model and design an architecture and appropriate mitigations that let you feel comfortable while acknowledging the cost, user, developer and operator experience balancing act.

jillesvangurp 2y ago

Disagree here. Basically if you use docker (which for most of the stuff you mention, you should), environment variables are pretty much how you configure your docker containers and a lot of sever software packaged up as docker containers expects to be configured this way. Building a lot of assumptions into your containers about where and how they are being deployed kind of defeats the point of using containers. You should inject configuration, including secrets, from the outside. The right time to access secret stores is just before you start the container as part of the deploy process or vm startup in cloud environments. And then you use environment variables to pass the information onto the container. Of course that does make some assumptions about the environment where you run your containers not being compromised. But then if that assumption breaks you are in big trouble anyway. Of course this tool is designed for developer machines and for that it seems useful. But I hope to never find this in a production environment.

vitalysh 2y ago
inject those variables via mount?
- wegfawefgawefg 2y ago
  When you launch the docker containers you can pass in process env vars or do it via file. Nowadays people do this via kubernetes config yamls, which passes env to docker. Or rather they used to. Most people now use Helm charts which pass in the env to k8 yaml which pass them to docker. But then they feel its not secure enough... so a lot of people have the env split halfway between github actions secrets, and amazon secrets. The yaml for your github action config sends aws secret uri to the runner, which runs cdk which grabs the aws secret, and passes that to helm which makes k8 yamls, which passes the env to docker, which passes it to the process. Then I killed myself and was reborn. Now I just use an env file.
jen20 2y ago
> The right time to access secret stores is just before you start the container as part of the deploy process or vm startup in cloud environments. So how do you rotate secrets without bouncing app servers..?!

darkwater 2y ago

Remember we are mainly talking about dev envs here. If you put the secret key in a file...where do you put the file? In a common location for all the dotenv instances? One per dotenv instance? What if people start putting it as a dotfile in the same project directory?