7 ms·

Cloud Run quietly swaps HOME env var in Docker

73 points by chanind 5y ago

yzmtf2008 5y ago
The solution here is to use `$XDG_CACHE_HOME` instead of `$HOME/.cache`, which may not be where a user wants their cache to live: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html https://specifications.freedesktop.org/basedir-spec/basedir-...
- fwip 5y ago
  Is that variable set to a consistent value at both build & run in Cloud Run?
- remram 5y ago
  $XDG_CACHE_HOME is usually not set, it is an override and software should default to $HOME/.cache. As per your link: > If $XDG_CACHE_HOME is either not set or empty, a default equal to $HOME/.cache should be used. So even if Huggingface is aware of that variable (and it probably is) that won't help at all.
  - jessaustin 5y ago
    This has always seemed sort of passive-aggressive on the part of the spec. "No, memorizing our various custom env var names is not sufficient. Developers must also implement our custom logic, and immediately update whenever the logic changes!" That seems a bit much to ask. Why not just trust the user, who will set the env vars if she really wants to use this scheme and who will do something else if she wants something else?
    - lights0123 5y ago
      > will do something else if she wants something else How should that be communicated by the user? A prompt when opening every application asking where you'd like the data, config, state, and cache dirs to live? And if so, how would the application figure out where that config is stored? These env vars are explicitly only (with the exception of XDG_RUNTIME_DIR) for when the user cares to override them.
      - jessaustin 5y ago
        Somehow lots of software functioned before the advent of XDG. Lots of software that knows nothing about XDG still functions. These env vars are explicitly only (with the exception of XDG_RUNTIME_DIR) for when the user cares to override them. Yeah, sure, that's what the spec says, and that's what is passive-aggressive. There is a complicated system and it is opt-out, which is considered abusive whenever anyone else does it. I actually like XDG, and I use it even when I have to specifically tell packages to do so. I have contributed patches to unrelated software for XDG support. However, I don't agree with the common online strategy of shitting on software that doesn't follow a spec that is deliberately more complicated than it needs to be.
        remram 5y ago
        How is an optional override "opt-out"? Who is "shitting on software"? This rant is inscrutable.
  - stormbrew 5y ago
    It does appear to[1], and it does help because you can set XDG_CACHE_HOME in your dockerfile so it'll be the same across all runs. I think that's what the GP was getting at. [1] https://huggingface.co/transformers/v4.3.3/installation.html?highlight=transformers_cache https://huggingface.co/transformers/v4.3.3/installation.html...
- deleted 5y ago
  [deleted]
reilly3000 5y ago
As an alternative, don’t use root for Dockerfiles. Ever. It only takes a few lines to create a user, group, and use said user, and closes a whole class of security issues.
- sjtindell 5y ago
  Closes security loopholes, increases operational burden (if only slightly). A tale as old as time.
  - wussboy 5y ago
    Agree. It only takes a few lines. But how long does it take to learn what those lines are?
- chanind 5y ago
  True! That would have also been a smart move
- Arnavion 5y ago
  Are you sure Cloud Run only overrides $HOME when the container runs as root? Because if it always overrides $HOME then what user the container runs as is meaningless (for this particular issue).
  - reilly3000 5y ago
    The problem is with root vs all other users is that it’s $HOME is literally /root/ while other users have their $HOME directory in /home/username/ ~ is a BASH tilde expansion for $HOME. So in this case Cloud Run would have been looking at /home/root/.cache/ which doesn’t exist (/root/.cache/ is what was being built), whereas another user would have /home/username/.cache/ and run as expected. PS. I was initially going to call ~ an alias, but I checked myself and found it’s actually considered a BASH tilde expansion. While ~ alone operates as an alias, I learned there are all kinds of other uses for variations of ~ which hopefully someone will find useful: http://www.gnu.org/software/bash/manual/html_node/Tilde-Expansion.html#Tilde-Expansion http://www.gnu.org/software/bash/manual/html_node/Tilde-Expa...
justin_oaks 5y ago
For those not familiar with Cloud Run, it's a Google Cloud service that allows you to run Docker containers without managing the server it runs on. https://cloud.google.com/run https://cloud.google.com/run
cmeacham98 5y ago
Does Huggingface not have a way to override the cache directory? I always specify directories like this manually when using docker specifically to avoid issues like the one in TFA.
- manojlds 5y ago
  They do - https://huggingface.co/transformers/v4.3.3/installation.html?highlight=transformers_cache#caching-models https://huggingface.co/transformers/v4.3.3/installation.html...
steren 5y ago
Cloud Run PM here. Sorry for the inconvenience. This is an issue for which we have a fix, but the fix still needs to roll out (internal bug 154368323 for googlers reading, we're still evaluating the best rollout strategy). Until then, I'll add it to https://cloud.google.com/run/docs/issues https://cloud.google.com/run/docs/issues.
- renewiltord 5y ago
  Cool beans. Thanks for the response.
- hn_throwaway_99 5y ago
  I only say this half in jest: Google, please give this person a raise. I'm not sure I've ever seen a Google PM be this up front and straightforward with a future roadmap item, even in HN, where I more commonly see more direct communications by PMs or engineers from various companies. While you're at it, if there is a way you can get the Google Identity Platform PM to let us know when some form of multi-factor auth besides SMS will be supported, that would be great!
  - NicoJuicy 5y ago
    I feel that employees from big Corp sometimes can't do good. If they respond here, it should be as you mentioned. They are monitoring media on mentions. If they don't, people can't fix their issues ( probably ) as fast. But responding here gives the observing effect of having a higher priority. Which is sometimes "ridiculed" ( don't know a better describing term) Either way, thanks to the PM for an update about this.
    - KptMarchewa 5y ago
      I guess the main problem is the second part of the subOP comment: >While you're at it, if there is a way you can get the Google Identity Platform PM to let us know when some form of multi-factor auth besides SMS will be supported, that would be great! When they respond they get inundated with random requests and ailments that they have exactly zero control on.
wodenokoto 5y ago
That sucks. My only advice is, when using this or cloud functions always start by creating a function/image that prints all env variables. For cloud functions these changes tremendously between Python version. Also logging changes completely between some Python versions, to the point where upping the runtime causes logs to stop being saved on cloud logging.
- GrayShade 5y ago
  But if you have secrets in your environment variables, make sure to filter them out.

73 points by chanind 5y ago

yzmtf2008 5y ago

The solution here is to use `$XDG_CACHE_HOME` instead of `$HOME/.cache`, which may not be where a user wants their cache to live: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html https://specifications.freedesktop.org/basedir-spec/basedir-...

fwip 5y ago
Is that variable set to a consistent value at both build & run in Cloud Run?
remram 5y ago
$XDG_CACHE_HOME is usually not set, it is an override and software should default to $HOME/.cache. As per your link: > If $XDG_CACHE_HOME is either not set or empty, a default equal to $HOME/.cache should be used. So even if Huggingface is aware of that variable (and it probably is) that won't help at all.
- jessaustin 5y ago
  This has always seemed sort of passive-aggressive on the part of the spec. "No, memorizing our various custom env var names is not sufficient. Developers must also implement our custom logic, and immediately update whenever the logic changes!" That seems a bit much to ask. Why not just trust the user, who will set the env vars if she really wants to use this scheme and who will do something else if she wants something else?
  - lights0123 5y ago
    > will do something else if she wants something else How should that be communicated by the user? A prompt when opening every application asking where you'd like the data, config, state, and cache dirs to live? And if so, how would the application figure out where that config is stored? These env vars are explicitly only (with the exception of XDG_RUNTIME_DIR) for when the user cares to override them.
    - jessaustin 5y ago
      Somehow lots of software functioned before the advent of XDG. Lots of software that knows nothing about XDG still functions. These env vars are explicitly only (with the exception of XDG_RUNTIME_DIR) for when the user cares to override them. Yeah, sure, that's what the spec says, and that's what is passive-aggressive. There is a complicated system and it is opt-out, which is considered abusive whenever anyone else does it. I actually like XDG, and I use it even when I have to specifically tell packages to do so. I have contributed patches to unrelated software for XDG support. However, I don't agree with the common online strategy of shitting on software that doesn't follow a spec that is deliberately more complicated than it needs to be.
      - remram 5y ago
        How is an optional override "opt-out"? Who is "shitting on software"? This rant is inscrutable.
- stormbrew 5y ago
  It does appear to[1], and it does help because you can set XDG_CACHE_HOME in your dockerfile so it'll be the same across all runs. I think that's what the GP was getting at. [1] https://huggingface.co/transformers/v4.3.3/installation.html?highlight=transformers_cache https://huggingface.co/transformers/v4.3.3/installation.html...
deleted 5y ago
[deleted]

reilly3000 5y ago

As an alternative, don’t use root for Dockerfiles. Ever. It only takes a few lines to create a user, group, and use said user, and closes a whole class of security issues.

sjtindell 5y ago
Closes security loopholes, increases operational burden (if only slightly). A tale as old as time.
- wussboy 5y ago
  Agree. It only takes a few lines. But how long does it take to learn what those lines are?
chanind 5y ago
True! That would have also been a smart move
Arnavion 5y ago
Are you sure Cloud Run only overrides $HOME when the container runs as root? Because if it always overrides $HOME then what user the container runs as is meaningless (for this particular issue).
- reilly3000 5y ago
  The problem is with root vs all other users is that it’s $HOME is literally /root/ while other users have their $HOME directory in /home/username/ ~ is a BASH tilde expansion for $HOME. So in this case Cloud Run would have been looking at /home/root/.cache/ which doesn’t exist (/root/.cache/ is what was being built), whereas another user would have /home/username/.cache/ and run as expected. PS. I was initially going to call ~ an alias, but I checked myself and found it’s actually considered a BASH tilde expansion. While ~ alone operates as an alias, I learned there are all kinds of other uses for variations of ~ which hopefully someone will find useful: http://www.gnu.org/software/bash/manual/html_node/Tilde-Expansion.html#Tilde-Expansion http://www.gnu.org/software/bash/manual/html_node/Tilde-Expa...

justin_oaks 5y ago

For those not familiar with Cloud Run, it's a Google Cloud service that allows you to run Docker containers without managing the server it runs on. https://cloud.google.com/run https://cloud.google.com/run

cmeacham98 5y ago

Does Huggingface not have a way to override the cache directory? I always specify directories like this manually when using docker specifically to avoid issues like the one in TFA.

manojlds 5y ago
They do - https://huggingface.co/transformers/v4.3.3/installation.html?highlight=transformers_cache#caching-models https://huggingface.co/transformers/v4.3.3/installation.html...

steren 5y ago

Cloud Run PM here. Sorry for the inconvenience. This is an issue for which we have a fix, but the fix still needs to roll out (internal bug 154368323 for googlers reading, we're still evaluating the best rollout strategy). Until then, I'll add it to https://cloud.google.com/run/docs/issues https://cloud.google.com/run/docs/issues.

renewiltord 5y ago
Cool beans. Thanks for the response.
hn_throwaway_99 5y ago
I only say this half in jest: Google, please give this person a raise. I'm not sure I've ever seen a Google PM be this up front and straightforward with a future roadmap item, even in HN, where I more commonly see more direct communications by PMs or engineers from various companies. While you're at it, if there is a way you can get the Google Identity Platform PM to let us know when some form of multi-factor auth besides SMS will be supported, that would be great!
- NicoJuicy 5y ago
  I feel that employees from big Corp sometimes can't do good. If they respond here, it should be as you mentioned. They are monitoring media on mentions. If they don't, people can't fix their issues ( probably ) as fast. But responding here gives the observing effect of having a higher priority. Which is sometimes "ridiculed" ( don't know a better describing term) Either way, thanks to the PM for an update about this.
  - KptMarchewa 5y ago
    I guess the main problem is the second part of the subOP comment: >While you're at it, if there is a way you can get the Google Identity Platform PM to let us know when some form of multi-factor auth besides SMS will be supported, that would be great! When they respond they get inundated with random requests and ailments that they have exactly zero control on.

wodenokoto 5y ago

That sucks. My only advice is, when using this or cloud functions always start by creating a function/image that prints all env variables. For cloud functions these changes tremendously between Python version. Also logging changes completely between some Python versions, to the point where upping the runtime causes logs to stop being saved on cloud logging.

GrayShade 5y ago
But if you have secrets in your environment variables, make sure to filter them out.