5 ms·

Congratulations! It seems the memory issue with "docker" daemon hasn't been admitted/resolved. (See https://github.com/docker/docker/releases/tag/v1.8.0 https:

by anh79 11y ago

Congratulations! It seems the memory issue with "docker" daemon hasn't been admitted/resolved. (See https://github.com/docker/docker/releases/tag/v1.8.0 https://github.com/docker/docker/releases/tag/v1.8.0) In all of my deployments, "docker" daemon memory keeps increasing. The worst case is 37% in a 512MB instance in Digital Ocean. Then docker daemon crashes... That's to say, congratulate but I don't really want to upgrade ... :(

cpuguy83 11y ago
Memory issues can be caused by a number of things. There's been a lot of work done to improve some leakages around the logging -- both by being able to disable logging if you don't need it or switch to a different logger, and general improvements to the default jsonlog logger. There's also been a known issue around creating lots of `docker exec` instances and those references not being released (so you can query for status later), this is also resolved. It would be awesome if you could profile it and submit a bug report with how you are using docker and where you are seeing memory growth. Let me know how I can help so we can get this resolved.
- anh79 11y ago
  Hi @cpuguy83, Thanks for your feedback. I couldn't gather enough information for bug reporting (But I had few rich logs that told "docker" has invalid memory access.) As a work-around, I use a special patch of docker-compose that helps the containers to restart automatically when "docker" daemon restarts :) > There's also been a known issue around creating lots of `docker exec` instances and those references not being released (so you can query for status later), this is also resolved. Woh, this seems my case, though I am not sure about "releasing" (We run "exec" and "exit"; we don't hold any "exec" for longer than 2 seconds.) I haven't seen this mentioned in changelog (FIXME). Let me try again in ticket system!
  - cpuguy83 11y ago
    Yes, it's docker that was holding on to these exec instances (and cluttering up the `docker inspect` output if you do a bunch). We now do GC on these. ps, you can add `reastart: always` to your compose file, and docker daemon will automatically restart those containers for you.
    - anh79 11y ago
      Thanks a lot. I wonder why this is not mentioned in the changelog :) Regarding the `docker-compose` issue, it's described here https://github.com/docker/compose/pull/1723 https://github.com/docker/compose/pull/1723 . Aanand Prasad has a better PR in https://github.com/docker/compose/pull/1754 https://github.com/docker/compose/pull/1754 (merged).

by anh79 11y ago

cpuguy83 11y ago

Memory issues can be caused by a number of things. There's been a lot of work done to improve some leakages around the logging -- both by being able to disable logging if you don't need it or switch to a different logger, and general improvements to the default jsonlog logger. There's also been a known issue around creating lots of `docker exec` instances and those references not being released (so you can query for status later), this is also resolved. It would be awesome if you could profile it and submit a bug report with how you are using docker and where you are seeing memory growth. Let me know how I can help so we can get this resolved.

anh79 11y ago
Hi @cpuguy83, Thanks for your feedback. I couldn't gather enough information for bug reporting (But I had few rich logs that told "docker" has invalid memory access.) As a work-around, I use a special patch of docker-compose that helps the containers to restart automatically when "docker" daemon restarts :) > There's also been a known issue around creating lots of `docker exec` instances and those references not being released (so you can query for status later), this is also resolved. Woh, this seems my case, though I am not sure about "releasing" (We run "exec" and "exit"; we don't hold any "exec" for longer than 2 seconds.) I haven't seen this mentioned in changelog (FIXME). Let me try again in ticket system!
- cpuguy83 11y ago
  Yes, it's docker that was holding on to these exec instances (and cluttering up the `docker inspect` output if you do a bunch). We now do GC on these. ps, you can add `reastart: always` to your compose file, and docker daemon will automatically restart those containers for you.
  - anh79 11y ago
    Thanks a lot. I wonder why this is not mentioned in the changelog :) Regarding the `docker-compose` issue, it's described here https://github.com/docker/compose/pull/1723 https://github.com/docker/compose/pull/1723 . Aanand Prasad has a better PR in https://github.com/docker/compose/pull/1754 https://github.com/docker/compose/pull/1754 (merged).