11 ms·
Tell HN: Docker just ate 19GB of production data
Be very careful with the live-restore feature of docker. Running 'docker volume prune' just removed all my named volumes, which were used by running containers.
See https://github.com/moby/moby/issues/38883 https://github.com/moby/moby/issues/38883
- zyztem 7y agoThat was definetely not a big data environment
- JoelMcCracken 7y agoNobody said anything about big data.
- gervu 7y agoAutomation of any sort will sometimes accidentally your data, whether due to periodic hiccups, system instabilities and bugs, operator misunderstandings or errors, or random cosmic ray strikes. The exact reason it blows up isn't even necessarily all that important, other than in its effect on what you should be doing to reduce the probability of downtime. Well-engineered systems are routinely developed from less than completely reliable parts. Stuff fails, we design for it. It's certainly not reason not to use it, if it's resulting in a net positive gain in your ability to get things done and maintain control and transparency over your deployed systems. But it's certainly a good reason (among a long list of good reasons) to make sure you have a good backup routine in place, including regular testing of both their integrity and your ability to restore a working prod system from them quickly.
- dsfyu404ed 7y ago>Automation of any sort will sometimes accidentally your data, Distributed scalable automation will accidentally your data slightly more often. The more stuff you have the more edge cases and bugs you have. Scale big fail big as I like to say.
- boobsbr 7y agoAccidentally what? =)
- aukaost 7y agoaccidentally the whole thing
- hinkley 7y agoyour data.
- deleted 7y ago[deleted]
- TrinaryWorksToo 7y agoDelete, but the word delete is deleted. Yo dawg.
- goalieca 7y agoYou weren’t around in the slashdot days were you?
- deleted 7y ago[deleted]
- jtchang 7y agoThis definitely sounds like a bug. docker volume prune says: "Remove all unused local volumes. Unused local volumes are those which are not referenced by any containers" If it removed a local volume that was being used by a container that is kinda bad.
- RocketSyntax 7y agoNot sure what kind of company you work at, but I'd export a copy of your logs so you don't get canned
- stcredzero 7y agoThis makes it sound like it's quite common to use docker containers operating in a heavily stateful fashion. Is that indeed common nowadays? (Though, the state in this case is only counted on to persist in the named volumes.)
- sz4kerto 7y agoThat's completely fine given that the important state is on volumes (named, persistent, bind mounted, whatever).
- johnchristopher 7y agoWell, you are supposed to be able to delete all your volumes, containers and networks and then regenerate it by running the recettes (edit:I mean.. recipes :D) (Dockerfile, docker-compose, kubernetes, volume backup, etc.).
- stcredzero 7y agoWell, you are supposed to be able to delete all your volumes, containers and networks and then regenerate So then Docker is designed to treat all of those as disposable. I just searched "recette" and only came up with French cooking references.
- ashtonbaker 7y ago> I just searched "recette" and only came up with French cooking references. Perhaps a phonetic spelling of "resets" by a French person :)
- johnchristopher 7y agoSomething like that :D. I was thinking "recette" as in "recipe" but somehow only the word "receipt" came to my mind so I thought "oh, it must be one of those words that are the same in both language". I am a bit tired.
- 7y ago
- sz4kerto 7y agoI really-really hope you are not relying on Docker only when protecting 19G of data. Docker volume operations are the equivalent of playing with sudo rm -rf, shit's going to happen once in a while.
- scarface74 7y agoI am a Docker newbie but I thought it wasn’t considered best practice to use Docker for anything where you care about the data in the container. I’ve only used for API’s batch jobs, etc.
- cheez 7y agoDocker volumes are persistent, unless they're not :-)
- scarface74 7y agoWell, I guess it also doesn’t help that I’ve only used Docker with AWS Fargate (aka “Serverless Docker”) - where nothing is persistent.
- q_queue 7y agoI've never been willing to consider docker volumes persistent. In the big picture, a requirement of "posix filesystem semantics" and "persistent" is a pretty inconvenient and/or expensive requirement.
- cheez 7y agoI am super paranoid about docker volumes. But I have application-level backup which is tested daily.
- kissgyorgy 7y agoLOL
- 7y ago
- frenchman99 7y ago`docker volume prune` is specifically there to remove volumes, so backing up before using it seems to be mandatory, just in case. But yeah, if this is a bug, it's a nasty one.
- praseodym 7y agoIn the Moby issue you mention that you are using live restore (https://docs.docker.com/config/containers/live-restore/ https://docs.docker.com/config/containers/live-restore/) which is most likely where the problem is. Docker daemon restarts, existing containers are kept alive, but the restarted Docker daemon doesn’t know about those existing containers yet and thus thinks their volumes are unused.
- DannyB2 7y agoComputers are wonderful. They can do the same work that would require a thousand people to accomplish in the same amount of time. Flip side . . . Computers are terrible. They can screw things up so bad it would require a thousand people to accomplish in the same amount of time.
- clinta 7y agoThis bug specifically says it affects anonymous volumes. If you had it delete a named volume that sounds like a new issue.
- orf 7y ago1. Why are you running docker volume prune in production? 2. Why are you running docker on ad-hoc machines you need to prune? 3. Why do you even need root access on production machines to fiddle around with docker commands? While this is obviously a bad bug (and there are many with Docker), it seems more of an operational procedures failure than anything else. You could be saying: “Beware of rm -rf /, it just deleted 20gb of production data” Ok. Sure. But why are you tools and procedures putting yourself in a position to make that mistake?
- reaperducer 7y agoOne of the most bothersome part of HN is when someone tells us about something that happened, and out come a ream of second-guessing replies. "Why didn't you just do this?" and Why didn't you just do that?" and any number of "It's so easy to just thing instead!" We don't know his environment. We don't know his company's policies. We don't know his hardware, connectivity, or budget issues. These kinds of passive aggressive responses are almost never helpful.
- orf 7y agoWhen you reduce it down the title here is “giving people access to running arbitrary, manual and presumably unrestricted maintenance commands in production leads to issues”. That’s not a surprise, and maybe the issue at the core here is not really Docker. That’s all.
- q_queue 7y agoSure, but regardless of people doing dumb things, it's still worth asking "why did docker delete non-orphaned named volumes?" -- though you could also question whether someone was actually mistaken about them not being "orphaned" - you could probably arrange an unfortunate timing collision between someone running prune and a container being respawned.
- uponcoffee 7y agoRight, that's what raising an issue with the software maintainers are for. Aside from anecdotes, there's little value in further discussion beyond the PSA that is the original post; save for prevention/recovery of such events.
- wiredfool 7y agoYou just won the “I dropped the production db” achievement. It’s surprisingly easy with docker, especially when dealing with .... legacy systems.
- LiamPa 7y ago> please assign this bug to an engineer. The joys of open source users..
- ironmagma 7y agoSeems like a reasonable ask.. I’m not sure what the problem with this is?
- LiamPa 7y agoMaintain an open source project and you will understand the problem of users having the ‘just fix it’ attitude.
- ironmagma 7y agoI have maintained open source projects before. Docker is a venture-backed company trying to sell its product which is mostly just hosting and support for their free product... and in addition, the user did ask nicely, they did not make demands. There's a big difference between asking that an engineer be assigned to a task and demanding something be fixed immediately for no bounty.
- acid303 7y agoMy browser ate 16GB of ram while I've been reading this. The system crashed but the tabs were here there after a reboot. I'm not even mad anymore.