15 ms·

Running Docker in production for 6 months

197 points by swap32 10y ago

vivekjuneja2010 10y ago
Good write-up considering that you had to manage the adventure of learning and scaling Docker along with the pressure of commitments from clients. I would like to share some pointers on how I and my team deals with the issues that you mentioned in this post. 1. Orchestration :- For us, Swarm never became a choice, as we started our journey adopting Docker in early 2015. That time there was no Swarm. We resolves to Mesos and Marathon for our orchestration. Both worked out well for us in the long run. We have production systems running this setup for last few months. Swarm is mature now, and with Docker 1.12 its been made more easier to use. The good thing with Swarm is that you could avoid adding another new system in your infrastructure like Mesos, Kubernetes etc. if you have reasonably simple requirements. We found that Orchestration also established service discovery and routing capabilities for us. We use HAProxy and Mesos DNS for our routing and discovery needs. Marathon-lb project is used to allow us to reconfigure HAProxy everytime a new Docker Container is deployed by the CD Pipeline. Marathon manages our service ports across the cluster, and every new Docker container gets its own unique service port. This service port is then informed to HAProxy, and reload happens. This setup worked good for us, although we had some initial trouble. We also practice Zero downtime deployment with our stateless services using the ZDD script inside the Marathon-lb project. 2. Running out of disk space :- This is a common problem especially with the idea of rebuilding and deploying disposable containers with the CD pipeline. In our case, we use Monit to gather system wide metrics at all times. We use a Garbage collection script that we developed in house to remove the old Docker images and Containers periodically whenever Monit detects file system usage beyond the set thresholds. We do continuous production deployments as often as we need, so this allows for our Docker image diff to be minimal. We avoid big bang releases so that the latency for docker push on the Build server and docker pull on the cluster is minimal. The Spotify Docker-GC project is a good choice according to me. 3. Docker registry :- We use Docker Registry container that runs on the Mesos cluster via Marathon. The Docker Registry is backed by a shared volume on the Docker hosts. We share the same volume on all Docker hosts in our cluster. So, if the registry crashes on one host, Marathon is able to redeploy the Registry Container on another host which has the access to the shared registry volume. We tried moving our Registry backed to S3, but never in production. For the systems we manage, we need the Docker images in house due to compliance requirements. Therefore, we could not use Gitlab Registry or Docker hub for our production deployments. But I heard good things about Gitlab registry. 4. Logging :- We use Logspout on each Docker hosts. It forwards the logs to our managed Logstash and further to Elasticsearch service. We use Kibana for log dashboard. Logs are rolled over on each Docker container, so that we avoid storing the logs on the host for long time. However, any distributed logging introduces log ordering and latency issues. So, we are tackling them as of now through some optimisations. 5. Dependency and Base Images :- We use hierarchical model of managing Base Images : One top-level Registry (Global), and isolated docker registry for each project. Every Base image gets into our Top-level Docker registry which is curated, and the associated Dockerfile for that Base image is checked into our Git Repository. We insist using these Base Images from our registry for all projects. Each project can then inherit the base image, and customize to the local needs of the project. We follow CI and CD for our Base Images as well. Each project gets a notification when the Base image changes. They are free to opt in or opt out. This model works for us, but many not be that interesting for smaller setups. I had written about it here:- http://thenewstack.io/bakery-foundation-container-images-microservices/ http://thenewstack.io/bakery-foundation-container-images-mic... 6. DB and Persistence :- We avoid running stateful services in production on Docker Container, as we have qualms about the persistence support in Docker. However, we do use Docker volumes for all purposes in non-prod environments including CI, Elasticsearch and other services. We are very interested to pursue this further with ClusterHQ and Flocker based offerings in the Docker ecosystem. I may blog about it in the coming days on this. But so far, I don't have any production experience with the Database in Docker. But I am optimistic that this will happen soon. 7. Longer build times :- This is correct as per your assessment, but widely varies across deployments. As said earlier, we want the team to have faster build times, so we build and release as often as possible.This allows to not have to deal with build latency. We use lightweight Base images like Alpine, and prevent the use of Configuration and Package mangers like Puppet inside the container. In our base images, we prevent bloating by avoid installing irrelevant packages. This has backfired some times in production, but we have found ways to go around it most times. Overall, I am aware of the challenges that you had, and can connect with all of them. Docker is not the panacea for all the infrastructure woes, but its certainly gives the taste to me and our team on how software development and delivery will change for good in the coming days.
- swap32 10y ago
  There are some excellent suggestions and workarounds here and looks like you've been using them in production for some time. Thanks for sharing.
orf 10y ago
I agree with most of the points here, but some stuff is misleading. For example the "you have to rebuild after everything that code change" is ridiculous. Just use "COPY SRC/ /app" In your dockerfile, and in dev mount SRC/ as a volume over /app. There, hot reloading sorted for development. Don't get me wrong, docker is one of the most frustrating technologies I've used (partly because it shows such promise), but a lot of the problems he describes can be sorted with the most cursory Google.
- blowski 10y ago
  Perhaps the article should rephrase it as "it's very complicated to learn how to do it properly, and not obvious when you're doing it wrong". My experience with Docker is that Google searches often turn up with configurations that other Google searches will say are a bad idea. There don't seem to be any kind of well-documented emergent best practices.
  - swap32 10y ago
    yes, that's pretty much what I meant. Our usage scope has been limited and so has been the time/resources we were willing to invest into researching fixes for things which were obvious for us before.
- gtirloni 10y ago
  What do you use to restart the app inside the container when the code changes?
  - majewsky 10y ago
    Grandparent is referring to hot reloading, wherein the application itself recognizes changed source files and reloads them on the fly. A common feature of server-side MVC frameworks like Django, Rails, etc.
  - swap32 10y ago
    Because since the source code folder was not mounted on the container, we had to rebuild the image with the updated code and then restart the container. On dev machines, since the source folder is mounted, django runserver's autoreload function works as usual.
afandian 10y ago
Linked at the end is this article: https://thehftguy.wordpress.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.wordpress.com/2016/11/01/docker-in-product... As someone starting considering Docker (and possibly Swarm), these seem to be pretty serious criticisms. Any experiences to corroborate / counter these two posts? Going by what's written here it would be suicide to use Docker, but many people are...
- pella 10y ago
  see: https://news.ycombinator.com/item?id=12872304 https://news.ycombinator.com/item?id=12872304
- swap32 10y ago
  I would put it this way - how much value gain are you expecting integrating docker in your processes? Is it a pressing need right now? How badly would it affect you if it doesn't work out for you? The article is the summary of our small scale experimentation. You can definitely experiment and weigh the gains for yourself. But its good to have the issues listed in the articles here at the back of your mind.
  - afandian 10y ago
    Thanks! My position is, I'm starting from a blank slate with a new project. I want feature set X, and Docker and Swarm supports feature set X. (where X is something like running a number of little microservices with failover, load balancing, local integration testing, mixed language platforms, sharing development across organisations). I could get them another way, but I want to start on the right foot with a new project.
    - swap32 10y ago
      I can assure you, you can do all that without getting yourself into the microservices mess and docker. As you said, please get them another way. If this is a serious project with deadlines and client money at stake, as you said please do it another way.
lizxrice 10y ago
I'm a big Docker fan, but this is interesting to me mostly because it shows how it's hard to get started, particularly the opening few sentences about the paucity of documentation for helping you get going from scratch. There have been good guides, but they go out of date quickly! There's so much change (because it's still very much under development) that it's hard for someone coming to it fresh to figure out what's up-to-date and what isn't. For those of us working in container tech and tools this is a good lesson about making sure the entry curve isn't too steep.
- d33 10y ago
  I'd argue that it wouldn't be that steep if Docker wasn't trying to do too much. There's so many moving parts - apart from regular process isolation, you've got to manage storage (which can be a pain in the ass) and firewall configuration (which is another very complicated and not well documented topic). In any non-trivial setup those two topics are already big. Now add container management - understanding the split between images and containers, then how to publish and download images (which IMHO should be a completely separate project), how caching works so you could write a decent Dockerfile... And then there are features like Docker Swarm which I never touched and seem particularly complex to me. It's way too many projects stuffed into one thing.
  - e12e 10y ago
    The feeling you describe reminds me of how I felt evaluating puppet and chef some years back - with a particular view to running the open source parts (not being dependent on third party support). It felt like every other link led back to the various sales teams for the enterprise solutions, and there existed no page with a straightforward overview of all the components and what puppet / chef actually did. Ansible was a breath of fresh air - as well as saltstack (but at the time it had too many holes in its security/transport story).
- k__ 10y ago
  On the other hand, which alternatives are easier? I'm a NixOS fan, but have to use Docker at work occasionally and I have the feeling they are both difficult, but in "their" ways.
romanovcode 10y ago
> Since the client wasn’t keen on spending on getting a more private repos, we managed with the single repo to save our base image with the most of the dependencies bundled into it. What's the point then?
- swap32 10y ago
  Well, most of your client are bound to do this. The point is we managed to get away with only one private repo. My guess is you'd need more than one private repo very rarely.
baconizer 10y ago
This article reflects our transition experience with Docker, it took us two years to finally feel comfortable (old chaps from decades of corp dev), but end result is very positive and rewarding. It is fair to say that this should be expected for any infrastructure migration, there is nothing wrong with being slow and careful as long as we are moving forward. Based on experience from our team, the problem has never been finding help/answers, problem is we were facing, one one hand, an encyclopedia of single page documentation like Dockerfile/docker run command, while practical guidances and gotchas are scattered around rest of teh Internet, blended with personal and business specific opinions. It is hard but eventually we figured the best way to work with containers from git to build server to deployment, that fits well with our productivity workflow;, as well as where to not use docker.
- swap32 10y ago
  It'd be great if you can share your learnings and increase in productivity with docker AND where not to use docker.
Walkman 10y ago
Yet another guy who doesn't know how to use Docker properly and just brags about it. There are a lot of misinformation in this blog post. The conclusion from these is not that Docker sucks, but YOU HAVE TO LEARN it. I agree that it's a very steep learning curve, but after the pieces come together, Docker solves quite a lot of problem and actually very useful.
- swap32 10y ago
  I think the article is very clear that this is about OUR experience of using docker. It is also stated right at the beginning that we have a business to run. If it going to take 2 years to integrate it into workflow (as it did for one of the commentors here) and hundreds of developer hours just to make it work, we have to decide whether it is viable for us. Other than that, I don't hear you refuting a single point that is made in the article. >> Yet another guy ... If there are a lot of people who are finding it difficult to grapple with it, maybe its really not their fault?
- atemerev 10y ago
  You have to learn it — where? Documentation is sketchy and mostly obsolete, docker-swarm is buggy, genuine production issues are routinely closed with WONTFIX resolution. Could you point me to good piece of documentation or success story of running Docker in production? (I do use Docker, but it requires to learn Kubernetes or DCOS to do anything production-related. And those are separate projects).
  - fireflash38 10y ago
    Every one of these blog posts begins with almost exactly the same thing: The documentation just plain sucks. The doc parts that don't suck are outdated, and therefore useless. As long as that is the case, you are going to have people who go through the exact same troubles with Docker time and time again (one theme is databases in docker).
  - cpuguy83 10y ago
    Maintainer here, can you please point to a production issue that is WONTFIX?
crb 10y ago
> Orchestration across multiple physical server gets even more nasty where you’d have to use something like Swarm. We’ve since realised that swarm is one of the lesser preferred options for orchestrating clusters. Could you elaborate on this? Did you settle on an orchestrator, and if so, which one?
- swap32 10y ago
  We didn't use orchestration across multiple physical machines so it was possible for us to get away with just docker-compose. Otherwise we would have had to go with swarm. Kubernetes is a way better choice than both compose and swarm but it has a even more steep learning curve.
  - atemerev 10y ago
    In fact no, it doesn't. Documentation is much better, and bugs are much less.
  - crb 10y ago
    I work on Kubernetes, and we're putting a lot of effort into making it easier to get started. Here are some things that might help: * minikube: http://kubernetes.io/docs/getting-started-guides/minikube/ http://kubernetes.io/docs/getting-started-guides/minikube/ * the new tutorial: http://kubernetes.io/docs/tutorials/kubernetes-basics/ http://kubernetes.io/docs/tutorials/kubernetes-basics/ * deploying in 2 steps with kubeadm: http://kubernetes.io/docs/getting-started-guides/kubeadm/ http://kubernetes.io/docs/getting-started-guides/kubeadm/ I'd love any honest feedback on how we can either (a) fix the learning curve, or (b) remove the perception that it's steep.
    - vmarsy 10y ago
      Just a quick heads up: On chrome v54.0.284.0.68 on my Android nexus 7 tablet the new tutorial page is half cut. I only see "asics" instead of what I assume is supposed to be Basics, all the text below is also cut: Overview asics through of the basics of the Kubernetes cluster module contains some background information ... There's no way to zoom out, and there's 2 hamburger menu : one white on black on the left and one blue on white on the right, both are broken The 2 other links work well
pacuna 10y ago
Kind of very basic issues IMO. The only one that's hard is logging. We are using K8s on AWS and I haven't found a good solution for logging centralization. Personally I don't like the Kibana interface for that kind of stuff.
- nepotism2016 10y ago
  I blame this on the way kibana is advertised. Some of web2.0 GUI with fancy color schemes and graphs. I use kibana to monitor logs over 60 containers. Sure, the initial dashboard is a pain to setup, but once you get going going its very useful
  - pmontra 10y ago
    My criticism to Kibana is that it runs all the queries for a dashboard in parallel. It overloads the Elasticsearch machine(s) even when there are only few data. A small single ES server would be enough if the queries would be run sequentially. I wonder if Elastic is in the business of selling multicore 32 GB servers that sit idle most of the time. A max_parallel_queries setting would be great.
nzoschke 10y ago
Thanks for sharing your experience as another cautionary tale. At Convox we have been running Docker in prod for 18 months successfully. The secrets? 1. Don't DIY. Building a custom deployment system with any tech (Docker, Kubernetes, Ansible, Packer, etc) is a challenge. All the small problems add up to one big burden on you. 6 months later you look back at a lot of wasted time... 2. Don't use all of Docker. Images, containers and the logging drivers are all simple great. Volumes, networks and orchestration are complex. 3. Use services. Using VPC is far simpler than Docker networking. Using ECS is much easier than maintaining your own etcd or Swarm cluster. Using Cloudwatch Logs is cheaper and more reliable than deploying a logging contraption into your cluster. Use a DB service like RDS is far far easier than building your own reliable data layer. Again thanks for sharing your experience as a cautionary tale. If you are starting a new business you should not take on building a deployment system as part of the challenge. Use a well-built and peer reviewed platform like Heroku, Elastic Beanstalk or Convox.
- nuschk 10y ago
  Re 1: Why wouldn't you recommend going with kubernetes? From someone that hasn't deployed anything with it (yet), it seems rather straight-forward if run on a managed cluster (gke). I might be wrong and am thus genuinely interested in what you did for deployments.
  - bryanlarsen 10y ago
    AFAICT, gke would be #3 (use services) not #1 (don't DIY)
    - nzoschke 10y ago
      That's right. GKE is awesome. I highly recommend using the GKE service that over running Kubernetes software on AWS or Digital Ocean.
      - nuschk 10y ago
        I'm relieved, thanks guys. I really don't see the complexity when deploying to a (managed) k8s cluster. Sure, we'll still have to configure a deployment pipeline, add some scripts to push and deploy the right images etc., but overall, k8s seems really well suited to the task.
coldcode 10y ago
Reading these articles and comment is fun, we are a very large organization and starting to use docker, and we love making things even more complex than other folks, so I can tell this is going to a big mess. Fundamentally we (as in this industry) seem to love simple ideas that become massively complex.
- qaq 10y ago
  Well that's how we ensure availability of employment opportunities :)
- fapjacks 10y ago
  What I've found is that as long as you have someone on the books that has more than a passing interest in learning about how Docker works, you'll be fine. The vast majority of problems with Docker boil down to not fully understanding the features being used. Docker is very easy to get going, and a very deep technical subject at the end of the day. There is a lot to absorb, and if you're going to throw it into production, it would behoove you to fully understand the technology. This isn't something you can expect to be without problems unless you invest in reading the documentation. This opinion isn't popular, because everybody wants to think of themselves as truly and fully understanding what they're using. But the truth is right there in the problems people run into.
pmontra 10y ago
Why is running a db in a container with the data directory mounted from the host is such a bad idea? Dropping the container on a new host with a copy of the data directory looks easier than having to install the db from scratch, even with automatic tools. Are there performance penalties at run time?
- swap32 10y ago
  It is fine for dev, but not for production. To quote from the link at the end of the article- "Docker is meant to be stateless. Containers have no permanent disk storage, whatever happens is ephemeral and is gone when the container stops. Containers are not meant to store data. Actually, they are meant by design to NOT store data. Any attempt to go against this philosophy is bound to disaster. Moreover. Docker is locking away processes and files through its abstraction, they are unreachable as if they didn’t exist. It prevents from doing any sort of recovery if something goes wrong" "A crash would destroy the database and affect all systems connecting to it. It is an erratic bug, triggered more frequently under intensive usage. A database is the ultimate IO intensive load, that’s a guaranteed kernel panic. Plus, there is another bug that can corrupt the docker mount (destroying all data) and possibly the system filesystem as well (if they’re on the same disk)."
  - cpuguy83 10y ago
    All these statements are patently false. 1. Containers are not ephemeral. They have a lifecycle. Data written in the container is persisted to disk and available after the container is stopped and then started again. 2. Processes/files/etc are not locked away as if they don't exist. See `ps aux` on the host. You will see all the processes running. You can inspect the filesystems for each container, etc. There is no magic here. 3. A database crash could cause data corruption inside a container or not. This has nothing to do with the container, and chances of a database crash are not made worse by being in a container. That said, I would let a volume driver manage persistent storage rather than manually managing this through the host fs... but that's my preference. --- EDIT --- Disclaimer: I work at Docker Inc, and am a maintainer on the Docker project.
qaq 10y ago
Not being a fan of docker but "Since images sizes can be as high as few GB, its easy to run out of disk space. This is another problem with docker which you have to figure out yourself. Despite the fact that everyone who’s ever used docker seriously has to come across this issue sooner or later; no one tells you about this at the outset" - monitoring free disk space is something you definitely want to do regardless of using docker or not.
- wccrawford 10y ago
  The size of docker images ended up being the reason we aren't using it. Transfering them took forever, and so if we had an emergency update to make, we'd have to go outside our normal routines, which is brutal. I'm sure we'll end up finding another way that's similar (provides the same benefits for us) but without such crazy image sizes.
  - fapjacks 10y ago
    Is your app gigabytes in size? Docker can't help you with that, but nothing else is going to help you, either. There has been a ton of momentum lately in getting Docker images pared down to miniscule (e.g. ~25MB... megabytes) image sizes with for example alpine. If you're using something like "ubuntu:14.04" for your images, this is absolutely a problem you can solve.
    - deleted 10y ago
      [deleted]
- kpcyrd 10y ago
  It's not that hard to work around this: I've deployed a global unit into our coreos cluster that regularly executes: #!/bin/sh while :; do # remove stopped containers w/ volumes docker ps -a -q -f status=exited | \ xargs -r docker rm -v # remove dangling images docker images -f "dangling=true" -q | \ xargs -r docker rmi sleep 6h done You need to be sure that you don't lose important data when running something like this in your setup, but it works nicely to remove old images. This script is deployed to non-coreos servers as well.
vkat 10y ago
Logging we’ve frequently experienced issues with django autoreload and delay in flushing logs >> Try using "ENV PYTHONUNBUFFERED 1" in your dockerfile
vuyani 10y ago
"we didn’t find enough reason to figure out if kubernetes was a better option." Then proceeds to name every problem kubernetes fixes
- eggie5 10y ago
  haha, that's exactly what I said in my comment.
- swap32 10y ago
  Yes. My bad! We really should have invested some time evaluating Kubernetes.
  - gtaylor 10y ago
    If you don't want to stand up your own cluster for eval, fire up Google Container Engine for a few days. It's cheap and very well done.
- x0x0 10y ago
  I think the problem started far earlier. The author couldn't name a single clear rationale for using docker at all. One client asked them to, but why continue? They were using virtualenv and it mostly worked fine. So why docker period? Complexity for the sake of complexity? The rest of the article then basically says complex things are complex.
  - swap32 10y ago
    Well, I didn't name the reasons of why we continued using Docker because I thought they were obvious to any serious docker user. The reasons are all the promises that docker makes to you without actually telling you the complexities involved. Namely - the promise of better portability, better dev-prod environment replication (You can even run vagrant for dev portability but you wouldn't use it for prod, but with docker, its just docker everywhere), better scalability with managing clusters and deployments, and so on... without actually running heavy virtual machines. And no, as per the docker promise, it wasn't supposed to be complex.
eggie5 10y ago
Orchestration: A lot of the issue I see him describe in production are fixed by Kubernetes. Compose works fine for local dev orchestration but pattern doesn’t work for deployment. The ideal world would be that I can run my compose file on my cloud provider, but Swarm isn’t there yet. I have to rewrite my compose file using kubernetes configs — it’s not a 1:1 mapping but the high level connection are there if you think of Cabernets Pods as Docker Containers. He mentions orchestration across a cluster with Swarm is nasty, but it’s elegant w/ Kubernetes. Docker Registry: Obviously, there is no constraint permitting him to use a 3rd party service. Why to let Google Container Engine (GKE) or AWS ECR handle it for you? Longer build times: I think this is really where he is missing the mark. It sounds like he has a fundamental misunderstand that if you mount the source code in dev you have to do it in prod too and that you have to have 1 container. Not true: You can mount the source code in dev using compose, so you don’t have to rebuild every time you change a line. Also, I think it’s a pattern in docker to try to keep your containers as atomic units of your app architecture. It sounds like they are trying to bake all competent of their app into 1 container (app + db + service, etc). Just break them up into containers, link them up w/ compose. This architecture them translates cleanly to one of the cloud providers for production: GKE or AWS. DB and Persistence: Yes, I think it is very clear that containers are stateless. So, yes if you want to run a DB in a container you’d have to mount an external drive somewhere. There merits and risks of that are another discussion, but as he states it’s generally frowned upon to containerize a DB. (not completely sure why, some argument about stability of container and corruption of data…) I talk more about this here: https://news.ycombinator.com/item?id=12913198 https://news.ycombinator.com/item?id=12913198 Logging: I think 12-factor style app containerized fit more smoothly into the docker compose style architecture. Accordingly if all you containers are logging to std out it, it’s all conveniently merged and printed out to the terminal if you run compose. Then on production, GKE handles it nicely too w/ the Logging system. In conclusion, I think most of his problems would been avoided if he didn’t skip researching Kubernetes and if he didn’t make the mounting oversight. The other big oversight I think he did, at least he didn't mention, is that he has no deployment tool. I wouldn't be able to effectively deploy w/o a build tool like Jenkins. I talk about a lot of these issues and how to fix them here: https://news.ycombinator.com/item?id=12860519 https://news.ycombinator.com/item?id=12860519
gigatexal 10y ago
There's a lot of "figured it out ourselves" without actually sharing what they figured out. It's like this post is a whole lot of nothing: we know docker in production is not easy but how about telling us more details
- swap32 10y ago
  Isn't every "figured it out ourselves" followed with the sentence describing what we figured? I see only one section where I didn't describe what we figured, which is the workflow part and I mentioned there that I would write a separate blog.
yotamoron 10y ago
Honestly, this whole post sounds like an amateur complaining about how hard it is to become a proffesional. Grow up.
- swap32 10y ago
  Please enlighten us with your professional wisdom and do point out which parts of the article are amateur.
kossae 10y ago
If anyone is looking into orchestration systems, I can't speak highly enough of Kontena (https://github.com/kontena/kontena https://github.com/kontena/kontena). While it is still in the early stages of development, it is a great "small-mid level" platform with a ton of features. The config and concepts are also easy to wrap your head around. We chose Kontena as Swarm is still unstable (imo not "production-ready" but I've seen counter-anecdotes) and Kubernetes was lacking some features we required. The devs have been super helpful with any problems we've had in deployment as well.
20andup 10y ago
I have been playing with docker for the past week, and its really hard to get started. I am only going to use it for integration test for now.
zwischenzug 10y ago
A lot of this was the motivation for writing the book we wrote on Docker in Practice [1]. The reality of implementing such a technology requires a combination of improvisation, technical nous, bluffing, and a willingness to work through the inevitable problems [2]. I've talked about the relative immaturity of Docker as a used system (outside of dev) [3] and am struck often by how rarely people understand that it's still a work in progress, albeit one that can massively transform your business. The hype works. That said, Docker can work fantastically in production, but you need to understand its limits and start small. [1] https://www.amazon.com/Docker-Practice-Ian-Miell/dp/1617292729 https://www.amazon.com/Docker-Practice-Ian-Miell/dp/16172927... - working on 2nd edition, if anyone has any suggestions @ianmiell [2] Blog: https://medium.com/@zwischenzugs https://medium.com/@zwischenzugs [3] https://www.youtube.com/watch?v=Yj5I-yOaDao https://www.youtube.com/watch?v=Yj5I-yOaDao
brown9-2 10y ago
We spent hours figuring out a good way to use a databases in both dev and production with docker. It was tricky since docker containers don’t support persistence unless you use a mount-point. There were a few patterns documented which didn’t work for us or we didn’t really like. We had to figure it out by ourselves. This is another area where you’re expected to figure out by yourself whether it is a good idea to use docker to run your production database. Hint- Its not. I think that a problem that leads to a lot of these articles being written is that the motivation of the authors to use Docker is unclear. Why bother to put your database inside a container if the way you ran your databases before worked just fine? You shouldn't just rush to put all your processes in containers because it is cool and Containerization Is A Good Thing. You should use technology that makes sense given the problems you are trying to solve at a give moment.
- swap32 10y ago
  >> Why bother to put your database inside a container if the way you ran your databases before worked just fine? For the same reason you'd bother putting anything in a container at all. The purpose of docker -as advertised by docker- is to containerize EVERY service. Unless you can point me to a single mention in official documentation that databases are an exception. Till date, I haven't found any and thus this blog.
- dglass 10y ago
  Completely agree. I don't know where or how people started thinking that putting a stateful service inside of a stateless container was a best practice, but no one should be doing that in production.
  - allochthon 10y ago
    I wouldn't contemplate it, except that I'm looking at migrating a Rails/Postgrep app to GKE, and containerizing PG seems like the lowest-friction path. Are there better alternatives for getting to GKE, or is GKE a bad fit for this scenario?
  - swap32 10y ago
    As I said, if it is really as obvious as you say it is, please point me to a single mention in official documentation that databases should not be containerized.
yahyaheee 10y ago
Sounds like you need k8s! I went though a similar progression: loving docker -> hating docker -> finding k8s

197 points by swap32 10y ago

vivekjuneja2010 10y ago

Good write-up considering that you had to manage the adventure of learning and scaling Docker along with the pressure of commitments from clients. I would like to share some pointers on how I and my team deals with the issues that you mentioned in this post. 1. Orchestration :- For us, Swarm never became a choice, as we started our journey adopting Docker in early 2015. That time there was no Swarm. We resolves to Mesos and Marathon for our orchestration. Both worked out well for us in the long run. We have production systems running this setup for last few months. Swarm is mature now, and with Docker 1.12 its been made more easier to use. The good thing with Swarm is that you could avoid adding another new system in your infrastructure like Mesos, Kubernetes etc. if you have reasonably simple requirements. We found that Orchestration also established service discovery and routing capabilities for us. We use HAProxy and Mesos DNS for our routing and discovery needs. Marathon-lb project is used to allow us to reconfigure HAProxy everytime a new Docker Container is deployed by the CD Pipeline. Marathon manages our service ports across the cluster, and every new Docker container gets its own unique service port. This service port is then informed to HAProxy, and reload happens. This setup worked good for us, although we had some initial trouble. We also practice Zero downtime deployment with our stateless services using the ZDD script inside the Marathon-lb project. 2. Running out of disk space :- This is a common problem especially with the idea of rebuilding and deploying disposable containers with the CD pipeline. In our case, we use Monit to gather system wide metrics at all times. We use a Garbage collection script that we developed in house to remove the old Docker images and Containers periodically whenever Monit detects file system usage beyond the set thresholds. We do continuous production deployments as often as we need, so this allows for our Docker image diff to be minimal. We avoid big bang releases so that the latency for docker push on the Build server and docker pull on the cluster is minimal. The Spotify Docker-GC project is a good choice according to me. 3. Docker registry :- We use Docker Registry container that runs on the Mesos cluster via Marathon. The Docker Registry is backed by a shared volume on the Docker hosts. We share the same volume on all Docker hosts in our cluster. So, if the registry crashes on one host, Marathon is able to redeploy the Registry Container on another host which has the access to the shared registry volume. We tried moving our Registry backed to S3, but never in production. For the systems we manage, we need the Docker images in house due to compliance requirements. Therefore, we could not use Gitlab Registry or Docker hub for our production deployments. But I heard good things about Gitlab registry. 4. Logging :- We use Logspout on each Docker hosts. It forwards the logs to our managed Logstash and further to Elasticsearch service. We use Kibana for log dashboard. Logs are rolled over on each Docker container, so that we avoid storing the logs on the host for long time. However, any distributed logging introduces log ordering and latency issues. So, we are tackling them as of now through some optimisations. 5. Dependency and Base Images :- We use hierarchical model of managing Base Images : One top-level Registry (Global), and isolated docker registry for each project. Every Base image gets into our Top-level Docker registry which is curated, and the associated Dockerfile for that Base image is checked into our Git Repository. We insist using these Base Images from our registry for all projects. Each project can then inherit the base image, and customize to the local needs of the project. We follow CI and CD for our Base Images as well. Each project gets a notification when the Base image changes. They are free to opt in or opt out. This model works for us, but many not be that interesting for smaller setups. I had written about it here:- http://thenewstack.io/bakery-foundation-container-images-microservices/ http://thenewstack.io/bakery-foundation-container-images-mic... 6. DB and Persistence :- We avoid running stateful services in production on Docker Container, as we have qualms about the persistence support in Docker. However, we do use Docker volumes for all purposes in non-prod environments including CI, Elasticsearch and other services. We are very interested to pursue this further with ClusterHQ and Flocker based offerings in the Docker ecosystem. I may blog about it in the coming days on this. But so far, I don't have any production experience with the Database in Docker. But I am optimistic that this will happen soon. 7. Longer build times :- This is correct as per your assessment, but widely varies across deployments. As said earlier, we want the team to have faster build times, so we build and release as often as possible.This allows to not have to deal with build latency. We use lightweight Base images like Alpine, and prevent the use of Configuration and Package mangers like Puppet inside the container. In our base images, we prevent bloating by avoid installing irrelevant packages. This has backfired some times in production, but we have found ways to go around it most times. Overall, I am aware of the challenges that you had, and can connect with all of them. Docker is not the panacea for all the infrastructure woes, but its certainly gives the taste to me and our team on how software development and delivery will change for good in the coming days.

swap32 10y ago
There are some excellent suggestions and workarounds here and looks like you've been using them in production for some time. Thanks for sharing.

orf 10y ago

I agree with most of the points here, but some stuff is misleading. For example the "you have to rebuild after everything that code change" is ridiculous. Just use "COPY SRC/ /app" In your dockerfile, and in dev mount SRC/ as a volume over /app. There, hot reloading sorted for development. Don't get me wrong, docker is one of the most frustrating technologies I've used (partly because it shows such promise), but a lot of the problems he describes can be sorted with the most cursory Google.

blowski 10y ago
Perhaps the article should rephrase it as "it's very complicated to learn how to do it properly, and not obvious when you're doing it wrong". My experience with Docker is that Google searches often turn up with configurations that other Google searches will say are a bad idea. There don't seem to be any kind of well-documented emergent best practices.
- swap32 10y ago
  yes, that's pretty much what I meant. Our usage scope has been limited and so has been the time/resources we were willing to invest into researching fixes for things which were obvious for us before.
gtirloni 10y ago
What do you use to restart the app inside the container when the code changes?
- majewsky 10y ago
  Grandparent is referring to hot reloading, wherein the application itself recognizes changed source files and reloads them on the fly. A common feature of server-side MVC frameworks like Django, Rails, etc.
- swap32 10y ago
  Because since the source code folder was not mounted on the container, we had to rebuild the image with the updated code and then restart the container. On dev machines, since the source folder is mounted, django runserver's autoreload function works as usual.

afandian 10y ago

Linked at the end is this article: https://thehftguy.wordpress.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.wordpress.com/2016/11/01/docker-in-product... As someone starting considering Docker (and possibly Swarm), these seem to be pretty serious criticisms. Any experiences to corroborate / counter these two posts? Going by what's written here it would be suicide to use Docker, but many people are...

pella 10y ago
see: https://news.ycombinator.com/item?id=12872304 https://news.ycombinator.com/item?id=12872304
swap32 10y ago
I would put it this way - how much value gain are you expecting integrating docker in your processes? Is it a pressing need right now? How badly would it affect you if it doesn't work out for you? The article is the summary of our small scale experimentation. You can definitely experiment and weigh the gains for yourself. But its good to have the issues listed in the articles here at the back of your mind.
- afandian 10y ago
  Thanks! My position is, I'm starting from a blank slate with a new project. I want feature set X, and Docker and Swarm supports feature set X. (where X is something like running a number of little microservices with failover, load balancing, local integration testing, mixed language platforms, sharing development across organisations). I could get them another way, but I want to start on the right foot with a new project.
  - swap32 10y ago
    I can assure you, you can do all that without getting yourself into the microservices mess and docker. As you said, please get them another way. If this is a serious project with deadlines and client money at stake, as you said please do it another way.

lizxrice 10y ago

I'm a big Docker fan, but this is interesting to me mostly because it shows how it's hard to get started, particularly the opening few sentences about the paucity of documentation for helping you get going from scratch. There have been good guides, but they go out of date quickly! There's so much change (because it's still very much under development) that it's hard for someone coming to it fresh to figure out what's up-to-date and what isn't. For those of us working in container tech and tools this is a good lesson about making sure the entry curve isn't too steep.

d33 10y ago
I'd argue that it wouldn't be that steep if Docker wasn't trying to do too much. There's so many moving parts - apart from regular process isolation, you've got to manage storage (which can be a pain in the ass) and firewall configuration (which is another very complicated and not well documented topic). In any non-trivial setup those two topics are already big. Now add container management - understanding the split between images and containers, then how to publish and download images (which IMHO should be a completely separate project), how caching works so you could write a decent Dockerfile... And then there are features like Docker Swarm which I never touched and seem particularly complex to me. It's way too many projects stuffed into one thing.
- e12e 10y ago
  The feeling you describe reminds me of how I felt evaluating puppet and chef some years back - with a particular view to running the open source parts (not being dependent on third party support). It felt like every other link led back to the various sales teams for the enterprise solutions, and there existed no page with a straightforward overview of all the components and what puppet / chef actually did. Ansible was a breath of fresh air - as well as saltstack (but at the time it had too many holes in its security/transport story).
k__ 10y ago
On the other hand, which alternatives are easier? I'm a NixOS fan, but have to use Docker at work occasionally and I have the feeling they are both difficult, but in "their" ways.

romanovcode 10y ago

> Since the client wasn’t keen on spending on getting a more private repos, we managed with the single repo to save our base image with the most of the dependencies bundled into it. What's the point then?

swap32 10y ago
Well, most of your client are bound to do this. The point is we managed to get away with only one private repo. My guess is you'd need more than one private repo very rarely.

baconizer 10y ago

This article reflects our transition experience with Docker, it took us two years to finally feel comfortable (old chaps from decades of corp dev), but end result is very positive and rewarding. It is fair to say that this should be expected for any infrastructure migration, there is nothing wrong with being slow and careful as long as we are moving forward. Based on experience from our team, the problem has never been finding help/answers, problem is we were facing, one one hand, an encyclopedia of single page documentation like Dockerfile/docker run command, while practical guidances and gotchas are scattered around rest of teh Internet, blended with personal and business specific opinions. It is hard but eventually we figured the best way to work with containers from git to build server to deployment, that fits well with our productivity workflow;, as well as where to not use docker.

swap32 10y ago
It'd be great if you can share your learnings and increase in productivity with docker AND where not to use docker.

Walkman 10y ago

Yet another guy who doesn't know how to use Docker properly and just brags about it. There are a lot of misinformation in this blog post. The conclusion from these is not that Docker sucks, but YOU HAVE TO LEARN it. I agree that it's a very steep learning curve, but after the pieces come together, Docker solves quite a lot of problem and actually very useful.

swap32 10y ago
I think the article is very clear that this is about OUR experience of using docker. It is also stated right at the beginning that we have a business to run. If it going to take 2 years to integrate it into workflow (as it did for one of the commentors here) and hundreds of developer hours just to make it work, we have to decide whether it is viable for us. Other than that, I don't hear you refuting a single point that is made in the article. >> Yet another guy ... If there are a lot of people who are finding it difficult to grapple with it, maybe its really not their fault?
atemerev 10y ago
You have to learn it — where? Documentation is sketchy and mostly obsolete, docker-swarm is buggy, genuine production issues are routinely closed with WONTFIX resolution. Could you point me to good piece of documentation or success story of running Docker in production? (I do use Docker, but it requires to learn Kubernetes or DCOS to do anything production-related. And those are separate projects).
- fireflash38 10y ago
  Every one of these blog posts begins with almost exactly the same thing: The documentation just plain sucks. The doc parts that don't suck are outdated, and therefore useless. As long as that is the case, you are going to have people who go through the exact same troubles with Docker time and time again (one theme is databases in docker).
- cpuguy83 10y ago
  Maintainer here, can you please point to a production issue that is WONTFIX?

crb 10y ago

> Orchestration across multiple physical server gets even more nasty where you’d have to use something like Swarm. We’ve since realised that swarm is one of the lesser preferred options for orchestrating clusters. Could you elaborate on this? Did you settle on an orchestrator, and if so, which one?

swap32 10y ago
We didn't use orchestration across multiple physical machines so it was possible for us to get away with just docker-compose. Otherwise we would have had to go with swarm. Kubernetes is a way better choice than both compose and swarm but it has a even more steep learning curve.
- atemerev 10y ago
  In fact no, it doesn't. Documentation is much better, and bugs are much less.
- crb 10y ago
  I work on Kubernetes, and we're putting a lot of effort into making it easier to get started. Here are some things that might help: * minikube: http://kubernetes.io/docs/getting-started-guides/minikube/ http://kubernetes.io/docs/getting-started-guides/minikube/ * the new tutorial: http://kubernetes.io/docs/tutorials/kubernetes-basics/ http://kubernetes.io/docs/tutorials/kubernetes-basics/ * deploying in 2 steps with kubeadm: http://kubernetes.io/docs/getting-started-guides/kubeadm/ http://kubernetes.io/docs/getting-started-guides/kubeadm/ I'd love any honest feedback on how we can either (a) fix the learning curve, or (b) remove the perception that it's steep.
  - vmarsy 10y ago
    Just a quick heads up: On chrome v54.0.284.0.68 on my Android nexus 7 tablet the new tutorial page is half cut. I only see "asics" instead of what I assume is supposed to be Basics, all the text below is also cut: Overview asics through of the basics of the Kubernetes cluster module contains some background information ... There's no way to zoom out, and there's 2 hamburger menu : one white on black on the left and one blue on white on the right, both are broken The 2 other links work well

pacuna 10y ago

Kind of very basic issues IMO. The only one that's hard is logging. We are using K8s on AWS and I haven't found a good solution for logging centralization. Personally I don't like the Kibana interface for that kind of stuff.

nepotism2016 10y ago
I blame this on the way kibana is advertised. Some of web2.0 GUI with fancy color schemes and graphs. I use kibana to monitor logs over 60 containers. Sure, the initial dashboard is a pain to setup, but once you get going going its very useful
- pmontra 10y ago
  My criticism to Kibana is that it runs all the queries for a dashboard in parallel. It overloads the Elasticsearch machine(s) even when there are only few data. A small single ES server would be enough if the queries would be run sequentially. I wonder if Elastic is in the business of selling multicore 32 GB servers that sit idle most of the time. A max_parallel_queries setting would be great.

nzoschke 10y ago

Thanks for sharing your experience as another cautionary tale. At Convox we have been running Docker in prod for 18 months successfully. The secrets? 1. Don't DIY. Building a custom deployment system with any tech (Docker, Kubernetes, Ansible, Packer, etc) is a challenge. All the small problems add up to one big burden on you. 6 months later you look back at a lot of wasted time... 2. Don't use all of Docker. Images, containers and the logging drivers are all simple great. Volumes, networks and orchestration are complex. 3. Use services. Using VPC is far simpler than Docker networking. Using ECS is much easier than maintaining your own etcd or Swarm cluster. Using Cloudwatch Logs is cheaper and more reliable than deploying a logging contraption into your cluster. Use a DB service like RDS is far far easier than building your own reliable data layer. Again thanks for sharing your experience as a cautionary tale. If you are starting a new business you should not take on building a deployment system as part of the challenge. Use a well-built and peer reviewed platform like Heroku, Elastic Beanstalk or Convox.

nuschk 10y ago
Re 1: Why wouldn't you recommend going with kubernetes? From someone that hasn't deployed anything with it (yet), it seems rather straight-forward if run on a managed cluster (gke). I might be wrong and am thus genuinely interested in what you did for deployments.
- bryanlarsen 10y ago
  AFAICT, gke would be #3 (use services) not #1 (don't DIY)
  - nzoschke 10y ago
    That's right. GKE is awesome. I highly recommend using the GKE service that over running Kubernetes software on AWS or Digital Ocean.
    - nuschk 10y ago
      I'm relieved, thanks guys. I really don't see the complexity when deploying to a (managed) k8s cluster. Sure, we'll still have to configure a deployment pipeline, add some scripts to push and deploy the right images etc., but overall, k8s seems really well suited to the task.

coldcode 10y ago

Reading these articles and comment is fun, we are a very large organization and starting to use docker, and we love making things even more complex than other folks, so I can tell this is going to a big mess. Fundamentally we (as in this industry) seem to love simple ideas that become massively complex.

qaq 10y ago
Well that's how we ensure availability of employment opportunities :)
fapjacks 10y ago
What I've found is that as long as you have someone on the books that has more than a passing interest in learning about how Docker works, you'll be fine. The vast majority of problems with Docker boil down to not fully understanding the features being used. Docker is very easy to get going, and a very deep technical subject at the end of the day. There is a lot to absorb, and if you're going to throw it into production, it would behoove you to fully understand the technology. This isn't something you can expect to be without problems unless you invest in reading the documentation. This opinion isn't popular, because everybody wants to think of themselves as truly and fully understanding what they're using. But the truth is right there in the problems people run into.

pmontra 10y ago

Why is running a db in a container with the data directory mounted from the host is such a bad idea? Dropping the container on a new host with a copy of the data directory looks easier than having to install the db from scratch, even with automatic tools. Are there performance penalties at run time?

swap32 10y ago
It is fine for dev, but not for production. To quote from the link at the end of the article- "Docker is meant to be stateless. Containers have no permanent disk storage, whatever happens is ephemeral and is gone when the container stops. Containers are not meant to store data. Actually, they are meant by design to NOT store data. Any attempt to go against this philosophy is bound to disaster. Moreover. Docker is locking away processes and files through its abstraction, they are unreachable as if they didn’t exist. It prevents from doing any sort of recovery if something goes wrong" "A crash would destroy the database and affect all systems connecting to it. It is an erratic bug, triggered more frequently under intensive usage. A database is the ultimate IO intensive load, that’s a guaranteed kernel panic. Plus, there is another bug that can corrupt the docker mount (destroying all data) and possibly the system filesystem as well (if they’re on the same disk)."
- cpuguy83 10y ago
  All these statements are patently false. 1. Containers are not ephemeral. They have a lifecycle. Data written in the container is persisted to disk and available after the container is stopped and then started again. 2. Processes/files/etc are not locked away as if they don't exist. See `ps aux` on the host. You will see all the processes running. You can inspect the filesystems for each container, etc. There is no magic here. 3. A database crash could cause data corruption inside a container or not. This has nothing to do with the container, and chances of a database crash are not made worse by being in a container. That said, I would let a volume driver manage persistent storage rather than manually managing this through the host fs... but that's my preference. --- EDIT --- Disclaimer: I work at Docker Inc, and am a maintainer on the Docker project.

qaq 10y ago

Not being a fan of docker but "Since images sizes can be as high as few GB, its easy to run out of disk space. This is another problem with docker which you have to figure out yourself. Despite the fact that everyone who’s ever used docker seriously has to come across this issue sooner or later; no one tells you about this at the outset" - monitoring free disk space is something you definitely want to do regardless of using docker or not.

wccrawford 10y ago
The size of docker images ended up being the reason we aren't using it. Transfering them took forever, and so if we had an emergency update to make, we'd have to go outside our normal routines, which is brutal. I'm sure we'll end up finding another way that's similar (provides the same benefits for us) but without such crazy image sizes.
- fapjacks 10y ago
  Is your app gigabytes in size? Docker can't help you with that, but nothing else is going to help you, either. There has been a ton of momentum lately in getting Docker images pared down to miniscule (e.g. ~25MB... megabytes) image sizes with for example alpine. If you're using something like "ubuntu:14.04" for your images, this is absolutely a problem you can solve.
  - deleted 10y ago
    [deleted]
kpcyrd 10y ago
It's not that hard to work around this: I've deployed a global unit into our coreos cluster that regularly executes: #!/bin/sh while :; do # remove stopped containers w/ volumes docker ps -a -q -f status=exited | \ xargs -r docker rm -v # remove dangling images docker images -f "dangling=true" -q | \ xargs -r docker rmi sleep 6h done You need to be sure that you don't lose important data when running something like this in your setup, but it works nicely to remove old images. This script is deployed to non-coreos servers as well.

vkat 10y ago

Logging we’ve frequently experienced issues with django autoreload and delay in flushing logs >> Try using "ENV PYTHONUNBUFFERED 1" in your dockerfile

vuyani 10y ago

"we didn’t find enough reason to figure out if kubernetes was a better option." Then proceeds to name every problem kubernetes fixes

eggie5 10y ago
haha, that's exactly what I said in my comment.
swap32 10y ago
Yes. My bad! We really should have invested some time evaluating Kubernetes.
- gtaylor 10y ago
  If you don't want to stand up your own cluster for eval, fire up Google Container Engine for a few days. It's cheap and very well done.
x0x0 10y ago
I think the problem started far earlier. The author couldn't name a single clear rationale for using docker at all. One client asked them to, but why continue? They were using virtualenv and it mostly worked fine. So why docker period? Complexity for the sake of complexity? The rest of the article then basically says complex things are complex.
- swap32 10y ago
  Well, I didn't name the reasons of why we continued using Docker because I thought they were obvious to any serious docker user. The reasons are all the promises that docker makes to you without actually telling you the complexities involved. Namely - the promise of better portability, better dev-prod environment replication (You can even run vagrant for dev portability but you wouldn't use it for prod, but with docker, its just docker everywhere), better scalability with managing clusters and deployments, and so on... without actually running heavy virtual machines. And no, as per the docker promise, it wasn't supposed to be complex.

eggie5 10y ago

Orchestration: A lot of the issue I see him describe in production are fixed by Kubernetes. Compose works fine for local dev orchestration but pattern doesn’t work for deployment. The ideal world would be that I can run my compose file on my cloud provider, but Swarm isn’t there yet. I have to rewrite my compose file using kubernetes configs — it’s not a 1:1 mapping but the high level connection are there if you think of Cabernets Pods as Docker Containers. He mentions orchestration across a cluster with Swarm is nasty, but it’s elegant w/ Kubernetes. Docker Registry: Obviously, there is no constraint permitting him to use a 3rd party service. Why to let Google Container Engine (GKE) or AWS ECR handle it for you? Longer build times: I think this is really where he is missing the mark. It sounds like he has a fundamental misunderstand that if you mount the source code in dev you have to do it in prod too and that you have to have 1 container. Not true: You can mount the source code in dev using compose, so you don’t have to rebuild every time you change a line. Also, I think it’s a pattern in docker to try to keep your containers as atomic units of your app architecture. It sounds like they are trying to bake all competent of their app into 1 container (app + db + service, etc). Just break them up into containers, link them up w/ compose. This architecture them translates cleanly to one of the cloud providers for production: GKE or AWS. DB and Persistence: Yes, I think it is very clear that containers are stateless. So, yes if you want to run a DB in a container you’d have to mount an external drive somewhere. There merits and risks of that are another discussion, but as he states it’s generally frowned upon to containerize a DB. (not completely sure why, some argument about stability of container and corruption of data…) I talk more about this here: https://news.ycombinator.com/item?id=12913198 https://news.ycombinator.com/item?id=12913198 Logging: I think 12-factor style app containerized fit more smoothly into the docker compose style architecture. Accordingly if all you containers are logging to std out it, it’s all conveniently merged and printed out to the terminal if you run compose. Then on production, GKE handles it nicely too w/ the Logging system. In conclusion, I think most of his problems would been avoided if he didn’t skip researching Kubernetes and if he didn’t make the mounting oversight. The other big oversight I think he did, at least he didn't mention, is that he has no deployment tool. I wouldn't be able to effectively deploy w/o a build tool like Jenkins. I talk about a lot of these issues and how to fix them here: https://news.ycombinator.com/item?id=12860519 https://news.ycombinator.com/item?id=12860519

gigatexal 10y ago

There's a lot of "figured it out ourselves" without actually sharing what they figured out. It's like this post is a whole lot of nothing: we know docker in production is not easy but how about telling us more details

swap32 10y ago
Isn't every "figured it out ourselves" followed with the sentence describing what we figured? I see only one section where I didn't describe what we figured, which is the workflow part and I mentioned there that I would write a separate blog.

yotamoron 10y ago

Honestly, this whole post sounds like an amateur complaining about how hard it is to become a proffesional. Grow up.

swap32 10y ago
Please enlighten us with your professional wisdom and do point out which parts of the article are amateur.

kossae 10y ago

If anyone is looking into orchestration systems, I can't speak highly enough of Kontena (https://github.com/kontena/kontena https://github.com/kontena/kontena). While it is still in the early stages of development, it is a great "small-mid level" platform with a ton of features. The config and concepts are also easy to wrap your head around. We chose Kontena as Swarm is still unstable (imo not "production-ready" but I've seen counter-anecdotes) and Kubernetes was lacking some features we required. The devs have been super helpful with any problems we've had in deployment as well.

20andup 10y ago

I have been playing with docker for the past week, and its really hard to get started. I am only going to use it for integration test for now.

zwischenzug 10y ago

A lot of this was the motivation for writing the book we wrote on Docker in Practice [1]. The reality of implementing such a technology requires a combination of improvisation, technical nous, bluffing, and a willingness to work through the inevitable problems [2]. I've talked about the relative immaturity of Docker as a used system (outside of dev) [3] and am struck often by how rarely people understand that it's still a work in progress, albeit one that can massively transform your business. The hype works. That said, Docker can work fantastically in production, but you need to understand its limits and start small. [1] https://www.amazon.com/Docker-Practice-Ian-Miell/dp/1617292729 https://www.amazon.com/Docker-Practice-Ian-Miell/dp/16172927... - working on 2nd edition, if anyone has any suggestions @ianmiell [2] Blog: https://medium.com/@zwischenzugs https://medium.com/@zwischenzugs [3] https://www.youtube.com/watch?v=Yj5I-yOaDao https://www.youtube.com/watch?v=Yj5I-yOaDao

brown9-2 10y ago

We spent hours figuring out a good way to use a databases in both dev and production with docker. It was tricky since docker containers don’t support persistence unless you use a mount-point. There were a few patterns documented which didn’t work for us or we didn’t really like. We had to figure it out by ourselves. This is another area where you’re expected to figure out by yourself whether it is a good idea to use docker to run your production database. Hint- Its not. I think that a problem that leads to a lot of these articles being written is that the motivation of the authors to use Docker is unclear. Why bother to put your database inside a container if the way you ran your databases before worked just fine? You shouldn't just rush to put all your processes in containers because it is cool and Containerization Is A Good Thing. You should use technology that makes sense given the problems you are trying to solve at a give moment.

swap32 10y ago
>> Why bother to put your database inside a container if the way you ran your databases before worked just fine? For the same reason you'd bother putting anything in a container at all. The purpose of docker -as advertised by docker- is to containerize EVERY service. Unless you can point me to a single mention in official documentation that databases are an exception. Till date, I haven't found any and thus this blog.
dglass 10y ago
Completely agree. I don't know where or how people started thinking that putting a stateful service inside of a stateless container was a best practice, but no one should be doing that in production.
- allochthon 10y ago
  I wouldn't contemplate it, except that I'm looking at migrating a Rails/Postgrep app to GKE, and containerizing PG seems like the lowest-friction path. Are there better alternatives for getting to GKE, or is GKE a bad fit for this scenario?
- swap32 10y ago
  As I said, if it is really as obvious as you say it is, please point me to a single mention in official documentation that databases should not be containerized.

yahyaheee 10y ago

Sounds like you need k8s! I went though a similar progression: loving docker -> hating docker -> finding k8s