9 ms·

Where are Docker images stored?

82 points by thoward37 13y ago

mey 13y ago
Is there any way to run your own Docker repository? Edit: Looks like the correct term is registry.
- deleted 13y ago
  [deleted]
- jimrhoskins 13y ago
  Yes, assuming you mean your own docker registry. Docker has open sourced https://github.com/dotcloud/docker-registry https://github.com/dotcloud/docker-registry Remember the registry deals with the actual data, and delegates auth and other stuff to an index. The docker-registry has a dummy implementation of an index that has no notion of authentication or authorization, so anything you push to your private registry is really public if you don't secure it with some other method. For me, I wanted to have a real private registry with access control limited to my team. It turned out to not be too difficult to make our own registry+index implementation that is private by default. It has a basic web interface too. I've open sourced development of it https://github.com/jimrhoskins/stevedore https://github.com/jimrhoskins/stevedore . It's still really rough, especially on the web interface stuff, but for push/pull operations with required authentication, it does the job now.
- chasb 13y ago
  You can fork the registry code [1]. We work in a regulated industry (healthcare) and need excellent access controls and auditing/logging. Forking the registry and rolling our own is possible, but not something we want to spend time on. So we looked around for private registries. We went with Quay [2] and have been really happy. They're responsive, performant, and on top of enterprise-level requirements. [1] http://blog.docker.io/2013/07/how-to-use-your-own-registry/ http://blog.docker.io/2013/07/how-to-use-your-own-registry/ [2] https://quay.io/ https://quay.io/
deleted 13y ago
[deleted]
peterwwillis 13y ago
Holy crap this is complicated. (how docker works, not your write-up)
j_s 13y ago
(1) Who pays the bills for the public registry (docker.io) and why? (2) Is there a future possibility similar to how so many Ruby projects fall down when GitHub goes down due to not 'pit-of-success-ing'[1] a copy of everything locally? [1] http://blogs.msdn.com/b/brada/archive/2003/10/02/50420.aspx http://blogs.msdn.com/b/brada/archive/2003/10/02/50420.aspx
- ceejayoz 13y ago
  (1) Docker, Inc., who plan to sell paid support, training, and services (think Canonical/Ubuntu).
- sergiotapia 13y ago
  Why do you '[1]' links?
  - deleted 13y ago
    [deleted]
  - gknoy 13y ago
    Part of HN's convention, especially since there are often more than one link, is to use end-noted links rather than inline ones. This makes the prose easier to read, and still makes it easy to identify and annotate the importance of links.
    - derefr 13y ago
      I kind of wonder why HN doesn't support [Markdown-style](http://daringfireball.net/projects/markdown/syntax#link http://daringfireball.net/projects/markdown/syntax#link) links. It almost makes me miss Reddit. I suppose it's to make it clear where links go, and discourage trolling... but browsers failing to show you where links go is a failing of browser chrome, and people who are concerned about that can install extensions that make link destinations more obvious.
  - j_s 13y ago
    I'm to lazy to do the classier footnote¹ most of the time.
yackob03 13y ago
Images may also be stored in one of the available private registries. <shameless plug> As the Co-Founder, I am partial to Quay.io [https://quay.io https://quay.io], which in my not so humble opinion has the best features, reliability, and support for businesses and organizations, but there are other options if for some reason Quay.io doesn't meet your needs. For those who prefer to self-host, we've also got an enterprise option, which brings all of the index and registry goodness behind your firewall. </shameless plug> That said, we love the Docker ecosystem and way of doing things. A sibling comment mentioned how complicated Docker is, but I think when you realize that they are trying to offer DVCS like features and paradigms, you will realize that it is complicated for a reason. We all thought git was complicated at first as well.
- chasb 13y ago
  Quay customer here - these guys are fantastic.
- thoward37 13y ago
  Regarding the shameless product plug: Quay looks like a very cool product. Love the history and diff views. Glad to see pricing mimics Github model "pay for private, but public is free and unlimited". Awesome! Regarding complexity in Docker: So here's the thing, people wanted npm, but they got git. How can we bridge the gap between a easy to use, out of your way package manager and a fully featured DVCS experience? I love the idea of merging them, but IMO, need to make the semantic model more accessible. Specifically, need to ensure concepts are properly orthogonal, not overloaded, and unambiguously defined. Might be too late to scrub this aspect though. Some other general problems are things like checksums, fingerprints, image signing, etc. How to verify the validity of an image?
  - yackob03 13y ago
    I will speak to the issues about which I am familiar. Checksums are currently uploaded by the client and verified by the registry. Signing is on the roadmap[1]. I'm not sure what you mean by a fingerprint, would this be analogous to an SSH host key? What function would it serve if you already had a signature that only you could reproduce? [1]: https://github.com/dotcloud/docker/issues/2700 https://github.com/dotcloud/docker/issues/2700
avgp 13y ago
"So why did it say that? I have no idea, but you can ignore it." - because it is "uploaded" from the CLI tool to the daemon. See http://docs.docker.io/en/latest/use/builder/#usage http://docs.docker.io/en/latest/use/builder/#usage
derefr 13y ago
To put it another way: 1. a Docker registry is like (or maybe just is) an S3 bucket: a dumb, private object-store. 2. A docker index is a database-backed web service with a REST API, that clients talk to. 3. The web service can generate temporary tokens that let you GET things from, and PUT things in, the bucket. 4. The web service's database has a model of an image "project" similar to a Git repository: version history, branches, and other metadata. 5. The bucket contains the image repository's "object pool." Just like git, when you pull a branch, the client downloads all the "objects" required to check out that branch.
bstar77 13y ago
I've started using quay.io to store my docker images. I have not used the service heavily (because I'm still building out my Dockerfiles), but what I have used has been great. Docker.io is great to use to store images, unless you don't want them public. I have proprietary apps loaded on my images so making them public is not an option. So far Docker is set to solve my scalability problem that i've been seeking for the past year. Since VMs are not ideal, I start with a farm of bare ubuntu servers and scale out to VM's in the cloud if needed. With Docker I can configure once and deploy to all of these nodes no matter how they were built. I stopped using Chef when I realized I could accomplish my goals with a fraction of the complexity and effort.
hoprocker 13y ago
Great writeup. I always appreciate somebody else doing a down-to-earth overview of something complicated. This post makes me wonder if some sort of flowchart or UML diagram of the docker system components wouldn't be a useful thing.

82 points by thoward37 13y ago

mey 13y ago

Is there any way to run your own Docker repository? Edit: Looks like the correct term is registry.

deleted 13y ago
[deleted]
jimrhoskins 13y ago
Yes, assuming you mean your own docker registry. Docker has open sourced https://github.com/dotcloud/docker-registry https://github.com/dotcloud/docker-registry Remember the registry deals with the actual data, and delegates auth and other stuff to an index. The docker-registry has a dummy implementation of an index that has no notion of authentication or authorization, so anything you push to your private registry is really public if you don't secure it with some other method. For me, I wanted to have a real private registry with access control limited to my team. It turned out to not be too difficult to make our own registry+index implementation that is private by default. It has a basic web interface too. I've open sourced development of it https://github.com/jimrhoskins/stevedore https://github.com/jimrhoskins/stevedore . It's still really rough, especially on the web interface stuff, but for push/pull operations with required authentication, it does the job now.
chasb 13y ago
You can fork the registry code [1]. We work in a regulated industry (healthcare) and need excellent access controls and auditing/logging. Forking the registry and rolling our own is possible, but not something we want to spend time on. So we looked around for private registries. We went with Quay [2] and have been really happy. They're responsive, performant, and on top of enterprise-level requirements. [1] http://blog.docker.io/2013/07/how-to-use-your-own-registry/ http://blog.docker.io/2013/07/how-to-use-your-own-registry/ [2] https://quay.io/ https://quay.io/

deleted 13y ago

[deleted]

peterwwillis 13y ago

Holy crap this is complicated. (how docker works, not your write-up)

j_s 13y ago

(1) Who pays the bills for the public registry (docker.io) and why? (2) Is there a future possibility similar to how so many Ruby projects fall down when GitHub goes down due to not 'pit-of-success-ing'[1] a copy of everything locally? [1] http://blogs.msdn.com/b/brada/archive/2003/10/02/50420.aspx http://blogs.msdn.com/b/brada/archive/2003/10/02/50420.aspx

ceejayoz 13y ago
(1) Docker, Inc., who plan to sell paid support, training, and services (think Canonical/Ubuntu).
sergiotapia 13y ago
Why do you '[1]' links?
- deleted 13y ago
  [deleted]
- gknoy 13y ago
  Part of HN's convention, especially since there are often more than one link, is to use end-noted links rather than inline ones. This makes the prose easier to read, and still makes it easy to identify and annotate the importance of links.
  - derefr 13y ago
    I kind of wonder why HN doesn't support [Markdown-style](http://daringfireball.net/projects/markdown/syntax#link http://daringfireball.net/projects/markdown/syntax#link) links. It almost makes me miss Reddit. I suppose it's to make it clear where links go, and discourage trolling... but browsers failing to show you where links go is a failing of browser chrome, and people who are concerned about that can install extensions that make link destinations more obvious.
- j_s 13y ago
  I'm to lazy to do the classier footnote¹ most of the time.

yackob03 13y ago

Images may also be stored in one of the available private registries. <shameless plug> As the Co-Founder, I am partial to Quay.io [https://quay.io https://quay.io], which in my not so humble opinion has the best features, reliability, and support for businesses and organizations, but there are other options if for some reason Quay.io doesn't meet your needs. For those who prefer to self-host, we've also got an enterprise option, which brings all of the index and registry goodness behind your firewall. </shameless plug> That said, we love the Docker ecosystem and way of doing things. A sibling comment mentioned how complicated Docker is, but I think when you realize that they are trying to offer DVCS like features and paradigms, you will realize that it is complicated for a reason. We all thought git was complicated at first as well.

chasb 13y ago
Quay customer here - these guys are fantastic.
thoward37 13y ago
Regarding the shameless product plug: Quay looks like a very cool product. Love the history and diff views. Glad to see pricing mimics Github model "pay for private, but public is free and unlimited". Awesome! Regarding complexity in Docker: So here's the thing, people wanted npm, but they got git. How can we bridge the gap between a easy to use, out of your way package manager and a fully featured DVCS experience? I love the idea of merging them, but IMO, need to make the semantic model more accessible. Specifically, need to ensure concepts are properly orthogonal, not overloaded, and unambiguously defined. Might be too late to scrub this aspect though. Some other general problems are things like checksums, fingerprints, image signing, etc. How to verify the validity of an image?
- yackob03 13y ago
  I will speak to the issues about which I am familiar. Checksums are currently uploaded by the client and verified by the registry. Signing is on the roadmap[1]. I'm not sure what you mean by a fingerprint, would this be analogous to an SSH host key? What function would it serve if you already had a signature that only you could reproduce? [1]: https://github.com/dotcloud/docker/issues/2700 https://github.com/dotcloud/docker/issues/2700

avgp 13y ago

"So why did it say that? I have no idea, but you can ignore it." - because it is "uploaded" from the CLI tool to the daemon. See http://docs.docker.io/en/latest/use/builder/#usage http://docs.docker.io/en/latest/use/builder/#usage

derefr 13y ago

To put it another way: 1. a Docker registry is like (or maybe just is) an S3 bucket: a dumb, private object-store. 2. A docker index is a database-backed web service with a REST API, that clients talk to. 3. The web service can generate temporary tokens that let you GET things from, and PUT things in, the bucket. 4. The web service's database has a model of an image "project" similar to a Git repository: version history, branches, and other metadata. 5. The bucket contains the image repository's "object pool." Just like git, when you pull a branch, the client downloads all the "objects" required to check out that branch.

bstar77 13y ago

I've started using quay.io to store my docker images. I have not used the service heavily (because I'm still building out my Dockerfiles), but what I have used has been great. Docker.io is great to use to store images, unless you don't want them public. I have proprietary apps loaded on my images so making them public is not an option. So far Docker is set to solve my scalability problem that i've been seeking for the past year. Since VMs are not ideal, I start with a farm of bare ubuntu servers and scale out to VM's in the cloud if needed. With Docker I can configure once and deploy to all of these nodes no matter how they were built. I stopped using Chef when I realized I could accomplish my goals with a fraction of the complexity and effort.

hoprocker 13y ago

Great writeup. I always appreciate somebody else doing a down-to-earth overview of something complicated. This post makes me wonder if some sort of flowchart or UML diagram of the docker system components wouldn't be a useful thing.