5 ms·
Docker Hub Is Down
- esafak 1y agoWhat's the easiest way to cache registries like docker, pypi, and npm these days?
- viraptor 1y agoYou pull the images you want to use, preferably with some automated process, then push them to your own repo. And anyways use your own repo when pulling for dev/production. It saves you from images disappearing as well.
- paulddraper 1y agoWhat do you like using for your own repo? Artifactory? Something else?
- __turbobrew__ 1y agoNote, artifactory SaaS had downtime today as well.
- GuinansEyebrows 1y agoI have experience with ECR. If you’re in the AWS ecosystem it does the job.
- paulddraper 1y agoYou've now replaced one service with another :) But maybe ECR is better than Docker Hub.
- GuinansEyebrows 1y agoyou're not wrong :) and i can't claim to have stress tested either.
- KronisLV 1y agoThere is Sonatype Nexus. A bit annoying to administer (automated cleanup works every time, 60% of the time), but supports most package formats (Maven, npm, NuGet and so on) alongside offering Docker registries, both hosted and proxy ones. Also can be deployed as a container itself.
- lambda 1y agoThe images I use the most, we pull and push to our own internal registry, so we have full control. There are still some we pull from Docker Hub, especially in the build process of our own images. To work around that, on AWS, you can prefix the image with public.ecr.aws/docker/library/ for example public.ecr.aws/docker/library/python:3.12 and it will pull from AWS's mirror of Docker Hub.
- doctoboggan 1y ago> To work around that, on AWS, you can prefix the image with public.ecr.aws/docker/library/ I believe anyone can pull from the public ecr, not just clients in AWS
- lambda 1y agoYeah, anyone can, but I think you'll be rate limited if you're not on AWS. So fine for a one off, but not something to put in your CI.
- holysoles 1y agoAnother reply had some good insight: https://news.ycombinator.com/item?id=45368092 https://news.ycombinator.com/item?id=45368092
- pm90 1y agoSomeone mentioned Artifactory; but its honestly not needed. I would very highly recommend an architecture where you build everything into a docker image and push it to an internal container registry (like ecr; all public clouds have one) for all production deployments. This way, outages only affect your build/deploy pipeline.
- deleted 1y ago[deleted]
- switz 1y agoI didn't even really realize it was a SPOF in my deploy chain. I figured at least most of it would be cached locally. Nope, can't deploy. I don't work on mission-critical software (nor do I have anyone to answer to) so it's not the end of the world, but has me wondering what my alternate deployment routes are. Is there a mirror registry with all the same basic images? (node/alpine) I suppose the fact that I didn't notice before says wonderful things about its reliability.
- tom1337 1y agoI guess the best way would be to have a self-hosted pull-through registry with a cache. This way you'd have all required images ready even when dockerhub is offline. Unfortunately that does not help in an outage because you cannot fill the cache now.
- pebble 1y agoThis is the way tho this can lead to fun moments like I was just setting up a new cluster and couldn't figure out why I was having problems pulling images when the other clusters were pulling just fine. Took me a while to think of checking the docker hub status page.
- cipherself 1y agoIn the case where you still have an image locally, trying to build will fail with an error complaining about not being able to load metadata for the image because a HEAD request failed. So, the real question is, why isn't there a way to disable the HEAD request for loading metadata for images? Perhaps there's a way and I don't know it.
- philip1209 1y agoDevelopment environment won't boot. Guess I'll go home early.
- deleted 1y ago[deleted]
- juan16 1y agohave same problem, visiting https://hub.docker.com/_/node https://hub.docker.com/_/node return error
- gnabgib 1y agoDupe https://news.ycombinator.com/item?id=45366942 https://news.ycombinator.com/item?id=45366942
- cipherself 1y agoI’ll admit I haven’t checked before posting, perhaps an admin can merge both submissions and change the URL on the one you linked to the one in this submission.
- XCSme 1y agoYup, my Coolify deployments were failing and I didn't know why : https://softuts.com/docker-hub-is-down/ https://softuts.com/docker-hub-is-down/ Also, isn't it weird that it takes so long to fix given the magnitude of the issue? Already down for 3 hours.
- taberiand 1y agoSo that's why. This gave me the kick I needed to finally switch over the remaining builds to the pull-through cache.
- deleted 1y ago[deleted]
- miller_joe 1y agoI was hoping google cloud artifact registry pull-thru caching would help. Alas, it does not. I can see an image tag available in the cache in my project on cloud.google.com, but after attempting to pull from the cache (and failing) the image is deleted from GAR :(
- qianli_cs 1y agoI think it was likely caused by the cache trying to compare the tag with Docker Hub: https://docs.docker.com/docker-hub/image-library/mirror/#what-if-the-content-changes-on-the-hub https://docs.docker.com/docker-hub/image-library/mirror/#wha... > "When a pull is attempted with a tag, the Registry checks the remote to ensure if it has the latest version of the requested content. Otherwise, it fetches and caches the latest content." So if the authentication service is down, it might also affect the caching service.
- jiggawatts 1y agoEven cloud vendors can’t get distributed systems design right.
- breatheoften 1y agoIn our ci setting up the docker buildx driver to use the artifact registry pull through cache involves (apparently) an auth transaction to dockerhub which fails out
- rshep 1y agoI’m able to pull by the digest, even images that are now missing a tag.
- Poomba 1y agoIs there a good alternative for DockerHub these days? Besides azure CR
- akerl_ 1y agoBasically all my Docker images were being built from Github repos anyways, so I just switched to Github's container registry.
- cyberax 1y agoQuay.io is nice (but you have to memorize the spelling of its name)
- viraptor 1y agoOr start a pronunciation revolution and say "kway". It's all made up anyway ;-)
- cyberax 1y agoIt _is_ pronounced "kway", and it _is_ a real word: https://www.merriam-webster.com/dictionary/quay https://www.merriam-webster.com/dictionary/quay !
- gnabgib 1y agoIt's pronounced keɪ (from your link - The spelling quay, first appearing in the sixteenth century, follows modern French. As noted by the Oxford English Dictionary, third edition, the expected outcome of Middle English keye would be /keɪ/ in Modern English). Or key (with modern spelling).
- frabonacci 1y agoDuplicate https://news.ycombinator.com/item?id=45366942 https://news.ycombinator.com/item?id=45366942
- momeabed 1y agoAlso GCP K8S have an partial outage! was this vibe coded release... insane...
- wolttam 1y agoAll I really need is for Debian to have their own OCI image registry I can pull from. :)
- lambda 1y agoNot Debian itself, but Red Hat's registry has them: https://quay.io/organization/lib https://quay.io/organization/lib
- manasdas 1y agoTherefore keep a local registry mirror. You will get it from local cache all the time.
- minsuseol 1y ago[dead]
- supravg 1y ago[dead]
- hexagonsun 1y agoexplains why my watchtower container was exploding
- Cameri 1y agosame
- thomasfromcdnjs 1y agoWas already struggling to do any work today and now my builds aren't working. https://xkcd.com/303/ https://xkcd.com/303/
- minsuseol 1y ago[dead]
- thehamkercat 1y agoI had some images in cache, but not all of them, and pull is failing for example, i have redis:7.2-alpine in cache, but not golang:1.24.5-alpine I needed the golang image to start my dev-backend so i replaced FROM golang:1.24.5-alpine with FROM redis:7.2-alpine, and manually installed golang with apk in the redis container :)
- zelphirkalt 1y agoYou changed your base image and docker build process for a temporary outage? Or do you mean that this in general will be better, as you avoid one in-between image?
- thehamkercat 1y agoI changed just for that outage, because i couldn't pull the other image
- __turbobrew__ 1y agoAnyone have recommendations for an image cache? Native kubernetes a plus. What would be really nice is a system with mutating admission webhooks for pods which kicks off a job to mirror the image to a local registry and then replaces the image reference with the mirrored location.
- andrewstuart2 1y agoCNCF has harbor [0], which I use at home and have deployed in a few clusters at work, and it works well as a pull through cache. In /etc/containers/registries.conf it's just another line below any registry you want mirrored. [[registry]] location = "docker.io" [[registry.mirror]] location = "core.yourharbor.example.com/hub" Where hub is the name of the proxy you configured for, in this case, docker.io. It's not quite what you're asking for but it can definitely be transparent to users. I think the bonus is that if you look at a podspec it's obvious where the image originates and you can pull it yourself on your machine, versus if you've mutated the podspec, you have to rely on convention. [0] https://goharbor.io/ https://goharbor.io/
- alias_neo 1y agoI would add, for anyone not familiar with it, that this (and more advanced mirroring, etc) is just as easily done from the really nice Web UI (if that's your cup of tea).
- andrewstuart2 1y agoYeah, to clarify, I had to first set up /hub as a caching proxy using the UI. Then the above configuration change was what I needed on my nodes in order to transparently use the proxy without changing podspecs.
- esseph 1y agoSeconding harbor
- da768 1y agoNot Google Artifact Registry... Our Docker Hub pull-through mirror went down with the Docker Hub outage. Images were still there but all image tags were gone
- ansraliant 1y ago[dead]
- sublinear 1y agoSomewhat unrelated, but GitLab put out a blog post earlier this year warning users about Docker Hub's rate limiting: https://about.gitlab.com/blog/prepare-now-docker-hub-rate-limits-will-impact-gitlab-ci-cd/ https://about.gitlab.com/blog/prepare-now-docker-hub-rate-li... We chose to move to GitLab's container registry for all the images we use. It's pretty easy to do and I'm glad we did. We used to only use it for our own builds. The package registry is also nice. I only wish they would get out of the "experimental" status for apt mirror support.
- thehamkercat 1y agoIt's up now, can pull images
- c0balt 1y agoExceeded their quota, probably, based on my recent experience with dockerhub
- blitzar 1y agoTech support needs to log the server in to its account to get a bigger quota.
- deleted 1y ago[deleted]
- zenmac 1y agoThis is one of the reasons I don't want to use docker on production machines and have started to use systemd again!!
- Too 1y agoHard to see if this is /s or not. Nobody is forcing you to run images straight from dockerhub lol. Every host keeps the images already on it. Running a in-house registry is also a good idea.
- blitzar 1y agoAt a reasonably modest scale - running an in-house registry is a polite thing to do for the rest of the internet.
- gvkhna 1y agoGithub actions buildx also going down is a really unintended consequence. It would be great if we could mirror away from docker entirely at this point but I digress.
- tfolbrecht 1y agoThere's a registry image for OCI containers that is pretty painless to set up and low maintenance, can use s3 as a storage backend. https://hub.docker.com/_/registry https://hub.docker.com/_/registry Your git provider probably also has a container registry service built in.
- rickette 1y agoWell to be fair: this doesn't happen very often. It's quite a stable service in my experience.
- s_ting765 1y agoStatus report says issue with authentication fixed but it's far worse than that. This incident also took down docker pull for public images with it.
- MASNeo 1y agoFor what it's worth, my debugging made me install the latest docker version. So the outage is good for something ;-)