5 ms·

I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50

I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50mb is a worthwhile trade off to avoid hard to debug musl-libc issues. eg https://www.linkedin.com/pulse/musl-libc-alpines-greatest-weakness-rogan-lynch/ https://www.linkedin.com/pulse/musl-libc-alpines-greatest-we...

leetrout 5y ago
What about busybox? It still has a glibc version doesnt it?
sk5t 5y ago
Yep. I had an (immutable) JVM ECS service that worked fine for a year, then started failing DNS in just one of several AWS AZs and for just one of various host names--one with something like 24 A records, a few but not an outrageous number. Occasionally forking a process to run 'dig' on the same name made it work for a little while. AWS support's only advice was "don't use Alpine"; annoyingly, switching the containers to a Debian base cured it, even though this would appear to make absolutely no sense with respect to it failing in just one AZ.
- michaeldwan 5y ago
  Failing in one AZ probably means they changed something with DNS servers for that region. We had a similar issue recently when we rolled out a new DNS server that returned longer SOA records which broke python's mysql driver in only some regions during the rollout. Debugging nightmare fuel.
  - sk5t 5y ago
    Yes, my theory was also some DNS variance in the AZ, but AWS stolidly refused to supply any information to that effect. Adding 'dig +trace' to the equation only deepened the mystery.
sylvain_kerkour 5y ago
Interesting! Thank you for sharing. Did you notice that some language are more affected than others? For example Go vs. Node.js?
- michaeldwan 5y ago
  Node, Ruby, and Python come to mind. Oddly enough the alpine versions of haproxy and nginx on Docker hub had issues too.
  - amjd 5y ago
    Any insight on Laravel/PHP? I've been seeing DNS resolution issues with Alpine as well, but they seem to mainly affect PHP services and not Go.
nikolay 5y ago
Also, isn't Python problematic with musl-libc [0]? [0]: https://pythonspeed.com/articles/alpine-docker-python/ https://pythonspeed.com/articles/alpine-docker-python/
bogantech 5y ago
The person responsible for musl's DNS support thinks broken DNS is good UX and purposefully didn't add tcp upgrade support to it. Note that some DNS resolvers do not provide truncated UDP results apparently so that might explain some of the weird DNS issues people see https://twitter.com/RichFelker/status/994629795551031296 https://twitter.com/RichFelker/status/994629795551031296

leetrout 5y ago

What about busybox? It still has a glibc version doesnt it?

sk5t 5y ago

Yep. I had an (immutable) JVM ECS service that worked fine for a year, then started failing DNS in just one of several AWS AZs and for just one of various host names--one with something like 24 A records, a few but not an outrageous number. Occasionally forking a process to run 'dig' on the same name made it work for a little while. AWS support's only advice was "don't use Alpine"; annoyingly, switching the containers to a Debian base cured it, even though this would appear to make absolutely no sense with respect to it failing in just one AZ.

michaeldwan 5y ago
Failing in one AZ probably means they changed something with DNS servers for that region. We had a similar issue recently when we rolled out a new DNS server that returned longer SOA records which broke python's mysql driver in only some regions during the rollout. Debugging nightmare fuel.
- sk5t 5y ago
  Yes, my theory was also some DNS variance in the AZ, but AWS stolidly refused to supply any information to that effect. Adding 'dig +trace' to the equation only deepened the mystery.

sylvain_kerkour 5y ago

Interesting! Thank you for sharing. Did you notice that some language are more affected than others? For example Go vs. Node.js?

michaeldwan 5y ago
Node, Ruby, and Python come to mind. Oddly enough the alpine versions of haproxy and nginx on Docker hub had issues too.
- amjd 5y ago
  Any insight on Laravel/PHP? I've been seeing DNS resolution issues with Alpine as well, but they seem to mainly affect PHP services and not Go.

nikolay 5y ago

Also, isn't Python problematic with musl-libc [0]? [0]: https://pythonspeed.com/articles/alpine-docker-python/ https://pythonspeed.com/articles/alpine-docker-python/

bogantech 5y ago

The person responsible for musl's DNS support thinks broken DNS is good UX and purposefully didn't add tcp upgrade support to it. Note that some DNS resolvers do not provide truncated UDP results apparently so that might explain some of the weird DNS issues people see https://twitter.com/RichFelker/status/994629795551031296 https://twitter.com/RichFelker/status/994629795551031296