9 ms·

Azure customers running Canonical Ubuntu 18.04 experiencing DNS errors

28 points by hereiskkb 4y ago

voidfunc 4y ago
This is more of a Canonical issue than an Azure issue IMO. They pushed a broken systemd package. Is there any QA even going on at Canonical?!
- Railsify 4y ago
  systemd resolved has been broken in many releases of Ubuntu, this isn't surprising. it works most of the time but frequently will return errors.
- dspillett 4y ago
  MS push having automatic unattended updates turned on. Azure's security recs complains without end if you don't. Much like desktop Windows deciding when it should update & reboot. I see the argument for getting security updates out there ASAP, but this case proves my point that it is a bad idea to do it automatically [unless perhaps there is a serious remotely vulnerable actively exploited issue] on any sort of production environment. Deploy to test environments and verify (at very least smoke test) first then apply elsewhere. If you don't have test environments, at least do the deploy to prod/other at your control when someone is available to quickly take action regarding any unexpected issues. So the initial problem may be with Ubuntu, or upstream systemd, but MS policies magnified it significantly.
Kukumber 4y ago
How unreliable azure is, it is impressive, just like with recent version of Windows It seems like something at this company is not functioning well, it seems to be operated like it's no big deal, no QA, no quality and performance culture, it's a giant mess It has infected Github too, impressive.. Google Cloud / Microsoft Azure, they have a looooooong way to go before being able to properly compete with AWS
- JoyrexJ9 4y ago
  But the same issue would happen on AWS, it's an Ubuntu problem caused by Canonical
  - fisf 4y ago
    It's hitting loganalytics. That's not really a service for which customers deal with VMS directly.
  - dspillett 4y ago
    MS push having automatic unattended updates turned on. Azure complains without end if you don't, sometimes Defender (which Azure complains about if you don't have installed) updates override you having turned them off when it updates. There being a bug in the package may be an Ubuntu thing, me losing time to diagnosing problems caused by the update getting applied without my knowledge is very much an MS/Azure problem. At least this incident gives me fuel for strongly suggesting that we turn unattended-updates off and darn well keep it off (obviously we need to keep alerts on, but we should install updates in an orderly manner, when we can monitor the results and smoke-test, preferably applying to test resources before uat/prod/other).
- jeremycarter 4y ago
  What other cockamamie opinions do you have? You should write a book, publish it to /dev/null
josephcsible 4y ago
Buried lede: it's yet another problem caused by systemd.
- puffoflogic 4y ago
  Why anyone ever switched from GNU OS to Systemd OS is beyond me. Oh wait it's not: they were lied to and scammed.
karmakaze 4y ago
> running Ubuntu 18.04 (bionic) VMs recently upgraded to systemd version 237-3ubuntu10.54 reported experiencing DNS errors when trying to access their resources. Reports of this issue are confined to this single Ubuntu version.

28 points by hereiskkb 4y ago

This is more of a Canonical issue than an Azure issue IMO. They pushed a broken systemd package. Is there any QA even going on at Canonical?!

Railsify 4y ago
systemd resolved has been broken in many releases of Ubuntu, this isn't surprising. it works most of the time but frequently will return errors.
dspillett 4y ago
MS push having automatic unattended updates turned on. Azure's security recs complains without end if you don't. Much like desktop Windows deciding when it should update & reboot. I see the argument for getting security updates out there ASAP, but this case proves my point that it is a bad idea to do it automatically [unless perhaps there is a serious remotely vulnerable actively exploited issue] on any sort of production environment. Deploy to test environments and verify (at very least smoke test) first then apply elsewhere. If you don't have test environments, at least do the deploy to prod/other at your control when someone is available to quickly take action regarding any unexpected issues. So the initial problem may be with Ubuntu, or upstream systemd, but MS policies magnified it significantly.

Kukumber 4y ago

How unreliable azure is, it is impressive, just like with recent version of Windows It seems like something at this company is not functioning well, it seems to be operated like it's no big deal, no QA, no quality and performance culture, it's a giant mess It has infected Github too, impressive.. Google Cloud / Microsoft Azure, they have a looooooong way to go before being able to properly compete with AWS

JoyrexJ9 4y ago
But the same issue would happen on AWS, it's an Ubuntu problem caused by Canonical
- fisf 4y ago
  It's hitting loganalytics. That's not really a service for which customers deal with VMS directly.
- dspillett 4y ago
  MS push having automatic unattended updates turned on. Azure complains without end if you don't, sometimes Defender (which Azure complains about if you don't have installed) updates override you having turned them off when it updates. There being a bug in the package may be an Ubuntu thing, me losing time to diagnosing problems caused by the update getting applied without my knowledge is very much an MS/Azure problem. At least this incident gives me fuel for strongly suggesting that we turn unattended-updates off and darn well keep it off (obviously we need to keep alerts on, but we should install updates in an orderly manner, when we can monitor the results and smoke-test, preferably applying to test resources before uat/prod/other).
jeremycarter 4y ago
What other cockamamie opinions do you have? You should write a book, publish it to /dev/null

josephcsible 4y ago

Buried lede: it's yet another problem caused by systemd.

puffoflogic 4y ago
Why anyone ever switched from GNU OS to Systemd OS is beyond me. Oh wait it's not: they were lied to and scammed.

karmakaze 4y ago

> running Ubuntu 18.04 (bionic) VMs recently upgraded to systemd version 237-3ubuntu10.54 reported experiencing DNS errors when trying to access their resources. Reports of this issue are confined to this single Ubuntu version.