9 ms·

Ask HN: Cloudflare broke my domain's DNSSEC making it unreachable since 4 days

139 points by medguru 4y ago

tl;dr - Cloudflare rendered my domain inaccessible and support has been ignoring the ticket for 4 days, what's the fastest way to get technical assistance when on a free plan? Last week I transferred a domain used for a personal project from my old registrar to Cloudflare. After the transfer was finalized and new NS records had propagated, everything resolved normally and everything was working fine. I then enabled DNSSEC, and after a while the domain would no longer resolve. Every DNS server I try - Google, Quad9, OpenDNS, even Cloudflare's own DNS on 1.1.1.1 - returns SERVFAIL. The excellent diagnostic tool on dnsviz.net tells me that the domain is returning bogus DNSKEY/DS/NSEC responses and bogus delegation status. "no SEP matching the DS found". I tried canceling the DNSSEC setup and waiting for over a day, with no effect. I re-enabled DNSSEC setup and waited for 3 days, with no effect. Cloudflare's control panel has since several days now been saying that DNSSEC will be enabled "in the next 24 hours". My site cannot be reached, and Cloudflare's support cannot be reached. I've been forced to migrate the project and its (few) users to a completely different domain. I cannot inconvenience users by bouncing them back and forth, so the domain Cloudflare ruined for me is now effectively lost, as is the "branding" of the project which was reflected in the domain's name. How can I get their attention without paying for an Enterprise plan? I would like to think that basic functional service should be accessible even when using Cloudflare only as a registrar with fundamental DNS on a free plan.

wnoise 4y ago
English language usage note: "since" takes a past point in time, not a duration. You want either "unreachable for 4 days", or "unreachable since 4 days ago".
InitialBP 4y ago
Sorry to hear about your problem, a quick recommendation would be to keep the temporary DNS name you bought and simply redirect to your previous name once you have the issue resolved (or vice versa if the branding is less important to you.) This way your users won't need to know or care about the change anymore aside from this temporary setback.
scrollaway 4y ago
> what's the fastest way to get technical assistance when on a free plan? Upgrading to a non-free plan? You don't have to upgrade to enterprise, but even their $20/mo plan comes with support. (Also, I hate to victim-blame here but using DNSSEC was a bad idea in the first place)
- tmikaeld 4y ago
  Why was enabling DNSSEC a bad idea? Clearly the origin registrar isn't handling DNSSEC requests properly, but the OP should still be able to revert to non-DNSSEC without issues.
  - pixl97 4y ago
    Wasn't there a thread on HN within the last couple of months that says switching back when things go wrong is actually very difficult? Intermediate servers or something cache the DNSSEC issues and things tend to break for 24 hours at a time. Unfortunately I can't remember the details.
  - scrollaway 4y ago
    This post makes a better argument than I could write. And nothing much has changed in the seven years since. https://sockpuppet.org/blog/2015/01/15/against-dnssec/ https://sockpuppet.org/blog/2015/01/15/against-dnssec/
    - mattashii 4y ago
      That article is a bad example of outdated arguments and whataboutisms. Sure, DNSSEC has some issues, but none are so bad that it means you shouldn't use it: It does help resolvers detect various attacks (e.g. cache poisoning, BGP hijacks, MitMs), which is fairly critical for security-minded organizations. (from that article, that is from 2015 and woefully outdated) > With TLS properly configured, DNSSEC adds nothing. This is false. DNSSEC adds address lookup security through response integrity, whereas TLS (only!) adds transport layer security to the endpoint you're connected to (hence the name). If you find a record in DNS with DNSSEC enabled, you know that the response is exactly as the sender intended it to be (and when connecting to the address returned for A- or AAAA-records, you'll be connecting to the intended IP address). Without DNSSEC, this is impossible to guarantee and record interception / MitM would be an attack vector. Additionally, "With TLS correctly configured" also implies CAA being set up, which only can be done securely using DNSSEC. As to why CAA must be configured: Not all CAs are made the same; and re-routing network traffic is fairly doable if you only need to target one of the many public CAs. Targeting only that CA allowed in the CAA record is presumably much harder. > Securing DNS lookups isn’t a high-priority task > DNSSEC’s real job is thus to replace the TLS CA system. This plan is called DANE. No, DNSSEC's job _is_ to secure DNS lookups. DANE is only one scheme that is made possible by DNSSEC; Secure CAA checking being another. > Real-world DNSSEC therefore relies on RSA with PKCS1v15 padding. Correct, but also relies on Ed25519 and P-256. A lot of authorative servers are still using the legacy RSA keys, but another lot is using P-256 and Ed25519 too. > [sections] DNSSEC is Expensive To Adopt / Deploy This is partly true, but any security is expensive to adopt/deploy. DNSSEC is fairly easy nowadays, though, with many hosted DNS services providing some form of DNSSEC. > DNSSEC doesn’t secure browser DNS lookups. It would, if you allowed your browser to recurse. > DNSSEC is Unsafe > Authenticated denial. Offline signers. Secret hostnames. Pick two. That's fine. Secrecy doesn't add security; Authenticated denial and Offline signers do. > DNSSEC is Architecturally Unsound I disagree with the conclusion here. Sure, it might be useful for US gov to be the writer of the spec, but what public scrutiny DNSSEC has had implies that the security part is sound.
b3lvedere 4y ago
-- removed. My apologies. --
- medguru 4y ago
  My domain and users have ended up in limbo beyond anyone's but Cloudflare's control. I cannot transfer it back to the working registrar, or I would without being "angry at some free service". Why do you think berating me with snide remarks is helpful?
  - codatory 4y ago
    When it comes to dnssec, the keys are handed down through the same layer of delegation that gives you your nameservers. This is out of Cloudflare's hands as well, you have bad data up through the registry and out to (your tld's) root nameservers. Just like changes to your NS record delegation don't take immediate effect, these keys don't take immediate effect either. It's still pretty early days for DNSSEC, if you're going to use it it's worthwhile to know a lot about it. Just look at the several Slack outages caused by their attempts to implement it. Eventually the tooling will catch up, and registrars will all give you warnings about moving DNS and registration and the importance of syncing up your keys but we just aren't there yet.
  - b3lvedere 4y ago
    My apologies if you read it as a snide remark. I'm usually baffled when support is demanded on free stuff, but i see your point in this particular scenario.
    - tedivm 4y ago
      No one "read it as a snide remark"- you were snide and rude. Just own it and apologize.
      - b3lvedere 4y ago
        I did and i removed the parent comment. If you'd like me to delete everything, i'd be happy to oblige.
elithrar 4y ago
(It sucks that I had to see this on HN) Can you email me - silverlock at cloudflare - with your ticket ID and domain name so I can understand what broke?
- medguru 4y ago
  Thank you for the attention, e-mail on its way.
  - gigatexal 4y ago
    Come back and tell us what happened and the resolution.
    - medguru 4y ago
      https://news.ycombinator.com/context?id=31413856 https://news.ycombinator.com/context?id=31413856
  - elithrar 4y ago
    Just replied. Ultimately it looks like the existing DS records for your domain weren't removed (and you can see that in your DNSViz output). Still have some questions for "how" it was working beforehand (see the email for those). For others: I'll let the OP share what details they would like to, as this is their domain.
- quinncom 4y ago
  Is there a way to pay for priority support while on a free plan? I'm nervous about migrating a domain to CF which is used for glue records, and want to have immediate support access if something goes wrong.
  - elithrar 4y ago
    The $20/mo Pro plan includes formal support (one of the big perks of Pro!)
jSherz 4y ago
Does your TLD definitely support DNSSEC?
- medguru 4y ago
  Yes. It had DNSSEC enabled for over a year when it was with the old registrar.
  - phillipseamore 4y ago
    You would usually need to disable DNSSEC, wait 24h, transfer, and then wait for at least 24h before enabling DNSSEC again.
    - medguru 4y ago
      Thanks for the info, I'll keep it in mind for eventual future transfers. But shouldn't I be able to disable DNSSEC regardless, instead of the domain being stuck in limbo and hijacked by what appears to be a deadlock type of bug?
      - belorn 4y ago
        Talking as a admin at a registrar, Yes, Indeed Yes, you should be able to disable DNSSEC regardless. DNSSEC signed is basically just that the TLD servers has a DS record listed for the domain. In order to remove dnssec you remove the DS record. This can be easy or hard depending on the interface that the TLD, but in theory very simple. The reason why its recommended to remove dnssec before transfer is to allow caches to timeout with the old DS record to expire. Some TLD also automatically remove DS when you do a transfer and a name server change, as it is a rather clear signal that the old key won't be useful. There is however some exciting new technology called multi-signer which is intended to resolve this problem in the future.
      - phillipseamore 4y ago
        Disabling DNSSEC doesn't propagate instantly. Have you queried the CF nameservers for the domain directly? In my experience everything involving DNSSEC requires a 24h wait (unless the domain hasn't been queried from anywhere - but that's usually not the case, something might have triggered distributed DNS lookups e.g. LE doing DNS validation for cert issuance etc).
pteraspidomorph 4y ago
I had a problem with a similar effect some time ago but I run my own DNS (no Cloudflare). I accidentally clicked a button in my control panel to regenerate the zone keys, which means the published keys mismatched the new zone signature for a couple of days until I was able to get the registrar to update them and everything propagated (even when a registrar supports the .eu TLD they are usually severely lacking in automation). The control panel devs have since added a confirmation dialog!
williamtwild 4y ago
>I've been forced to migrate the project and its (few) users to a completely different domain. I cannot inconvenience users by bouncing them back and forth, so the domain Cloudflare ruined for me is now effectively lost, as is the "branding" of the project which was reflected in the domain's name. If this was that important then you should not have used the free plan.
- medguru 4y ago
  Why do you presume the issue would have gotten immediate attention for the sum of $20? Customers don't make Cloudflare's terms, and customers didn't decide for Cloudflare to offer a free plan with zero markup for their registrar operations. There is by users' own hands no way out of domain registration issues like these, sooner than 30-45 days when the domain can be transferred once again. Those who decide to offer registrar services, even for free, must hold some liability towards the users and the ecosystem and offer some support to make sure their product actually works.
  - Spunkie 4y ago
    They are also the only domain registrar that I've interacted with this decade that does not allow you to set your own nameservers. Effectively locking you into using cloudflare DNS and related services until you can transfer again. Its frankly, disgusting.
xbar 4y ago
dnsviz.net is awesome.
oneplane 4y ago
> How can I get their attention without paying for an Enterprise plan? By paying for the cheapest plan, or any plan at all for that matter.
_wldu 4y ago
DNSSEC is notorious for breaking things [1]. I use it on most of my domains, but I would not just 'enable' it on a domain that I cared about and that had real users without a lot of thought and planning. Nor should you. [1] - https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html
- medguru 4y ago
  I figured it would work since there were no problems for the handful of my domains using DNSSEC with the previous registrar. Maybe the button should come with a warning label. I'll certainly be a bit cautious from now on.
- paulnpace 4y ago
  > DNSSEC is notorious for breaking things My understanding is that people break things.
  - ceejayoz 4y ago
    https://en.wikipedia.org/wiki/Just_culture https://en.wikipedia.org/wiki/Just_culture > Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved. In a just culture, after an incident, the question asked is, "What went wrong?" rather than "Who caused the problem?". Prominent (and very effective) example: Aviation safety. DNSSEC is both easy to break and hard to fix. https://sockpuppet.org/blog/2015/01/15/against-dnssec/ https://sockpuppet.org/blog/2015/01/15/against-dnssec/
    - marcosdumay 4y ago
      There's no property of DNSSEC that makes it prone to breaking (and really any real problem on your link applies just as well to HTTPS). It just breaks because those large entities don't care about fixing it or care a big deal about breaking it on purpose.
    - _wldu 4y ago
jgrahamc 4y ago
Reading this hurts. I see that @elithrar has given out his email address and is following up but I will also be following this internally to understand what happened.
- hericium 4y ago
  Since the issue was made public, we'd love to see some details in case OP (throwaway) doesn't come back.
  - medguru 4y ago
    I'll be back.
    - medguru 4y ago
      https://news.ycombinator.com/context?id=31413856 https://news.ycombinator.com/context?id=31413856
redm 4y ago
Enterprise plans no longer come with "premium" support either, you are looking at 20% over contract value to get a similar level of previously included support and an SLA. To be fair, CloudFlare provides a lot of services for free and $20 premium plan with upgraded support seems like a pretty good deal!
- medguru 4y ago
  For any form of business endeavour that would be the obvious lowest starting point. For private users a $20/mo. price tag for registering one or a few domains would turn effective TLD pricing on its head. Suddenly owning a single .net domain is no longer $20 per year like it is with all the other thousand registrars, but instead it would be $260 per year. Cloudflare's kind policy of zero markup on domain registrations on a free plan is remarkably generous. OK, sure, the traffic data has an obvious value to them, but maybe the support environment could improve with, I dunno, a tiny 5% markup.
groffee 4y ago
> How can I get their attention without paying for an Enterprise plan? Just comment on HN and they'll crawl out of the woodwork.
- paulnpace 4y ago
  Also, the community forums are generally quite useful when something isn't working. I've posted there with an actual problem maybe twice, but the problems were resolved within 30 minutes.
  - medguru 4y ago
    Which is a bit strange. I would have guessed that the support ticket system would be prioritized by staff.
ejjpi 4y ago
I'm also noticing that Cloudflare support is going terribly downhill. I have an issue with the Cloudflare infrastructure on my domain since WEEKS, giving me thousands of 503 Service Temporarily Unavailable errors per day (cloudflare side, not the origin server) and nobody seems to care or able to resolve. Removing the ability to create support tickets on free plan doesn't help at all, I mean, I get it why they're doing it, but asking on their community forum as an alternative it's not an acceptable solution. Neither going after Cloudflare employees on social media platforms hoping for a reply. If I'm also going to pay for their services such as Zero Trust, domains registrar and R2, why do I have to switch to a Pro plan just to open a support ticket? Perhaps a middle-ground solution like 1 free support ticket per month on a free plan would be a good compromise? I still think they're giving an incredible service and value for free, but this sucks.
- jgrahamc 4y ago
  Can you email me (jgc@cloudflare.com) with details?
  - juliancox 4y ago
    I can also confirm the support seems to be terrible at the moment, which is disappointing as in general I think cloudflare is a great service and I've had good experiences with most of the features (though the docs often leave a bit to be desired) and have been a user since cloudflare's early beta days. We're on a pro plan and have had an outstanding support ticket since March 22nd. With the last cloudflare response being 19 days ago. I can't seem to get cloudflare to talk directly to backblaze (it's a domain mapping issue) and playing the middle-man in a back and forth between cloudflare and backblaze support seems to be recipe for not getting things resolved promptly. I know it's covid times and organizations may be short staffed but compare this to cloud66 support who implemented a whole code update to support a special edge case for a non-paying customer within 48 hours. That makes an almost 2 month old unresolved ticket seem a bit tired. @jgrahamc I'll email you ticket details in case you'd like to take a look.
    - juliancox 4y ago
Yeri 4y ago
(CF TAM here) All plans come with support. Even the free plans (community, or email, the bot will deflect the request but if you email you're still stuck, you will get a reply _eventually_ (due to heavy support load, it can take a while though). The correct procedure would be: * turn off DNSsec on old registrar (and wait a day or two) * update NS and/or migrate domain * wait a while and make sure it works * turn on DNSsec in CF dash and update DNSsec settings in the domain It's not that DNSsec doesn't work -- it's doing exactly what it's supposed to be doing.
- Spunkie 4y ago
  I mean when you are adding a new domain, CF can clearly tell if a domain already has dnssec on or not. Seems like something that should raise a warning to the user.
  - medguru 4y ago
    That would have been very helpful. The [Enable DNSSEC] button in the control panel is very assertive and confident through its casual and innocuous appearance. Source: me, having lost access to my domain for 4 days for reasons that are not yet fully clear to me.
- mike_d 4y ago
  The correct steps are to completely disable DNSSEC and remove the relevant records. It is far too fragile for the majority of use cases. Effort should instead be put into deploying things like DNSCrypt that implement transport security and confidentiality. Transport security is like HTTPS. DNSSEC was the equivalent of PGP signing every webpage. The former brings value to the end user, the latter not so much. Even the government has issued memo M-18-23 ("Shifting From Low-Value to High-Value Work") that rescinds the requirements for the government to implement DNSSEC.
  - mike_hearn 4y ago
    DNSSEC is however the only way you can make TLS really work. The whole TLS ecosystem is dependent on CAs that are basically just a giant hack. They're signing a statement that they did a bunch of DNS resolutions at a point in time from different network vantage points (maybe, hopefully), and got consistent answers. DNSSEC+DANE lets you get the actual data you want (domain name->public key binding) from the root source, without needing the complicated middlemen.
warrenm 4y ago
First problem - trusting Cloudflare :| I've had nothing but problems with them personally I know some people swear by them ... I'm in the "swear at them" camp
andrewstuart 4y ago
I had the same problem. I registered a domain at Google Domains. Then I configured the domain at CloudFlare. At first it worked OK then I started getting SERVFAIL. I found the problem was there was still DNSSEC configuration set up at Google Domains. I deleted that and everything worked OK. Cloudflare was not at fault in my case.
medguru 4y ago
Update for those who were curious: Roughly one hour after I e-mailed @elithrar who kindly reached out and offered to expediate the issue, the broken DNSSEC records were partly fixed. The domain once again resolved through all major DNSes, and public access was restored. At that point dnsviz.net told me that A, MX, etc. records were "insecure", though name resolution worked fine. A few minutes ago I took another look with dnsviz and it's now telling me that all records are secure. Everything looks normal again. Thanks a bunch for helping out, @elithrar. I really appreciate that you were proactive. If the problem had somehow fixed itself or if the support ticket had gotten any attention or feedback at all within a day or two instead of just being "snoozed" by support staff, I wouldn't have made any noise about it. After four days of complete silence a bit of "cry-baby consumer activism" seemed like the only resort. If CF reconnects to me with an update on why the domain dead-locked and why it took 4 days to untilt everything I'll add that info as well. I've been OP and this has been an update about my domain woes.

139 points by medguru 4y ago

wnoise 4y ago

English language usage note: "since" takes a past point in time, not a duration. You want either "unreachable for 4 days", or "unreachable since 4 days ago".

InitialBP 4y ago

Sorry to hear about your problem, a quick recommendation would be to keep the temporary DNS name you bought and simply redirect to your previous name once you have the issue resolved (or vice versa if the branding is less important to you.) This way your users won't need to know or care about the change anymore aside from this temporary setback.

scrollaway 4y ago

> what's the fastest way to get technical assistance when on a free plan? Upgrading to a non-free plan? You don't have to upgrade to enterprise, but even their $20/mo plan comes with support. (Also, I hate to victim-blame here but using DNSSEC was a bad idea in the first place)

tmikaeld 4y ago
Why was enabling DNSSEC a bad idea? Clearly the origin registrar isn't handling DNSSEC requests properly, but the OP should still be able to revert to non-DNSSEC without issues.
- pixl97 4y ago
  Wasn't there a thread on HN within the last couple of months that says switching back when things go wrong is actually very difficult? Intermediate servers or something cache the DNSSEC issues and things tend to break for 24 hours at a time. Unfortunately I can't remember the details.
- scrollaway 4y ago
  This post makes a better argument than I could write. And nothing much has changed in the seven years since. https://sockpuppet.org/blog/2015/01/15/against-dnssec/ https://sockpuppet.org/blog/2015/01/15/against-dnssec/
  - mattashii 4y ago
    That article is a bad example of outdated arguments and whataboutisms. Sure, DNSSEC has some issues, but none are so bad that it means you shouldn't use it: It does help resolvers detect various attacks (e.g. cache poisoning, BGP hijacks, MitMs), which is fairly critical for security-minded organizations. (from that article, that is from 2015 and woefully outdated) > With TLS properly configured, DNSSEC adds nothing. This is false. DNSSEC adds address lookup security through response integrity, whereas TLS (only!) adds transport layer security to the endpoint you're connected to (hence the name). If you find a record in DNS with DNSSEC enabled, you know that the response is exactly as the sender intended it to be (and when connecting to the address returned for A- or AAAA-records, you'll be connecting to the intended IP address). Without DNSSEC, this is impossible to guarantee and record interception / MitM would be an attack vector. Additionally, "With TLS correctly configured" also implies CAA being set up, which only can be done securely using DNSSEC. As to why CAA must be configured: Not all CAs are made the same; and re-routing network traffic is fairly doable if you only need to target one of the many public CAs. Targeting only that CA allowed in the CAA record is presumably much harder. > Securing DNS lookups isn’t a high-priority task > DNSSEC’s real job is thus to replace the TLS CA system. This plan is called DANE. No, DNSSEC's job _is_ to secure DNS lookups. DANE is only one scheme that is made possible by DNSSEC; Secure CAA checking being another. > Real-world DNSSEC therefore relies on RSA with PKCS1v15 padding. Correct, but also relies on Ed25519 and P-256. A lot of authorative servers are still using the legacy RSA keys, but another lot is using P-256 and Ed25519 too. > [sections] DNSSEC is Expensive To Adopt / Deploy This is partly true, but any security is expensive to adopt/deploy. DNSSEC is fairly easy nowadays, though, with many hosted DNS services providing some form of DNSSEC. > DNSSEC doesn’t secure browser DNS lookups. It would, if you allowed your browser to recurse. > DNSSEC is Unsafe > Authenticated denial. Offline signers. Secret hostnames. Pick two. That's fine. Secrecy doesn't add security; Authenticated denial and Offline signers do. > DNSSEC is Architecturally Unsound I disagree with the conclusion here. Sure, it might be useful for US gov to be the writer of the spec, but what public scrutiny DNSSEC has had implies that the security part is sound.

b3lvedere 4y ago

-- removed. My apologies. --

medguru 4y ago
My domain and users have ended up in limbo beyond anyone's but Cloudflare's control. I cannot transfer it back to the working registrar, or I would without being "angry at some free service". Why do you think berating me with snide remarks is helpful?
- codatory 4y ago
  When it comes to dnssec, the keys are handed down through the same layer of delegation that gives you your nameservers. This is out of Cloudflare's hands as well, you have bad data up through the registry and out to (your tld's) root nameservers. Just like changes to your NS record delegation don't take immediate effect, these keys don't take immediate effect either. It's still pretty early days for DNSSEC, if you're going to use it it's worthwhile to know a lot about it. Just look at the several Slack outages caused by their attempts to implement it. Eventually the tooling will catch up, and registrars will all give you warnings about moving DNS and registration and the importance of syncing up your keys but we just aren't there yet.
- b3lvedere 4y ago
  My apologies if you read it as a snide remark. I'm usually baffled when support is demanded on free stuff, but i see your point in this particular scenario.
  - tedivm 4y ago
    No one "read it as a snide remark"- you were snide and rude. Just own it and apologize.
    - b3lvedere 4y ago
      I did and i removed the parent comment. If you'd like me to delete everything, i'd be happy to oblige.

elithrar 4y ago

(It sucks that I had to see this on HN) Can you email me - silverlock at cloudflare - with your ticket ID and domain name so I can understand what broke?

medguru 4y ago
Thank you for the attention, e-mail on its way.
- gigatexal 4y ago
  Come back and tell us what happened and the resolution.
  - medguru 4y ago
    https://news.ycombinator.com/context?id=31413856 https://news.ycombinator.com/context?id=31413856
- elithrar 4y ago
  Just replied. Ultimately it looks like the existing DS records for your domain weren't removed (and you can see that in your DNSViz output). Still have some questions for "how" it was working beforehand (see the email for those). For others: I'll let the OP share what details they would like to, as this is their domain.
quinncom 4y ago
Is there a way to pay for priority support while on a free plan? I'm nervous about migrating a domain to CF which is used for glue records, and want to have immediate support access if something goes wrong.
- elithrar 4y ago
  The $20/mo Pro plan includes formal support (one of the big perks of Pro!)

jSherz 4y ago

Does your TLD definitely support DNSSEC?

medguru 4y ago
Yes. It had DNSSEC enabled for over a year when it was with the old registrar.
- phillipseamore 4y ago
  You would usually need to disable DNSSEC, wait 24h, transfer, and then wait for at least 24h before enabling DNSSEC again.
  - medguru 4y ago
    Thanks for the info, I'll keep it in mind for eventual future transfers. But shouldn't I be able to disable DNSSEC regardless, instead of the domain being stuck in limbo and hijacked by what appears to be a deadlock type of bug?
    - belorn 4y ago
      Talking as a admin at a registrar, Yes, Indeed Yes, you should be able to disable DNSSEC regardless. DNSSEC signed is basically just that the TLD servers has a DS record listed for the domain. In order to remove dnssec you remove the DS record. This can be easy or hard depending on the interface that the TLD, but in theory very simple. The reason why its recommended to remove dnssec before transfer is to allow caches to timeout with the old DS record to expire. Some TLD also automatically remove DS when you do a transfer and a name server change, as it is a rather clear signal that the old key won't be useful. There is however some exciting new technology called multi-signer which is intended to resolve this problem in the future.
    - phillipseamore 4y ago
      Disabling DNSSEC doesn't propagate instantly. Have you queried the CF nameservers for the domain directly? In my experience everything involving DNSSEC requires a 24h wait (unless the domain hasn't been queried from anywhere - but that's usually not the case, something might have triggered distributed DNS lookups e.g. LE doing DNS validation for cert issuance etc).

pteraspidomorph 4y ago

I had a problem with a similar effect some time ago but I run my own DNS (no Cloudflare). I accidentally clicked a button in my control panel to regenerate the zone keys, which means the published keys mismatched the new zone signature for a couple of days until I was able to get the registrar to update them and everything propagated (even when a registrar supports the .eu TLD they are usually severely lacking in automation). The control panel devs have since added a confirmation dialog!

williamtwild 4y ago

>I've been forced to migrate the project and its (few) users to a completely different domain. I cannot inconvenience users by bouncing them back and forth, so the domain Cloudflare ruined for me is now effectively lost, as is the "branding" of the project which was reflected in the domain's name. If this was that important then you should not have used the free plan.

medguru 4y ago
Why do you presume the issue would have gotten immediate attention for the sum of $20? Customers don't make Cloudflare's terms, and customers didn't decide for Cloudflare to offer a free plan with zero markup for their registrar operations. There is by users' own hands no way out of domain registration issues like these, sooner than 30-45 days when the domain can be transferred once again. Those who decide to offer registrar services, even for free, must hold some liability towards the users and the ecosystem and offer some support to make sure their product actually works.
- Spunkie 4y ago
  They are also the only domain registrar that I've interacted with this decade that does not allow you to set your own nameservers. Effectively locking you into using cloudflare DNS and related services until you can transfer again. Its frankly, disgusting.

xbar 4y ago

dnsviz.net is awesome.

oneplane 4y ago

> How can I get their attention without paying for an Enterprise plan? By paying for the cheapest plan, or any plan at all for that matter.

_wldu 4y ago

DNSSEC is notorious for breaking things [1]. I use it on most of my domains, but I would not just 'enable' it on a domain that I cared about and that had real users without a lot of thought and planning. Nor should you. [1] - https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html

medguru 4y ago
I figured it would work since there were no problems for the handful of my domains using DNSSEC with the previous registrar. Maybe the button should come with a warning label. I'll certainly be a bit cautious from now on.
paulnpace 4y ago
> DNSSEC is notorious for breaking things My understanding is that people break things.
- ceejayoz 4y ago
  https://en.wikipedia.org/wiki/Just_culture https://en.wikipedia.org/wiki/Just_culture > Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved. In a just culture, after an incident, the question asked is, "What went wrong?" rather than "Who caused the problem?". Prominent (and very effective) example: Aviation safety. DNSSEC is both easy to break and hard to fix. https://sockpuppet.org/blog/2015/01/15/against-dnssec/ https://sockpuppet.org/blog/2015/01/15/against-dnssec/
  - marcosdumay 4y ago
    There's no property of DNSSEC that makes it prone to breaking (and really any real problem on your link applies just as well to HTTPS). It just breaks because those large entities don't care about fixing it or care a big deal about breaking it on purpose.
  - _wldu 4y ago

jgrahamc 4y ago

Reading this hurts. I see that @elithrar has given out his email address and is following up but I will also be following this internally to understand what happened.

hericium 4y ago
Since the issue was made public, we'd love to see some details in case OP (throwaway) doesn't come back.
- medguru 4y ago
  I'll be back.
  - medguru 4y ago
    https://news.ycombinator.com/context?id=31413856 https://news.ycombinator.com/context?id=31413856

redm 4y ago

Enterprise plans no longer come with "premium" support either, you are looking at 20% over contract value to get a similar level of previously included support and an SLA. To be fair, CloudFlare provides a lot of services for free and $20 premium plan with upgraded support seems like a pretty good deal!

medguru 4y ago
For any form of business endeavour that would be the obvious lowest starting point. For private users a $20/mo. price tag for registering one or a few domains would turn effective TLD pricing on its head. Suddenly owning a single .net domain is no longer $20 per year like it is with all the other thousand registrars, but instead it would be $260 per year. Cloudflare's kind policy of zero markup on domain registrations on a free plan is remarkably generous. OK, sure, the traffic data has an obvious value to them, but maybe the support environment could improve with, I dunno, a tiny 5% markup.

groffee 4y ago

> How can I get their attention without paying for an Enterprise plan? Just comment on HN and they'll crawl out of the woodwork.

paulnpace 4y ago
Also, the community forums are generally quite useful when something isn't working. I've posted there with an actual problem maybe twice, but the problems were resolved within 30 minutes.
- medguru 4y ago
  Which is a bit strange. I would have guessed that the support ticket system would be prioritized by staff.

ejjpi 4y ago

I'm also noticing that Cloudflare support is going terribly downhill. I have an issue with the Cloudflare infrastructure on my domain since WEEKS, giving me thousands of 503 Service Temporarily Unavailable errors per day (cloudflare side, not the origin server) and nobody seems to care or able to resolve. Removing the ability to create support tickets on free plan doesn't help at all, I mean, I get it why they're doing it, but asking on their community forum as an alternative it's not an acceptable solution. Neither going after Cloudflare employees on social media platforms hoping for a reply. If I'm also going to pay for their services such as Zero Trust, domains registrar and R2, why do I have to switch to a Pro plan just to open a support ticket? Perhaps a middle-ground solution like 1 free support ticket per month on a free plan would be a good compromise? I still think they're giving an incredible service and value for free, but this sucks.

jgrahamc 4y ago
Can you email me (jgc@cloudflare.com) with details?
- juliancox 4y ago
  I can also confirm the support seems to be terrible at the moment, which is disappointing as in general I think cloudflare is a great service and I've had good experiences with most of the features (though the docs often leave a bit to be desired) and have been a user since cloudflare's early beta days. We're on a pro plan and have had an outstanding support ticket since March 22nd. With the last cloudflare response being 19 days ago. I can't seem to get cloudflare to talk directly to backblaze (it's a domain mapping issue) and playing the middle-man in a back and forth between cloudflare and backblaze support seems to be recipe for not getting things resolved promptly. I know it's covid times and organizations may be short staffed but compare this to cloud66 support who implemented a whole code update to support a special edge case for a non-paying customer within 48 hours. That makes an almost 2 month old unresolved ticket seem a bit tired. @jgrahamc I'll email you ticket details in case you'd like to take a look.
  - juliancox 4y ago

Yeri 4y ago

(CF TAM here) All plans come with support. Even the free plans (community, or email, the bot will deflect the request but if you email you're still stuck, you will get a reply _eventually_ (due to heavy support load, it can take a while though). The correct procedure would be: * turn off DNSsec on old registrar (and wait a day or two) * update NS and/or migrate domain * wait a while and make sure it works * turn on DNSsec in CF dash and update DNSsec settings in the domain It's not that DNSsec doesn't work -- it's doing exactly what it's supposed to be doing.

Spunkie 4y ago
I mean when you are adding a new domain, CF can clearly tell if a domain already has dnssec on or not. Seems like something that should raise a warning to the user.
- medguru 4y ago
  That would have been very helpful. The [Enable DNSSEC] button in the control panel is very assertive and confident through its casual and innocuous appearance. Source: me, having lost access to my domain for 4 days for reasons that are not yet fully clear to me.
mike_d 4y ago
The correct steps are to completely disable DNSSEC and remove the relevant records. It is far too fragile for the majority of use cases. Effort should instead be put into deploying things like DNSCrypt that implement transport security and confidentiality. Transport security is like HTTPS. DNSSEC was the equivalent of PGP signing every webpage. The former brings value to the end user, the latter not so much. Even the government has issued memo M-18-23 ("Shifting From Low-Value to High-Value Work") that rescinds the requirements for the government to implement DNSSEC.
- mike_hearn 4y ago
  DNSSEC is however the only way you can make TLS really work. The whole TLS ecosystem is dependent on CAs that are basically just a giant hack. They're signing a statement that they did a bunch of DNS resolutions at a point in time from different network vantage points (maybe, hopefully), and got consistent answers. DNSSEC+DANE lets you get the actual data you want (domain name->public key binding) from the root source, without needing the complicated middlemen.

warrenm 4y ago

First problem - trusting Cloudflare :| I've had nothing but problems with them personally I know some people swear by them ... I'm in the "swear at them" camp

andrewstuart 4y ago

I had the same problem. I registered a domain at Google Domains. Then I configured the domain at CloudFlare. At first it worked OK then I started getting SERVFAIL. I found the problem was there was still DNSSEC configuration set up at Google Domains. I deleted that and everything worked OK. Cloudflare was not at fault in my case.

medguru 4y ago

Update for those who were curious: Roughly one hour after I e-mailed @elithrar who kindly reached out and offered to expediate the issue, the broken DNSSEC records were partly fixed. The domain once again resolved through all major DNSes, and public access was restored. At that point dnsviz.net told me that A, MX, etc. records were "insecure", though name resolution worked fine. A few minutes ago I took another look with dnsviz and it's now telling me that all records are secure. Everything looks normal again. Thanks a bunch for helping out, @elithrar. I really appreciate that you were proactive. If the problem had somehow fixed itself or if the support ticket had gotten any attention or feedback at all within a day or two instead of just being "snoozed" by support staff, I wouldn't have made any noise about it. After four days of complete silence a bit of "cry-baby consumer activism" seemed like the only resort. If CF reconnects to me with an update on why the domain dead-locked and why it took 4 days to untilt everything I'll add that info as well. I've been OP and this has been an update about my domain woes.