8 ms·

I wonder about the trade-offs in writing a completely new server compared to a backend for BIND or PowerDNS.

tptacek 5y ago
Straightforward authority servers like these are very easy to write, in memory-safe languages. It might actually take longer to figure out how to get a BIND configuration doing what Replit does here, but even if it didn't, the resulting server is much safer than BIND, and does exactly and only what you want it to do. I think: a pretty easy engineering decision.
- nexuist 5y ago
  To add on to this, DNS is meant to be a simple protocol. The problems we encounter with it today are usually due to the thousands of little patches added on through the decades to tack functionality onto what is essentially supposed to be a key-value store.
  - deleted 5y ago
    [deleted]
- tialaramex 5y ago
  I don't disagree overall that a competent team really could get this right in the same time it'd take to make it work how you wanted with BIND, but although DNS is simple there are some corner cases where an incorrect solution will appear to work in trivial scenarios - "It worked in my Chrome" but may have either functional problems or security problems. For example the 0x20 trick. The specification for DNS is clear that you aren't supposed to care about bit 0x20 in labels. ClOWnS and cLowNs and clowns and CLOWNS are all the same label as far as DNS is concerned. However your answers need to bit-for-bit match the question you were asked. So if you answer "ClOWns A?" with "CLOWNS A 10.20.30.40" that's a mistake, you were asked about "ClOWns" not "CLOWNS". In 1995 if your DNS server got this wrong nothing of consequence breaks. But in 2021 if you get this wrong some important things magically don't work. The transaction IDs that should make forging DNS answers hard are very short, and so to beef that up slightly some stacks will hide more bits in the 0x20 bit of labels where they will be echo'd back by a compliant implementation. But to reap this reward they must ignore answers that get the 0x20 bits wrong, like yours. I feel like if "You can't get this wrong" (a stronger claim that you admittedly didn't make) was true, my visits to the Let's Encrypt community site wouldn't all begin by ignoring the people whose problem is obviously just that their DNS server doesn't work properly. Some of them have problems an authority server doesn't care about, but lots of them have dumb problems you'd imagine are impossible and yet apparently people have successfully sold commercial DNS servers with those problems.
  - throw0101a 5y ago
    For anyone curious about 0x20: * https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20 https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20 I'm not surprised that Vixie is involved. :)
  - bluejekyll 5y ago
    The simple solution to this is to return the same binary query as was received from the request. It’s easy to screw up, definitely agree, but fairly clear how to fix when it’s pointed out. (I made that mistake)
  - nine_k 5y ago
    If you don't mind, what things would magically break if the case of the answer does not match? Comparisons should be case-insensitive anyway. Also, since DNS labels are strictly ASCII (this is why punycode exists), why converting all of them to the canonical upper case won't be a good idea?
    - 0x0 5y ago
      The random mixed-case patterns are used to reduce the probability that someone evil could send spoofed DNS replies to your queries and have your DNS resolver trust those fake replies.
      - nine_k 5y ago
        But if an adversary can send a spoofed reply with the desired domain name at all, I expect that the adversary could read the original request packet, too?
        tialaramex 5y ago
        They're racing. The adversary needs to have their spoofed reply arrive first so you'll accept it as genuine. They will most often seek to arrange to reply to a query they guess you've asked, such that their answer arrives after it's asked but before you receive the honest answer. This is why that transaction ID matters, the honest answer will copy the transaction ID verbatim from your question in the answer, so you get to pick it at random (back when I was a child it might just be a sequential counter) and your adversary has to guess it. But, alas the ID isn't very wide, so they really do have a good chance to just guess it. Hence, let's hide more random bits elsewhere in our queries to get a better chance of foiling the adversary. How does an adversary guess what you're asking? Well, for one thing they might have chosen the question you're about to ask. When a bad guy's web site has an image at the top with <IMG SRC="http://real.website.example/header.jpg http://real.website.example/header.jpg"> doesn't your web browser try to look up real.website.example to go get the image? Very predictable.
ignoramous 5y ago
We had some success with CoreDNS (over BIND et al): https://coredns.io/manual/toc/ https://coredns.io/manual/toc/ Switched later to OctoDNS, mostly because we didn't want to run DNS infrastructure or deal with racing updates to records: https://github.com/octodns/octodns https://github.com/octodns/octodns
cbrewster 5y ago
Hi, author here! tptacek is right on the money. The authority server is really small and simple, and writing it in Go meant we had access to our existing internal packages that had the logic to fetch the data we need for each DNS query. This seemed like the most straight-forward path.
- tptacek 5y ago
  miekg/dns is really excellent. Our custom DNS server is written in Rust (with the NLNet libraries, which are also great), but I used miekg/dns to throw together a DNS telemetry system that we use to keep metrics on our DNS (and UDP) service from around the world using off-net hosts, which sounds cool to type out but was an absurdly simple coding project because of how good the libraries are. More people should do cool weird stuff with DNS. (And Replit should host stuff on Fly! But also the DNS stuff we're talking about.) It's a great post, thanks for writing it.
  - ignoramous 5y ago
    > And Replit should host stuff on Fly! But also the DNS stuff we're talking about. Having used both, I'd wager replit competes with fly.
    - tptacek 5y ago
      That wasn't a dig!
- rgacogne 5y ago
  Hi, PowerDNS dev here! Would you mind fixing your handling of NODATA answers, which are currently lacking the AA bit and a SOA record in authority? It does not seem to affect all instances, so perhaps it only happens on the legacy infra. You can get the details here, for example: https://dnsviz.net/d/b.b.b.b.b.b.b.b.a.a.a.a.a.a.nope.repl.co/dnssec/ https://dnsviz.net/d/b.b.b.b.b.b.b.b.a.a.a.a.a.a.nope.repl.c...
  - cbrewster 5y ago
deleted 5y ago
[deleted]
tyingq 5y ago
Unbound is pretty popular with the folks I work with, but in this case, they are only serving up things they are authoritative for, so they don't need much.
- wut42 5y ago
  Isn't unbound a resolver, and not an authoritative server (nsd) ?
  - tyingq 5y ago
    Ah, yep, right. Argh, too late to edit/delete.
gregmac 5y ago
At work we have a service built on powerdns-backend (http). The biggest pain there was lack of docs. There's circumstances where powerdns would make multiple requests to the backend app server, and figuring out how to answer its first direct queries in a way to satisfy it on the first query took some trial and error (this was a few years ago, so I don't remember specifics). I seem to remember it asking the backend for ANY records and then a bunch of metadata stuff, which meant having to spend time understanding not just how to answer the simple DNS TXT and A queries we were trying to do, but understanding how powerdns would interpret the client query, pass it to the backend, and interpret the result (which was quite different from straight DNS). That also made it really hard to test, short of setting up a full end-to-end integration test including running powerdns (which I think we have, but isn't fully automated). If I was building that service again, I'd definitely have a serious look into building it directly as a standalone DNS server rather than a backend for something else.
- Habbie 5y ago
  Hello! PowerDNS developer here. Did you spot https://doc.powerdns.com/authoritative/appendices/internals.html https://doc.powerdns.com/authoritative/appendices/internals.... and https://doc.powerdns.com/authoritative/appendices/backend-writers-guide.html https://doc.powerdns.com/authoritative/appendices/backend-wr... ? And if not (or also if you did), can you suggest documentation additions that would have helped you here?
  - gregmac 5y ago
    I think that page would have been incredibly useful, but doesn't look like it existed. I was just looking through the source history of the project a bit as I wasn't the original developer (I've just done some fixes; most recently fixing some things going from 4.2 to 4.4). The original code was written in 2016, when the documentation was a lot more sparse [1]. Kudos for the improvements! I just filed a PR [2] fixing the doc issues I ran into (probably should have done that at the time). To enumerate some things I see right now (using remote http backend): * pdns always queries /lookup/example.com./SOA, /getAllDomainMetadata/example.com. /lookup/example.com./ANY -- in my case that seems a bit wasteful: there's no metadata, and SOA is returned in the /ANY request anyway. I'm unclear if there's a misconfiguration, something we're doing wrong, or this is just "how it works". * If I query `some.domain.example.com.` (and we aren't serving requests for any part of that), the backend returns `{result:false}` -- and then pdns proceeds to query `domain.example.com.`, `example.com.`, `com.`, and `.` before finally returning SERVFAIL. It would be nice if (1) it didn't have to run so many queries, but also (2) my understanding is the appropriate response should be REFUSED (as this is not a general DNS resolver) -- but I don't know how to get PowerDNS to do that. That last bit is what I've found most frustrating about working with pdns: everything feels like "just do what pdns asks and it'll sort out what it wants to do as a result" and I can't just tell it "returned REFUSED for that query". Maybe some examples would help with this? [1] https://web.archive.org/web/20170615183449/https://doc.powerdns.com/md/authoritative/backend-remote/ https://web.archive.org/web/20170615183449/https://doc.power... [2] https://github.com/PowerDNS/pdns/pull/10345 https://github.com/PowerDNS/pdns/pull/10345

Straightforward authority servers like these are very easy to write, in memory-safe languages. It might actually take longer to figure out how to get a BIND configuration doing what Replit does here, but even if it didn't, the resulting server is much safer than BIND, and does exactly and only what you want it to do. I think: a pretty easy engineering decision.

nexuist 5y ago
To add on to this, DNS is meant to be a simple protocol. The problems we encounter with it today are usually due to the thousands of little patches added on through the decades to tack functionality onto what is essentially supposed to be a key-value store.
- deleted 5y ago
  [deleted]
tialaramex 5y ago
I don't disagree overall that a competent team really could get this right in the same time it'd take to make it work how you wanted with BIND, but although DNS is simple there are some corner cases where an incorrect solution will appear to work in trivial scenarios - "It worked in my Chrome" but may have either functional problems or security problems. For example the 0x20 trick. The specification for DNS is clear that you aren't supposed to care about bit 0x20 in labels. ClOWnS and cLowNs and clowns and CLOWNS are all the same label as far as DNS is concerned. However your answers need to bit-for-bit match the question you were asked. So if you answer "ClOWns A?" with "CLOWNS A 10.20.30.40" that's a mistake, you were asked about "ClOWns" not "CLOWNS". In 1995 if your DNS server got this wrong nothing of consequence breaks. But in 2021 if you get this wrong some important things magically don't work. The transaction IDs that should make forging DNS answers hard are very short, and so to beef that up slightly some stacks will hide more bits in the 0x20 bit of labels where they will be echo'd back by a compliant implementation. But to reap this reward they must ignore answers that get the 0x20 bits wrong, like yours. I feel like if "You can't get this wrong" (a stronger claim that you admittedly didn't make) was true, my visits to the Let's Encrypt community site wouldn't all begin by ignoring the people whose problem is obviously just that their DNS server doesn't work properly. Some of them have problems an authority server doesn't care about, but lots of them have dumb problems you'd imagine are impossible and yet apparently people have successfully sold commercial DNS servers with those problems.
- throw0101a 5y ago
  For anyone curious about 0x20: * https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20 https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20 I'm not surprised that Vixie is involved. :)
- bluejekyll 5y ago
  The simple solution to this is to return the same binary query as was received from the request. It’s easy to screw up, definitely agree, but fairly clear how to fix when it’s pointed out. (I made that mistake)
- nine_k 5y ago
  If you don't mind, what things would magically break if the case of the answer does not match? Comparisons should be case-insensitive anyway. Also, since DNS labels are strictly ASCII (this is why punycode exists), why converting all of them to the canonical upper case won't be a good idea?
  - 0x0 5y ago
    The random mixed-case patterns are used to reduce the probability that someone evil could send spoofed DNS replies to your queries and have your DNS resolver trust those fake replies.
    - nine_k 5y ago
      But if an adversary can send a spoofed reply with the desired domain name at all, I expect that the adversary could read the original request packet, too?
      - tialaramex 5y ago
        They're racing. The adversary needs to have their spoofed reply arrive first so you'll accept it as genuine. They will most often seek to arrange to reply to a query they guess you've asked, such that their answer arrives after it's asked but before you receive the honest answer. This is why that transaction ID matters, the honest answer will copy the transaction ID verbatim from your question in the answer, so you get to pick it at random (back when I was a child it might just be a sequential counter) and your adversary has to guess it. But, alas the ID isn't very wide, so they really do have a good chance to just guess it. Hence, let's hide more random bits elsewhere in our queries to get a better chance of foiling the adversary. How does an adversary guess what you're asking? Well, for one thing they might have chosen the question you're about to ask. When a bad guy's web site has an image at the top with <IMG SRC="http://real.website.example/header.jpg http://real.website.example/header.jpg"> doesn't your web browser try to look up real.website.example to go get the image? Very predictable.

ignoramous 5y ago

We had some success with CoreDNS (over BIND et al): https://coredns.io/manual/toc/ https://coredns.io/manual/toc/ Switched later to OctoDNS, mostly because we didn't want to run DNS infrastructure or deal with racing updates to records: https://github.com/octodns/octodns https://github.com/octodns/octodns

cbrewster 5y ago

Hi, author here! tptacek is right on the money. The authority server is really small and simple, and writing it in Go meant we had access to our existing internal packages that had the logic to fetch the data we need for each DNS query. This seemed like the most straight-forward path.

tptacek 5y ago
miekg/dns is really excellent. Our custom DNS server is written in Rust (with the NLNet libraries, which are also great), but I used miekg/dns to throw together a DNS telemetry system that we use to keep metrics on our DNS (and UDP) service from around the world using off-net hosts, which sounds cool to type out but was an absurdly simple coding project because of how good the libraries are. More people should do cool weird stuff with DNS. (And Replit should host stuff on Fly! But also the DNS stuff we're talking about.) It's a great post, thanks for writing it.
- ignoramous 5y ago
  > And Replit should host stuff on Fly! But also the DNS stuff we're talking about. Having used both, I'd wager replit competes with fly.
  - tptacek 5y ago
    That wasn't a dig!
rgacogne 5y ago
Hi, PowerDNS dev here! Would you mind fixing your handling of NODATA answers, which are currently lacking the AA bit and a SOA record in authority? It does not seem to affect all instances, so perhaps it only happens on the legacy infra. You can get the details here, for example: https://dnsviz.net/d/b.b.b.b.b.b.b.b.a.a.a.a.a.a.nope.repl.co/dnssec/ https://dnsviz.net/d/b.b.b.b.b.b.b.b.a.a.a.a.a.a.nope.repl.c...
- cbrewster 5y ago

deleted 5y ago

[deleted]

tyingq 5y ago

Unbound is pretty popular with the folks I work with, but in this case, they are only serving up things they are authoritative for, so they don't need much.

wut42 5y ago
Isn't unbound a resolver, and not an authoritative server (nsd) ?
- tyingq 5y ago
  Ah, yep, right. Argh, too late to edit/delete.

gregmac 5y ago

At work we have a service built on powerdns-backend (http). The biggest pain there was lack of docs. There's circumstances where powerdns would make multiple requests to the backend app server, and figuring out how to answer its first direct queries in a way to satisfy it on the first query took some trial and error (this was a few years ago, so I don't remember specifics). I seem to remember it asking the backend for ANY records and then a bunch of metadata stuff, which meant having to spend time understanding not just how to answer the simple DNS TXT and A queries we were trying to do, but understanding how powerdns would interpret the client query, pass it to the backend, and interpret the result (which was quite different from straight DNS). That also made it really hard to test, short of setting up a full end-to-end integration test including running powerdns (which I think we have, but isn't fully automated). If I was building that service again, I'd definitely have a serious look into building it directly as a standalone DNS server rather than a backend for something else.

Habbie 5y ago
Hello! PowerDNS developer here. Did you spot https://doc.powerdns.com/authoritative/appendices/internals.html https://doc.powerdns.com/authoritative/appendices/internals.... and https://doc.powerdns.com/authoritative/appendices/backend-writers-guide.html https://doc.powerdns.com/authoritative/appendices/backend-wr... ? And if not (or also if you did), can you suggest documentation additions that would have helped you here?
- gregmac 5y ago
  I think that page would have been incredibly useful, but doesn't look like it existed. I was just looking through the source history of the project a bit as I wasn't the original developer (I've just done some fixes; most recently fixing some things going from 4.2 to 4.4). The original code was written in 2016, when the documentation was a lot more sparse [1]. Kudos for the improvements! I just filed a PR [2] fixing the doc issues I ran into (probably should have done that at the time). To enumerate some things I see right now (using remote http backend): * pdns always queries /lookup/example.com./SOA, /getAllDomainMetadata/example.com. /lookup/example.com./ANY -- in my case that seems a bit wasteful: there's no metadata, and SOA is returned in the /ANY request anyway. I'm unclear if there's a misconfiguration, something we're doing wrong, or this is just "how it works". * If I query `some.domain.example.com.` (and we aren't serving requests for any part of that), the backend returns `{result:false}` -- and then pdns proceeds to query `domain.example.com.`, `example.com.`, `com.`, and `.` before finally returning SERVFAIL. It would be nice if (1) it didn't have to run so many queries, but also (2) my understanding is the appropriate response should be REFUSED (as this is not a general DNS resolver) -- but I don't know how to get PowerDNS to do that. That last bit is what I've found most frustrating about working with pdns: everything feels like "just do what pdns asks and it'll sort out what it wants to do as a result" and I can't just tell it "returned REFUSED for that query". Maybe some examples would help with this? [1] https://web.archive.org/web/20170615183449/https://doc.powerdns.com/md/authoritative/backend-remote/ https://web.archive.org/web/20170615183449/https://doc.power... [2] https://github.com/PowerDNS/pdns/pull/10345 https://github.com/PowerDNS/pdns/pull/10345