7 ms·

DNSimple DDOS Attack

125 points by dedene 12y ago

whafro 12y ago
We're kinda tied into DNSimple since we use an ALIAS record for our bare/naked/root domain. Amazon's Route53 supports aliases, but via a 301 redirect, which doesn't work in an SSL context (without browser warnings). Nonetheless, we just spun up a Route53 zone, exported our zone from DNSimple, imported to Route53, and hand-migrated our ALIAS records to static A records in the new zone. Not perfect or permanent, but we've gotten around the outage. Also, I just learned that pointhq has (seemingly-undocumented) support for ALIAS records in the same style as DNSimple, so this could be another avenue to explore.
- petercooper 12y ago
  How are you exporting the zone with things as they are?
  - algiecas 12y ago
    same question here - how did you do this while dnsimple.com is inaccessible?
    - zuccs 12y ago
      You can force your hosts file locally. Set "dnsimple.com" to "50.31.213.210". On Mac use "sudo pico /etc/hosts" and follow the formatting of the other entries.
  - whafro 12y ago
    Oof, really good point – I suppose I got lucky in that I was at dnsimple.com just a few hours ago for other purposes. Their DNS was still cached for me, and it all worked flawlessly. I didn't even suspect their outage extended to their web servers, though that makes perfect sense. FWIW, the IP I have cached is 50.31.213.210.
    - Pwnguinz 12y ago
      How were you able to find out what IP was cached for a particular domain?
- mef 12y ago
  FYI you can use ALIAS records without a 301 redirect if the ALIAS is to an ELB.
  - deleted 12y ago
    [deleted]
- EwanToo 12y ago
  A few people do ALIAS style records, DNSMadeEasy and EasyDNS do "ANAME" records which are pretty much the same.
  - joshdotsmith 12y ago
    Will these services not have the same thin pipe issue that's currently affecting DNSimple?
    - dentarg 12y ago
      Not that I have any reliable info, but what I've heard, DNS Made Easy is a pretty stable and established DNS provider. They brag about "99.9999% uptime history" at http://www.dnsmadeeasy.com/technology/ http://www.dnsmadeeasy.com/technology/. Though they doesn't seem as innovative and nice as DNSimple. Really hope things work out for DNSimple (really like the idea of their beta feature GitHub sync).
      - stevekemp 12y ago
        How does their beta-feature with github work? I can't find any obvious link and it seems frustratingly close to something I offer over at https://dns-api.com/ https://dns-api.com/ .. (I wrap Amazon's route53 with DNS entries read from github/gitbucket/similar.)
        zuccs 12y ago
        Can you use custom nameservers with your service?
        stevekemp 12y ago
        When you upload a new zone it will be assigned a set of nameservers - four. You can't choose what they are in advance, and you can't specify the TLDs. So you might end up with "ns-1933.awsdns-49.co.uk.", "ns-1109.awsdns-10.org.", or similar. Does that answer the question? I'm a little hazy on what you're actually asking.
pkfrank 12y ago
Can anyone expand on what this means: "This attack is volumetric in nature." (?)
- binarylogicben 12y ago
  Pretty sure it means the traffic is overwhelming them.
- styger 12y ago
  According to http://www.arbornetworks.com/attack-ddos http://www.arbornetworks.com/attack-ddos: "DDoS attacks...will generally fall into one of three broad categories: Volumetric Attacks: Attempt to consume the bandwidth either within the target network/service, or between the target network/service and the rest of the Internet. These attacks are simply about causing congestion."
jameskilton 12y ago
We can watch this happen live @ http://map.ipviking.com/ http://map.ipviking.com/ Fascinating traffic floods from various locations, but the attack is not continuous.
- jtcchan 12y ago
  Are these live attacks on any servers?
- shoxxx 12y ago
  Noticed a lot of hits to St. Louis, where they're based. Do you think this has anything to do with the attacks?
- diminoten 12y ago
  How does this link show this specific attack? As far as I understand, ipviking simply hosts honeypots around the world and uses those to graph "attacks" against IP blocks, etc. I would very much like someone to correct me if this assumption is incorrect, because it'd be neat to actually watch targeted DDoS attacks, but I don't think that's what ipviking is offering.
deleted 12y ago
[deleted]
beck5 12y ago
What are the recommended practices to prevent too much down time when your DNS provider goes down?
- mjibson 12y ago
  My site is currently offline from this attack. I am considering that providers like DNSimple simply cannot provide the networking availability to mitigate these kinds of attacks. This is because the solution to these kinds of attacks, often, is that you need a larger pipe than the attackers. Very few people are good at both <some service> and having a huge pipe. For web apps, you can use CloudFlare, which does have a bigger pipe and is designed to mitigate this. But DNS is not a web app, so you can't just put DNSimple behind CloudFlare. Hence, I'm going to try CloudFlare (assuming they take over DNS hosting, I need to check) and Google Cloud DNS, because then all parts of my site (from DNS to CSS hosting) will be with providers with bigger pipes than attackers can create. Hopefully that will prevent this kind of attack from taking my site down.
  - throwaway90446 12y ago
    Yes, CloudFlare has a full-featured DNS offering, even if you don't use their proxying services.
    - stevekemp 12y ago
      Which has had a lot of problems this past week: https://news.ycombinator.com/item?id=8665367 https://news.ycombinator.com/item?id=8665367
      - throwaway90446 12y ago
        CloudFlare != CloudFront
        stevekemp 12y ago
        My apologies, cloudfront had issues, cloudflare did not.
ericskiff 12y ago
For anyone else who needs to mitigate this in a hurry: Set up a new account on another host that does ALIAS records (I used pointDNS) Create your new record without much in it Change your nameservers on your domain now - they'll take time to propagate Fill in the records on your domain. If you can't remember them, print out most of your existing records with dig yourdomain.com ANY Add the rest of the records to pointDNS Wait for the new Nameservers to propagate (0-24 hours - it took 15-30 min for us on a small-medium traffic domain today during sales crunch)
- scott_karana 12y ago
  For those with a more deterministic bent: "propagation" time has a maximum bound of your TTL, which will show with any dig queries.
  - colmmacc 12y ago
    Propagation is bound at the DNS TTL plus whatever time it takes your DNS provider/setup to relay records to all of its authoritative servers.
    - scott_karana 12y ago
      Hmm, never considered that. Is it a slow, static process for ISPs to do that? I just assumed that they ran more-or-less stock DNS resolvers with in-memory caches.
- colmmacc 12y ago
  Publishing an incomplete copy of your zone seems like a bad idea, an ANY dig at the zone apex is going to retrieve less than 30% of the records (on average) for real-world zones. Negative responses in DNS are cacheable - so you can easily poison yourself. If you really really want to do it anyway, most caches use either the TTL on your SOA record, or the final field in the SOA record as the negative cache TTL; so lower both of those values to something like 60 seconds.
shoxxx 12y ago
Anyone switching from DNSimple? I really don't want to, but we've been down for almost 3 hours. I've seen chatter about Cloudfare and it looks pretty good, reviews?
- hglaser 12y ago
  We switched periscope.io from DNSimple to Amazon Route 53. DNSimple doesn't have an exporter so it took about an hour, including having one engineer review the other engineer's work. Many customers were able to resolve the domain in the minutes immediately following the switch, and the rest seem to be trickling in.
  - walmartian 12y ago
    Are you talking about a zone file exporter? DNSimple does have one, we just used it to migrate to Route53. http://support.dnsimple.com/articles/zone-export/ http://support.dnsimple.com/articles/zone-export/ EDIT: right, must have been able to log in during a brief period where dnsimple was not down.
    - shoxxx 12y ago
      link doesnt work
    - hglaser 12y ago
      That page doesn't load for me. :) We went by this: https://twitter.com/dnsimple/status/539521794802483202 https://twitter.com/dnsimple/status/539521794802483202 https://twitter.com/dnsimple/status/539520808599957505 https://twitter.com/dnsimple/status/539520808599957505
  - stevekemp 12y ago
    Amazon's route53 is an awesome service :)
- jonathannorris 12y ago
  We are having Cloudflare issues right now... https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
aberoham 12y ago
If you have an active DNSimple web UI session (or API key) you can change your root nameservers by hitting their web tier directly at 50.31.213.210. We've successfully switched our domains over to nsone.net.
- jredburn 12y ago
  Even without an active session, I was able to get this working by adding an entry to my /etc/hosts file with that IP and dnsimple.com.
  - tortilla 12y ago
    Thanks for the tip! Switched to NSONE.
- mctx 12y ago
  This worked for us - moved the DNS to Cloudflare, kept the domain at DNSimple
scott_karana 12y ago
DNS is so straightforward, so easily distributed, and so fundamental, that I'm always astounded when it's a single point of failure for so many operations. I wonder how many of the affected companies do have redundant appservers and load balancers, but missed this piece of the puzzle...
- asstonian 12y ago
  exactly my thought. the year is 2014 and DNS is least of my problems, since 1999 or something...
- beevek 12y ago
  dns is less easily distributed when fancy features like ALIAS (which dnsimple is widely known for) are in the mix. and wide distribution isn't enough to win vs truly volumetric attacks. it takes a lot of ports and compute to absorb 100Gbps+ attacks which are not uncommon against major providers.
  - scott_karana 12y ago
    DNSimple is widely know for the ALIAS pseudo-"record" because they invented it[1]. Small wonder that a proprietary syntactical sugar leaves you at the mercy of select vendors? As for volumetric attacks: your point is correct, but is irrelevant if you're using multiple vendors, and a specific, single vendor is the target, like it appears here. Your other authoritative servers would be unaffected. 1 http://support.dnsimple.com/articles/alias-record/ http://support.dnsimple.com/articles/alias-record/, or http://webcache.googleusercontent.com/search?q=cache:ST1BABjLDEgJ:support.dnsimple.com/articles/alias-record/+&cd=1&hl=en&ct=clnk http://webcache.googleusercontent.com/search?q=cache:ST1BABj...
    - deleted 12y ago
      [deleted]
    - beevek 12y ago
      good luck finding any major online property or infrastructure that isn't making use of some kind of proprietary syntactical dns sugar. it doesn't mean you can't span providers, but it does mean it takes a lot more work to do so. anyway, you're not wrong, the best approach to mitigate this kind of thing is to leverage multiple dns networks. but doing so is not easy unless the application is still using dns like it was in 1995, and that is increasingly rarely the case.
deleted 12y ago
[deleted]
- dsl 12y ago
  One of the most valuable things I learned in my career was to never kick your competitors when they are down. It upsets the karma gods, and makes you look like a total douchebag. The best thing you can do right now is to reach out and offer your help, privately. Even from a selfish perspective, you'll learn a lot about the attack that is taking them down now which will help you out when the targeted customer inevitably signs up for your service. EDIT: The parent comment was spam from Kris Beevers at NSone.
  - beevek 12y ago
    hi. i did not mean to spam or kick dnsimple, we know them and they are a great company and service. we are actively receiving inbound queries about this from folks asking for help, so thought it made sense to chime in publicly here. but you're right, i should have kept it on topic to the discussion at hand instead of offering anything else up. you're not wrong: in this industry you never kick your competitors when they're down, everyone is subject to the same constraints, attacks, and complications. that wasn't my intention and i said so in the post.
    - cmdrfred 12y ago
      dsl (1402 days old, 4664 karma) - beevek (123 days old, 4 karma). beevek you just lost our business.
      - smachiz 12y ago
        That seems a little extreme; perhaps what he did was in poor taste, but to use karma and account age as an barometer for your business decisions seems crazy (or an arbiter in an internet catfight).
dedene 12y ago
"30 minute ETA from our network provider to begin scrubbing traffic in a location with capacity." https://twitter.com/dnsimplestatus/status/539551209452232705 https://twitter.com/dnsimplestatus/status/539551209452232705
- chrisbolt 12y ago
  It's surprising that they're appear to not be multihomed... http://bgp.he.net/AS32771 http://bgp.he.net/AS32771 Unlike Dyn or CloudFlare: http://bgp.he.net/AS33517 http://bgp.he.net/AS33517 http://bgp.he.net/AS13335 http://bgp.he.net/AS13335
  - feld 12y ago
    They're in ServerCentral's datacenter and ServerCentral is very much multi homed. They wouldn't gain anything by doing native BGP to all these peers in the exact same datacenter when SC's backbone will handle this stuff for them.
    - toomuchtodo 12y ago
      Doing native BGP would allow them to anycast, which would increase their reliability and allow them to sink traffic much more easily. DDOS traffic sink starts announcing your AS and anycast IP block close to the traffic source, sinking that traffic and allowing real traffic through.
      - feld 12y ago
        I'm pretty sure they're only in one of Server Central's datacenters. Anycast won't help. That's why I said they don't gain anything by directly peering.
- jtcchan 12y ago
  "New ETA is 30 minutes from now, trying to get systems wired up in the data center." https://twitter.com/dnsimple/status/539560631863877632 https://twitter.com/dnsimple/status/539560631863877632
soci 12y ago
Unfortunately, it's not the first time it happens, my app is down and customers unhappy. I always wonder, why is it that someone wants to attack a small company like DNSimple ? Is it that they were blackmailed and did not surrender to the criminals? If so, why would anyone be interested in blackmailing such a small company?
- whafro 12y ago
  More likely, I'd guess based on past experience, it's that someone wanted to take down one of their customers, and decided (or found) that the weakest link was the DNS provider.
- callmeed 12y ago
  My guess is one of their customers was being attacked–and this was the easiest route to get to them.
- p8952 12y ago
  It could also be an attack against one of their customers, again for the same reasons. DNS is a very weak point of the overall system so a very good place to target.
scott_karana 12y ago
For those wondering about alternatives to ALIAS: if you use a www subdomain, then you can simply use CNAMEs. (Though the appearance is a matter of taste...) Google, Facebook, etc, all use this approach.
- dbrgn 12y ago
  Yes, but if someone visits your apex domain directly, you want to redirect him/her to the www version... And if you use cloud based hosting where the IP can change quite often, that's a pain to maintain manually.
kjttm 12y ago
Does anyone have a simple explanation or link to an article / blog that explains the naked domain / ALIAS "problem" that DNSSimple solves? I recently set up DNS with DNSimple (due to nudging by Heroku) and am affected by this DDoS. I am still struggling to understand the exact nature of this issue. All of Heroku's documentation is pretty cryptic (to me): "Some DNS hosts provide a way to get CNAME-like functionality at the zone apex using a custom record type. " .. and then on to suggest DNSimple as their first suggestion.
- beevek 12y ago
  http://blog.cloudflare.com/introducing-cname-flattening-rfc-compliant-cnames-at-a-domains-root/ http://blog.cloudflare.com/introducing-cname-flattening-rfc-... is a reasonable explanation. fundamentally a CNAME says "when you get queries for this name, go look at this other name instead". among other things, doing a CNAME at the zone apex means resolvers can't then find your NS, MX, or other records at the apex, which is problematic.
- callmeed 12y ago
  Basically, naked/apex domains generally require an A RECORD pointing to an IP address. Heroku prefers you didn't use A RECORDS at all because the IP addresses in their underlying architecture might change. [1] [1] https://devcenter.heroku.com/articles/apex-domains https://devcenter.heroku.com/articles/apex-domains
deleted 12y ago
[deleted]
stockkid 12y ago
RubyGems.org and Travis-ci are down as a result of this! Not helping with my productivity this morning.
Cantdog 12y ago
Can someone help me understand what happens to email sent to a domain hosted by DNSimple while it's down? I'm hoping it will get queued by the sending server, and make it's way back when DNSimple is up and running. Is that correct?
- abraham 12y ago
  It will depend on the configuration of the sending server. Some will retry for a while, some will return failures.
  - Cantdog 12y ago
    Thanks!
zuccs 12y ago
I moved from Zerigo to DNSimple, and it's been awesome until now! What can you do to prevent this in future? Can you run multiple DNS providers simultaneously? So, ns1/ns2 go to DNSimple, and ns3/ns4 go to another provider?
- robvolk 12y ago
  Yes - that should work. I'm about to make that change to bring back our site. 2 on one name server, 2 on another.
  - zuccs 12y ago
    Cool. Do you need a provider that supports 'zone transfers'? Or is that only to keep things in sync automatically?
    - toomuchtodo 12y ago
      > Do you need a provider that supports 'zone transfers'? No. > Or is that only to keep things in sync automatically? Yes.
      - zuccs 12y ago
        Do you know of any providers that will work nicely/automatically with DNSimple?
        toomuchtodo 12y ago
        I don't unfortunately.
      - dbrgn 12y ago
        DNSimple does not currently support zone transfers (see http://support.dnsimple.com/articles/master-slave-support/ http://support.dnsimple.com/articles/master-slave-support/), but you should write an e-mail to the support (address is on the linked page) requesting it :) With dozens of domains, managing multiple DNS servers is a pain. Zone transfers make this very easy.
webandtech 12y ago
Free solution that worked for me: Set up a free account on cloudflare.com, duplicate all dns records (thankfully I have a simple setup)... but next time I will keep a backup zone file! FYI - Instead of an Alias record on DNSimple, CloudFlare will allow a CNAME record for the root domain using "CNAME flattening". You can now set CloudFlare's DNS service to "bypass Cloudfare" on all records by clicking the icon so you don't get any of their magic (unless you want it). Then add CloudFlare's 2 nameserves to your domain as your first 2 name servers. No need to remove dnsimple's name servers. Now you have 2 DNS providers in case one fails, just make sure the records are the same across them both!
- kyletns 12y ago
  Hey thanks for the post, but how do I add CloudFlare's nameservers? DNSimple won't let me add NS records on the root domain: "You may only delegate subdomains". So in theory, I can fix the www subdomain, but my naked redirect to www won't happen if DNSimple is still down. Any ideas?
  - webandtech 12y ago
    You would set your name servers with your registrar, such as godaddy. If dnsimple is your registrar as well as your DNS provider you may be out of luck until they come back online. The help page is here though: http://support.dnsimple.com/articles/setting-name-servers/ http://support.dnsimple.com/articles/setting-name-servers/
- allr 12y ago
  Anybody can shed some light on the difference between "CNAME flattening" and ALIAS?
  - beevek 12y ago
    they are essentially the same thing -- both are "proprietary" names for the same feature, which is behind-the-scenes recursive CNAME chain lookups by the authoritative nameserver, to return A records directly.
englishm 12y ago
Here's where you can request your cached SERVFAILs be flushed from Google's public DNS (i.e. 8.8.8.8): https://developers.google.com/speed/public-dns/cache https://developers.google.com/speed/public-dns/cache
- bowyakka 12y ago
  Thank you, I wish I could buy you a beer for that
ataco 12y ago
DNSimple is my registrar and (was my only) DNS provider. Now that they're back up I've exported the zone file and imported it to route 53 for redundancy in case this happens again. I also I updated the name servers in DNSimple to be 2 route 53, and 2 DNSimple, in that order. Is that the right way to do it? Does the order of the NS records matter? I set them up so that they're in the same order in both places.
- anderly 12y ago
  That should be right. I'm doing the same thing with CloudFlare. However, it appears that DNSimple won't keep your secondary name servers in the order entered. They are showing for me sorted alphabetically. Apparently, they are working on providing this (http://blog.dnsimple.com/2014/12/incident-report-ddos/ http://blog.dnsimple.com/2014/12/incident-report-ddos/) for failover in case of an event like this again.
anderly 12y ago
You can use my cross-platform cli for dnsimple to export your zone files easily to txt or json format: https://www.npmjs.org/package/dnsimple-cli https://www.npmjs.org/package/dnsimple-cli dnsimple domain record list example.com > example.txt OR dnsimple domain record list example.com --json > example.json
brianarmstrong 12y ago
I wrote a follow-up article about what we at Canopy.co learned from this incident. Check it out (this covers and expands on some of the ideas talked about here): https://medium.com/@brianarmstrong/youre-probably-doing-dns-wrong-like-we-were-6625efaed390 https://medium.com/@brianarmstrong/youre-probably-doing-dns-...
boopadoop 12y ago
DNSimple says it was not a direct attack on them but rather domains being brought over by new customers. Does anyone know the actual target?

125 points by dedene 12y ago

We're kinda tied into DNSimple since we use an ALIAS record for our bare/naked/root domain. Amazon's Route53 supports aliases, but via a 301 redirect, which doesn't work in an SSL context (without browser warnings). Nonetheless, we just spun up a Route53 zone, exported our zone from DNSimple, imported to Route53, and hand-migrated our ALIAS records to static A records in the new zone. Not perfect or permanent, but we've gotten around the outage. Also, I just learned that pointhq has (seemingly-undocumented) support for ALIAS records in the same style as DNSimple, so this could be another avenue to explore.

petercooper 12y ago
How are you exporting the zone with things as they are?
- algiecas 12y ago
  same question here - how did you do this while dnsimple.com is inaccessible?
  - zuccs 12y ago
    You can force your hosts file locally. Set "dnsimple.com" to "50.31.213.210". On Mac use "sudo pico /etc/hosts" and follow the formatting of the other entries.
- whafro 12y ago
  Oof, really good point – I suppose I got lucky in that I was at dnsimple.com just a few hours ago for other purposes. Their DNS was still cached for me, and it all worked flawlessly. I didn't even suspect their outage extended to their web servers, though that makes perfect sense. FWIW, the IP I have cached is 50.31.213.210.
  - Pwnguinz 12y ago
    How were you able to find out what IP was cached for a particular domain?
mef 12y ago
FYI you can use ALIAS records without a 301 redirect if the ALIAS is to an ELB.
- deleted 12y ago
  [deleted]
EwanToo 12y ago
A few people do ALIAS style records, DNSMadeEasy and EasyDNS do "ANAME" records which are pretty much the same.
- joshdotsmith 12y ago
  Will these services not have the same thin pipe issue that's currently affecting DNSimple?
  - dentarg 12y ago
    Not that I have any reliable info, but what I've heard, DNS Made Easy is a pretty stable and established DNS provider. They brag about "99.9999% uptime history" at http://www.dnsmadeeasy.com/technology/ http://www.dnsmadeeasy.com/technology/. Though they doesn't seem as innovative and nice as DNSimple. Really hope things work out for DNSimple (really like the idea of their beta feature GitHub sync).
    - stevekemp 12y ago
      How does their beta-feature with github work? I can't find any obvious link and it seems frustratingly close to something I offer over at https://dns-api.com/ https://dns-api.com/ .. (I wrap Amazon's route53 with DNS entries read from github/gitbucket/similar.)
      - zuccs 12y ago
        Can you use custom nameservers with your service?
        stevekemp 12y ago
        When you upload a new zone it will be assigned a set of nameservers - four. You can't choose what they are in advance, and you can't specify the TLDs. So you might end up with "ns-1933.awsdns-49.co.uk.", "ns-1109.awsdns-10.org.", or similar. Does that answer the question? I'm a little hazy on what you're actually asking.

pkfrank 12y ago

Can anyone expand on what this means: "This attack is volumetric in nature." (?)

binarylogicben 12y ago
Pretty sure it means the traffic is overwhelming them.
styger 12y ago
According to http://www.arbornetworks.com/attack-ddos http://www.arbornetworks.com/attack-ddos: "DDoS attacks...will generally fall into one of three broad categories: Volumetric Attacks: Attempt to consume the bandwidth either within the target network/service, or between the target network/service and the rest of the Internet. These attacks are simply about causing congestion."

jameskilton 12y ago

We can watch this happen live @ http://map.ipviking.com/ http://map.ipviking.com/ Fascinating traffic floods from various locations, but the attack is not continuous.

jtcchan 12y ago
Are these live attacks on any servers?
shoxxx 12y ago
Noticed a lot of hits to St. Louis, where they're based. Do you think this has anything to do with the attacks?
diminoten 12y ago
How does this link show this specific attack? As far as I understand, ipviking simply hosts honeypots around the world and uses those to graph "attacks" against IP blocks, etc. I would very much like someone to correct me if this assumption is incorrect, because it'd be neat to actually watch targeted DDoS attacks, but I don't think that's what ipviking is offering.

deleted 12y ago

[deleted]

beck5 12y ago

What are the recommended practices to prevent too much down time when your DNS provider goes down?

mjibson 12y ago
My site is currently offline from this attack. I am considering that providers like DNSimple simply cannot provide the networking availability to mitigate these kinds of attacks. This is because the solution to these kinds of attacks, often, is that you need a larger pipe than the attackers. Very few people are good at both <some service> and having a huge pipe. For web apps, you can use CloudFlare, which does have a bigger pipe and is designed to mitigate this. But DNS is not a web app, so you can't just put DNSimple behind CloudFlare. Hence, I'm going to try CloudFlare (assuming they take over DNS hosting, I need to check) and Google Cloud DNS, because then all parts of my site (from DNS to CSS hosting) will be with providers with bigger pipes than attackers can create. Hopefully that will prevent this kind of attack from taking my site down.
- throwaway90446 12y ago
  Yes, CloudFlare has a full-featured DNS offering, even if you don't use their proxying services.
  - stevekemp 12y ago
    Which has had a lot of problems this past week: https://news.ycombinator.com/item?id=8665367 https://news.ycombinator.com/item?id=8665367
    - throwaway90446 12y ago
      CloudFlare != CloudFront
      - stevekemp 12y ago
        My apologies, cloudfront had issues, cloudflare did not.

ericskiff 12y ago

For anyone else who needs to mitigate this in a hurry: Set up a new account on another host that does ALIAS records (I used pointDNS) Create your new record without much in it Change your nameservers on your domain now - they'll take time to propagate Fill in the records on your domain. If you can't remember them, print out most of your existing records with dig yourdomain.com ANY Add the rest of the records to pointDNS Wait for the new Nameservers to propagate (0-24 hours - it took 15-30 min for us on a small-medium traffic domain today during sales crunch)

scott_karana 12y ago
For those with a more deterministic bent: "propagation" time has a maximum bound of your TTL, which will show with any dig queries.
- colmmacc 12y ago
  Propagation is bound at the DNS TTL plus whatever time it takes your DNS provider/setup to relay records to all of its authoritative servers.
  - scott_karana 12y ago
    Hmm, never considered that. Is it a slow, static process for ISPs to do that? I just assumed that they ran more-or-less stock DNS resolvers with in-memory caches.
colmmacc 12y ago
Publishing an incomplete copy of your zone seems like a bad idea, an ANY dig at the zone apex is going to retrieve less than 30% of the records (on average) for real-world zones. Negative responses in DNS are cacheable - so you can easily poison yourself. If you really really want to do it anyway, most caches use either the TTL on your SOA record, or the final field in the SOA record as the negative cache TTL; so lower both of those values to something like 60 seconds.

shoxxx 12y ago

Anyone switching from DNSimple? I really don't want to, but we've been down for almost 3 hours. I've seen chatter about Cloudfare and it looks pretty good, reviews?

hglaser 12y ago
We switched periscope.io from DNSimple to Amazon Route 53. DNSimple doesn't have an exporter so it took about an hour, including having one engineer review the other engineer's work. Many customers were able to resolve the domain in the minutes immediately following the switch, and the rest seem to be trickling in.
- walmartian 12y ago
  Are you talking about a zone file exporter? DNSimple does have one, we just used it to migrate to Route53. http://support.dnsimple.com/articles/zone-export/ http://support.dnsimple.com/articles/zone-export/ EDIT: right, must have been able to log in during a brief period where dnsimple was not down.
  - shoxxx 12y ago
    link doesnt work
  - hglaser 12y ago
    That page doesn't load for me. :) We went by this: https://twitter.com/dnsimple/status/539521794802483202 https://twitter.com/dnsimple/status/539521794802483202 https://twitter.com/dnsimple/status/539520808599957505 https://twitter.com/dnsimple/status/539520808599957505
- stevekemp 12y ago
  Amazon's route53 is an awesome service :)
jonathannorris 12y ago
We are having Cloudflare issues right now... https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/

aberoham 12y ago

If you have an active DNSimple web UI session (or API key) you can change your root nameservers by hitting their web tier directly at 50.31.213.210. We've successfully switched our domains over to nsone.net.

jredburn 12y ago
Even without an active session, I was able to get this working by adding an entry to my /etc/hosts file with that IP and dnsimple.com.
- tortilla 12y ago
  Thanks for the tip! Switched to NSONE.
mctx 12y ago
This worked for us - moved the DNS to Cloudflare, kept the domain at DNSimple

scott_karana 12y ago

DNS is so straightforward, so easily distributed, and so fundamental, that I'm always astounded when it's a single point of failure for so many operations. I wonder how many of the affected companies do have redundant appservers and load balancers, but missed this piece of the puzzle...

asstonian 12y ago
exactly my thought. the year is 2014 and DNS is least of my problems, since 1999 or something...
beevek 12y ago
dns is less easily distributed when fancy features like ALIAS (which dnsimple is widely known for) are in the mix. and wide distribution isn't enough to win vs truly volumetric attacks. it takes a lot of ports and compute to absorb 100Gbps+ attacks which are not uncommon against major providers.
- scott_karana 12y ago
  DNSimple is widely know for the ALIAS pseudo-"record" because they invented it[1]. Small wonder that a proprietary syntactical sugar leaves you at the mercy of select vendors? As for volumetric attacks: your point is correct, but is irrelevant if you're using multiple vendors, and a specific, single vendor is the target, like it appears here. Your other authoritative servers would be unaffected. 1 http://support.dnsimple.com/articles/alias-record/ http://support.dnsimple.com/articles/alias-record/, or http://webcache.googleusercontent.com/search?q=cache:ST1BABjLDEgJ:support.dnsimple.com/articles/alias-record/+&cd=1&hl=en&ct=clnk http://webcache.googleusercontent.com/search?q=cache:ST1BABj...
  - deleted 12y ago
    [deleted]
  - beevek 12y ago
    good luck finding any major online property or infrastructure that isn't making use of some kind of proprietary syntactical dns sugar. it doesn't mean you can't span providers, but it does mean it takes a lot more work to do so. anyway, you're not wrong, the best approach to mitigate this kind of thing is to leverage multiple dns networks. but doing so is not easy unless the application is still using dns like it was in 1995, and that is increasingly rarely the case.

deleted 12y ago

[deleted]

dsl 12y ago
One of the most valuable things I learned in my career was to never kick your competitors when they are down. It upsets the karma gods, and makes you look like a total douchebag. The best thing you can do right now is to reach out and offer your help, privately. Even from a selfish perspective, you'll learn a lot about the attack that is taking them down now which will help you out when the targeted customer inevitably signs up for your service. EDIT: The parent comment was spam from Kris Beevers at NSone.
- beevek 12y ago
  hi. i did not mean to spam or kick dnsimple, we know them and they are a great company and service. we are actively receiving inbound queries about this from folks asking for help, so thought it made sense to chime in publicly here. but you're right, i should have kept it on topic to the discussion at hand instead of offering anything else up. you're not wrong: in this industry you never kick your competitors when they're down, everyone is subject to the same constraints, attacks, and complications. that wasn't my intention and i said so in the post.
  - cmdrfred 12y ago
    dsl (1402 days old, 4664 karma) - beevek (123 days old, 4 karma). beevek you just lost our business.
    - smachiz 12y ago
      That seems a little extreme; perhaps what he did was in poor taste, but to use karma and account age as an barometer for your business decisions seems crazy (or an arbiter in an internet catfight).

dedene 12y ago

"30 minute ETA from our network provider to begin scrubbing traffic in a location with capacity." https://twitter.com/dnsimplestatus/status/539551209452232705 https://twitter.com/dnsimplestatus/status/539551209452232705

chrisbolt 12y ago
It's surprising that they're appear to not be multihomed... http://bgp.he.net/AS32771 http://bgp.he.net/AS32771 Unlike Dyn or CloudFlare: http://bgp.he.net/AS33517 http://bgp.he.net/AS33517 http://bgp.he.net/AS13335 http://bgp.he.net/AS13335
- feld 12y ago
  They're in ServerCentral's datacenter and ServerCentral is very much multi homed. They wouldn't gain anything by doing native BGP to all these peers in the exact same datacenter when SC's backbone will handle this stuff for them.
  - toomuchtodo 12y ago
    Doing native BGP would allow them to anycast, which would increase their reliability and allow them to sink traffic much more easily. DDOS traffic sink starts announcing your AS and anycast IP block close to the traffic source, sinking that traffic and allowing real traffic through.
    - feld 12y ago
      I'm pretty sure they're only in one of Server Central's datacenters. Anycast won't help. That's why I said they don't gain anything by directly peering.
jtcchan 12y ago
"New ETA is 30 minutes from now, trying to get systems wired up in the data center." https://twitter.com/dnsimple/status/539560631863877632 https://twitter.com/dnsimple/status/539560631863877632

soci 12y ago

Unfortunately, it's not the first time it happens, my app is down and customers unhappy. I always wonder, why is it that someone wants to attack a small company like DNSimple ? Is it that they were blackmailed and did not surrender to the criminals? If so, why would anyone be interested in blackmailing such a small company?

whafro 12y ago
More likely, I'd guess based on past experience, it's that someone wanted to take down one of their customers, and decided (or found) that the weakest link was the DNS provider.
callmeed 12y ago
My guess is one of their customers was being attacked–and this was the easiest route to get to them.
p8952 12y ago
It could also be an attack against one of their customers, again for the same reasons. DNS is a very weak point of the overall system so a very good place to target.

scott_karana 12y ago

For those wondering about alternatives to ALIAS: if you use a www subdomain, then you can simply use CNAMEs. (Though the appearance is a matter of taste...) Google, Facebook, etc, all use this approach.

dbrgn 12y ago
Yes, but if someone visits your apex domain directly, you want to redirect him/her to the www version... And if you use cloud based hosting where the IP can change quite often, that's a pain to maintain manually.

kjttm 12y ago

Does anyone have a simple explanation or link to an article / blog that explains the naked domain / ALIAS "problem" that DNSSimple solves? I recently set up DNS with DNSimple (due to nudging by Heroku) and am affected by this DDoS. I am still struggling to understand the exact nature of this issue. All of Heroku's documentation is pretty cryptic (to me): "Some DNS hosts provide a way to get CNAME-like functionality at the zone apex using a custom record type. " .. and then on to suggest DNSimple as their first suggestion.

beevek 12y ago
http://blog.cloudflare.com/introducing-cname-flattening-rfc-compliant-cnames-at-a-domains-root/ http://blog.cloudflare.com/introducing-cname-flattening-rfc-... is a reasonable explanation. fundamentally a CNAME says "when you get queries for this name, go look at this other name instead". among other things, doing a CNAME at the zone apex means resolvers can't then find your NS, MX, or other records at the apex, which is problematic.
callmeed 12y ago
Basically, naked/apex domains generally require an A RECORD pointing to an IP address. Heroku prefers you didn't use A RECORDS at all because the IP addresses in their underlying architecture might change. [1] [1] https://devcenter.heroku.com/articles/apex-domains https://devcenter.heroku.com/articles/apex-domains

deleted 12y ago

[deleted]

stockkid 12y ago

RubyGems.org and Travis-ci are down as a result of this! Not helping with my productivity this morning.

Cantdog 12y ago

Can someone help me understand what happens to email sent to a domain hosted by DNSimple while it's down? I'm hoping it will get queued by the sending server, and make it's way back when DNSimple is up and running. Is that correct?

abraham 12y ago
It will depend on the configuration of the sending server. Some will retry for a while, some will return failures.
- Cantdog 12y ago
  Thanks!

zuccs 12y ago

I moved from Zerigo to DNSimple, and it's been awesome until now! What can you do to prevent this in future? Can you run multiple DNS providers simultaneously? So, ns1/ns2 go to DNSimple, and ns3/ns4 go to another provider?

robvolk 12y ago
Yes - that should work. I'm about to make that change to bring back our site. 2 on one name server, 2 on another.
- zuccs 12y ago
  Cool. Do you need a provider that supports 'zone transfers'? Or is that only to keep things in sync automatically?
  - toomuchtodo 12y ago
    > Do you need a provider that supports 'zone transfers'? No. > Or is that only to keep things in sync automatically? Yes.
    - zuccs 12y ago
      Do you know of any providers that will work nicely/automatically with DNSimple?
      - toomuchtodo 12y ago
        I don't unfortunately.
    - dbrgn 12y ago
      DNSimple does not currently support zone transfers (see http://support.dnsimple.com/articles/master-slave-support/ http://support.dnsimple.com/articles/master-slave-support/), but you should write an e-mail to the support (address is on the linked page) requesting it :) With dozens of domains, managing multiple DNS servers is a pain. Zone transfers make this very easy.

webandtech 12y ago

Free solution that worked for me: Set up a free account on cloudflare.com, duplicate all dns records (thankfully I have a simple setup)... but next time I will keep a backup zone file! FYI - Instead of an Alias record on DNSimple, CloudFlare will allow a CNAME record for the root domain using "CNAME flattening". You can now set CloudFlare's DNS service to "bypass Cloudfare" on all records by clicking the icon so you don't get any of their magic (unless you want it). Then add CloudFlare's 2 nameserves to your domain as your first 2 name servers. No need to remove dnsimple's name servers. Now you have 2 DNS providers in case one fails, just make sure the records are the same across them both!

kyletns 12y ago
Hey thanks for the post, but how do I add CloudFlare's nameservers? DNSimple won't let me add NS records on the root domain: "You may only delegate subdomains". So in theory, I can fix the www subdomain, but my naked redirect to www won't happen if DNSimple is still down. Any ideas?
- webandtech 12y ago
  You would set your name servers with your registrar, such as godaddy. If dnsimple is your registrar as well as your DNS provider you may be out of luck until they come back online. The help page is here though: http://support.dnsimple.com/articles/setting-name-servers/ http://support.dnsimple.com/articles/setting-name-servers/
allr 12y ago
Anybody can shed some light on the difference between "CNAME flattening" and ALIAS?
- beevek 12y ago
  they are essentially the same thing -- both are "proprietary" names for the same feature, which is behind-the-scenes recursive CNAME chain lookups by the authoritative nameserver, to return A records directly.

englishm 12y ago

Here's where you can request your cached SERVFAILs be flushed from Google's public DNS (i.e. 8.8.8.8): https://developers.google.com/speed/public-dns/cache https://developers.google.com/speed/public-dns/cache

bowyakka 12y ago
Thank you, I wish I could buy you a beer for that

ataco 12y ago

DNSimple is my registrar and (was my only) DNS provider. Now that they're back up I've exported the zone file and imported it to route 53 for redundancy in case this happens again. I also I updated the name servers in DNSimple to be 2 route 53, and 2 DNSimple, in that order. Is that the right way to do it? Does the order of the NS records matter? I set them up so that they're in the same order in both places.

anderly 12y ago
That should be right. I'm doing the same thing with CloudFlare. However, it appears that DNSimple won't keep your secondary name servers in the order entered. They are showing for me sorted alphabetically. Apparently, they are working on providing this (http://blog.dnsimple.com/2014/12/incident-report-ddos/ http://blog.dnsimple.com/2014/12/incident-report-ddos/) for failover in case of an event like this again.

anderly 12y ago

You can use my cross-platform cli for dnsimple to export your zone files easily to txt or json format: https://www.npmjs.org/package/dnsimple-cli https://www.npmjs.org/package/dnsimple-cli dnsimple domain record list example.com > example.txt OR dnsimple domain record list example.com --json > example.json

brianarmstrong 12y ago

I wrote a follow-up article about what we at Canopy.co learned from this incident. Check it out (this covers and expands on some of the ideas talked about here): https://medium.com/@brianarmstrong/youre-probably-doing-dns-wrong-like-we-were-6625efaed390 https://medium.com/@brianarmstrong/youre-probably-doing-dns-...

boopadoop 12y ago

DNSimple says it was not a direct attack on them but rather domains being brought over by new customers. Does anyone know the actual target?