8 ms·

HTTPS on Stack Overflow: The End of a Long Road

574 points by Nick-Craver 9y ago

phsource 9y ago
This is incredibly detailed; in short, CDNs, cookies/authentication , tons of subdomains, and 3rd-party/user-generated content make it a pain to move onto HTTPS. I was chatting with a non-engineer friend about why it's hard to estimate how long tasks often take, and this seems like a prime illustration: the dependencies are endless. I also love the Easter egg: "The password to our data center is pickles. I didn’t think anyone would read this far and it seemed like a good place to store it."
- niklasrde 9y ago
  Just know the username and you can log onto https://stackoverflow.com/admin.php https://stackoverflow.com/admin.php
  - oefrha 9y ago
    Warning: link's NSFW.
    - nhylated 9y ago
      It opens a random youtube video everytime (I got several 10 hour vids including Jeff Goldblum laugh and relaxing hairdryer sound).
      - bpicolo 9y ago
        Seems there's a limited set. It's a pool of 10 hour vids
deleted 9y ago
[deleted]
lmm 9y ago
> The password to our data center is pickles. I didn’t think anyone would read this far and it seemed like a good place to store it. You ought to have more confidence in your writing. BRB stealing all your servers.
- DamonHD 9y ago
  Only if you get there first. And it may in fact be pickles2.
  - ghvknvgvh 9y ago
    This guy works at Slashdot by the way. Nothing worth stealing in their servers, methinks.
  - SimbaOnSteroids 9y ago
    It's actually hunter2
    - hunter2_ 9y ago
      Shhh
kalleboo 9y ago
Note to self: Use subdirectories, not subdomains in the future
- baby 9y ago
  TLS kills this kind of "cool" features which is kind of sad :( Unless you can afford wildcard certs. What's the argument behind LetsEncrypt not doing that? Extended Validation stuff?
  - tomschlick 9y ago
    Probably their agreements with their partner CAs. Given that those partners sell wildcards themselves...
    - prdonahue 9y ago
      Their "Let’s Encrypt Authority X3" intermediate is signed by their own root (ISRG Root X1). See https://letsencrypt.org/certificates/ https://letsencrypt.org/certificates/.
  - cookiecaper 9y ago
    Subdomains were killed by SEO a long time ago (afaik, Google does not transfer domain PageRank credit to subdomains), so this is not limited solely by the cost of wildcard certs.
    - baby 9y ago
      But this is orthogonal to the issue of LetsEncrypt not delivering wildcard certs.
    - baby 9y ago
      But this is orthogonal to the issue of LetsEncrypt not delivering wildcard certs.
  - eridius 9y ago
    There's a long StackExchange answer about this: https://security.stackexchange.com/a/158164 https://security.stackexchange.com/a/158164 But it boils down to there being no practical way for Let's Encrypt to automatically validate that a wildcard certificate is safe to issue.
Zekio 9y ago
Your blog posts are always an interesting read
mrunkel 9y ago
At $previous_job we once turned on HTTPS for our entire customer website and online store, only to have our customer support team be bombarded by phone calls claiming that our "website was down." After much teeth gnashing and research, we determined that a large segment of our user base was still using WinXP and the encryption protocols we offered weren't available to them. We didn't think this would be a problem because the current version of the software wasn't compatible with WinXP any longer. There was some debate internally whether the better fix was to including the legacy encryption protocols or just leave the HTTP version of the site running and use Strict-Transport-Security to move capable browsers to HTTPS. In the end we had to include the legacy protocols so those customers could use our online store.
- adventured 9y ago
  Out of curiosity - roughly what year was that and what percentage of the customer base would you say was still on Windows XP at the time?
- gatlinnewhouse 9y ago
  > We didn't think this would be a problem because the current version of the software wasn't compatible with WinXP any longer. > There was some debate internally whether the better fix was to including the legacy encryption protocols or just leave the HTTP version of the site running and use Strict-Transport-Security to move capable browsers to HTTPS. Where can I read about this? Is there any way to display a special "Your browser is outdated" page for the users on WinXP? Sorry if these seem like basic questions. I am just curious and would like to hear some expert advice.
  - shortimer 9y ago
    At our place, we put a redirect on the front end networking device that detected if a browser couldn't support more modern encryption protocols, and sent them to an HTTP information page (instead of to the application itself) if so. This allowed us to update the core app to force newer protocols, while still providing some sort of UX for those left behind. We used Piwik to track the hits on the redirect page to get a sense for how many users were left behind.
tomschlick 9y ago
Just a reminder, HTTPS isn't enough. Be sure to turn the other security knobs with headers... https://securityheaders.io/?q=https%3A%2F%2Fstackoverflow.com&followRedirects=on https://securityheaders.io/?q=https%3A%2F%2Fstackoverflow.co...
- mixmastamyk 9y ago
  Helpful site, but all these headers will slow down a site that doesn't need them. Too bad they aren't defaults. Hopefully http2 mitigates that enough.
  - tomschlick 9y ago
    The only header that I can think of that might slow down a site is Content-Security-Policy. Even that is negligible as long as you don't have 1000 entries.
  - hdhzy 9y ago
    Http2 compresses headers so it they are not changing the amount of overhead is negligible (see HPACK).
    - Nick-Craver 9y ago
      Well, yes and no - it depends on the length. Let's take 3 common examples. Here's GitHub's relevant headers (that we don't have): Content-Security-Policy:default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src render.githubusercontent.com; connect-src 'self' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-user-asset-79cafe.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; img-src 'self' data: assets-cdn.github.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' assets-cdn.github.com Public-Key-Pins:max-age=5184000; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="RRM1dGqnDFsCJXBTHky16vi1obOlCgFFn/yOhI/y+ho="; pin-sha256="k2v657xBsOVe1PQRwOsHsw3bsGT2VzIqz5K+59sNQws="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="IQBnNBEiFuhj+8x6X8XLgh01V9Ic5/V3IRQLNFFc7v4="; pin-sha256="iie1VXtL7HzAMF+/PVPR9xzT80kQxdZeJ+zduCB3uj0="; pin-sha256="LvRiGEjRqfzurezaWuj8Wie2gyHMrW5Q06LspMnox7A="; includeSubDomains Those are 1220 bytes. I'm not sure what they'll compress down to, but it's still non-trivial and not near 0 (anyone want to run the numbers?). The same pair of headers are 969 bytes for facebook.com and 2,772 for gmail.com. I don't know what ours would be - since we're open-ended on the image domain side it's a bit apples-to-oranges compared to the big players. When you take into account that you can only send 10 packets down the first response (in almost all cases today) due to TCP congestion window specifications (google: CWND), they get more expensive as a percentage of what you can send. It may be that you can't send enough of the page to render, or the browser isn't getting to a critical stylesheet link until the second wave of packets after the ACK. This can greatly affect load times. Does HPACK affect this? Yeah absolutely, but I disagree on "negligible". It depends, and if something critical gets pushed to that 11th packet as a result, you can drastically increase actual page render time for users. If it helps, I did a blog post with some details about this a while back: https://nickcraver.com/blog/2015/03/24/optimization-considerations/ https://nickcraver.com/blog/2015/03/24/optimization-consider...
janwillemb 9y ago
Wow, I didn't expect this ("switching" to HTTPS) to be so hard.
- Ajedi32 9y ago
  It very much depends on the complexity and scale of your site. StackOverflow is a bit of an extreme case. For example, if instead of having hundreds of domains serving millions of users with tons of user-generated content you're just serving static content from a single server on a small site, the entire process for you might actually be as simple as just running `certbot-auto` on the production server. I suspect the difficulty of switching for most sites will fall somewhere between these two extremes.
- irrational 9y ago
  Yeah, we've been working on this for about a year (not continually, but as we have time to try to work through the problems). We do use subdomains though, so that is part of the problem. We keep feeling like we are getting close, but then we run into another issue. It's like a rabbit hole that has no bottom.
  - skyisblue 9y ago
    > We keep feeling like we are getting close, but then we run into another issue. It's like a rabbit hole that has no bottom. That's exactly what we experienced migrating a bunch of sites to https. There were so many things that we didn't anticipate.
user5994461 9y ago
Funny how the main reason for lack of SSL is said to be the lack of support from 3rd party services... and the first service quoted is ads. https://nickcraver.com/blog/2013/04/23/stackoverflow-com-the-road-to-ssl/ https://nickcraver.com/blog/2013/04/23/stackoverflow-com-the...
- therealdrag0 9y ago
  Funny how?
quintin 9y ago
Has anyone tried running Fastly behind Cloudflare? Are the tradeoffs worth it?
- tyingq 9y ago
  Is there some reason other than cost to do that? Curious.
  - quintin 9y ago
    Mitigate attacks much better than Fastly, flatten CNAME etc.
    - bpicolo 9y ago
      > Mitigate attacks much better than Fastly if that's the concern, probably just better to configure switching than put both in front all the time.
- pbarnes_1 9y ago
  Why would you want to double your CDN costs for negligible benefit?
tomschlick 9y ago
Side question: any plans for IPv6?
- astrodust 9y ago
  Given the scale of Stack Overflow, you'd think they could set up AAAA records that point to a proper TLS 1.3+ server and leave the peasants on IPv4 going to one that's more...accommodating.
  - Nick-Craver 9y ago
    We could - but the network side isn't the problem. There's a lot of logging, user banning, etc. pieces that need IPv6 love first. We just haven't had the time yet. There are network bits we'd have to evaluate heavily as well, e.g. firewall rules - basically the very limited benefits don't make it a priority, yet. When things change there, we'll do it.
- alienth 9y ago
  Not any immediate plans. Decent amount of development is necessary there. There are so many places in our various systems that work with IP addresses, and many of them don't support v6 addresses.
  - ec109685 9y ago
    One thing that places do is support IPv6 on the outside and translate to IPv4 calls in the next hop beyond the servers that sit beyond the edge.
- tepmoc 9y ago
  Here recent detailed answer from Nick https://meta.stackoverflow.com/questions/348223/stack-overflow-not-reachable-via-ipv6/348224#348224 https://meta.stackoverflow.com/questions/348223/stack-overfl...
jontro 9y ago
Regarding the section "Mistakes: APIs and .internal" Why wouldn't they use split horizon DNS for this? Seems like the perfect use case
- Nick-Craver 9y ago
  Split horizon would point you at the same data center, rather than the writeable one. So that's more of a .local than a .internal. We discussed this, but ultimately the AD version we're on (pre-2016 Geo-DNS) it's not actually supported the way you'd need, and it's a nightmare to debug. We'd consider it for a .local, when the support it properly there in 2016. Even subnet prioritization is busted internally, so that's a bit of an issue. Evidently no one tried to use a wildcard with dual records on 2 subnets before (we prioritize the /16, which is a data center) and it's totally busted. Microsoft has simply said this isn't supported and won't be fixed. A records work, unless they're a wildcard. So specifically, the <star>.stackexchange.com record which we mirror internally at <star>.stackexchange.com.internal for that IP set is particularly problematic. TL;DR: Microsoft AD DNS is busted and they have no intention of fixing it. It's not worth it to try and work around it.
  - jontro 9y ago
    Interesting, thanks!
fareesh 9y ago
Despite the "Google gives a boost to https" reasoning, which comes from Google itself, in practice I've read several first-hand accounts of how traffic (non XP) dropped significantly right after the switch.
gbrayut 9y ago
If you like working on these kinds of projects, the SRE team at Stack Overflow is hiring and we allow remote work full time! https://stackoverflow.com/jobs/143725/site-reliability-engineer-generalist-stack-overflow https://stackoverflow.com/jobs/143725/site-reliability-engin...
dzdt 9y ago
Stack Exchange is no longer available from my workplace due to this change. We have a strict no-posting-code-fragments policy, and SE was viewed as too risky to allow without some restriction in place to make it read only. Before HTTPS, the IT department had worked out such a read-only restriction by blocking the SE login with firewall rules. But with HTTPS that kludge is no longer possible, so the site is blocked.
- maddyboo 9y ago
  Wow, that sounds ridiculous. What's the reasoning behind that policy?
  - zcdziura 9y ago
    Seems obvious that someone high up on the corporate ladder, with no practical knowledge in how the nitty-gritty work gets done, made the decision. Probably to "minimize IP theft".
    - astrodust 9y ago
      "What do you mean our competitor is using 'for' loops? We invented those!"
- Shoop 9y ago
  What sort of company do you work at? Why can't everyone just be told not to post code?
  - dogruck 9y ago
    What sort of company do you work at where every employee obeys every directive?
    - sqren 9y ago
      A company that trust their employees. There are so many ways to get around this anyway so it doesn't make sense to try to enforce it in the first place (considering the issues that follows).
bullen 9y ago
I said it before and I'll say it again: HTTPS is a waste of electricity.
merb 9y ago
sadly that haproxy (which stack overflow uses) does not support http/2 directly, you need to terminate it via nginx or anything else.
jcadam 9y ago
I work at a government facility. Stack Overflow and github are now both blocked (in addition to all social media and webmail). But Hacker News is apparently ok.
souenzzo 9y ago
How many questions on stackoverflow to these migration?
gub09 9y ago
It would be better if scripts like jquery were not encrypted. This forces users to use e.g. a google service instead of caching/hosting the scripts themselves or getting them from another CDN. I do not understand why so many people do not consider the privacy implications of every single webpage requiring calls to google services. There are ways to avoid this, but it gets a lot more complicated when that requires MITM methods for SSL. Please: use a non-tracking CDN, host it yourself, or at least leave it HTTP.

574 points by Nick-Craver 9y ago

This is incredibly detailed; in short, CDNs, cookies/authentication , tons of subdomains, and 3rd-party/user-generated content make it a pain to move onto HTTPS. I was chatting with a non-engineer friend about why it's hard to estimate how long tasks often take, and this seems like a prime illustration: the dependencies are endless. I also love the Easter egg: "The password to our data center is pickles. I didn’t think anyone would read this far and it seemed like a good place to store it."

niklasrde 9y ago
Just know the username and you can log onto https://stackoverflow.com/admin.php https://stackoverflow.com/admin.php
- oefrha 9y ago
  Warning: link's NSFW.
  - nhylated 9y ago
    It opens a random youtube video everytime (I got several 10 hour vids including Jeff Goldblum laugh and relaxing hairdryer sound).
    - bpicolo 9y ago
      Seems there's a limited set. It's a pool of 10 hour vids

deleted 9y ago

[deleted]

lmm 9y ago

> The password to our data center is pickles. I didn’t think anyone would read this far and it seemed like a good place to store it. You ought to have more confidence in your writing. BRB stealing all your servers.

DamonHD 9y ago
Only if you get there first. And it may in fact be pickles2.
- ghvknvgvh 9y ago
  This guy works at Slashdot by the way. Nothing worth stealing in their servers, methinks.
- SimbaOnSteroids 9y ago
  It's actually hunter2
  - hunter2_ 9y ago
    Shhh

kalleboo 9y ago

Note to self: Use subdirectories, not subdomains in the future

baby 9y ago
TLS kills this kind of "cool" features which is kind of sad :( Unless you can afford wildcard certs. What's the argument behind LetsEncrypt not doing that? Extended Validation stuff?
- tomschlick 9y ago
  Probably their agreements with their partner CAs. Given that those partners sell wildcards themselves...
  - prdonahue 9y ago
    Their "Let’s Encrypt Authority X3" intermediate is signed by their own root (ISRG Root X1). See https://letsencrypt.org/certificates/ https://letsencrypt.org/certificates/.
- cookiecaper 9y ago
  Subdomains were killed by SEO a long time ago (afaik, Google does not transfer domain PageRank credit to subdomains), so this is not limited solely by the cost of wildcard certs.
  - baby 9y ago
    But this is orthogonal to the issue of LetsEncrypt not delivering wildcard certs.
  - baby 9y ago
    But this is orthogonal to the issue of LetsEncrypt not delivering wildcard certs.
- eridius 9y ago
  There's a long StackExchange answer about this: https://security.stackexchange.com/a/158164 https://security.stackexchange.com/a/158164 But it boils down to there being no practical way for Let's Encrypt to automatically validate that a wildcard certificate is safe to issue.

Zekio 9y ago

Your blog posts are always an interesting read

mrunkel 9y ago

At $previous_job we once turned on HTTPS for our entire customer website and online store, only to have our customer support team be bombarded by phone calls claiming that our "website was down." After much teeth gnashing and research, we determined that a large segment of our user base was still using WinXP and the encryption protocols we offered weren't available to them. We didn't think this would be a problem because the current version of the software wasn't compatible with WinXP any longer. There was some debate internally whether the better fix was to including the legacy encryption protocols or just leave the HTTP version of the site running and use Strict-Transport-Security to move capable browsers to HTTPS. In the end we had to include the legacy protocols so those customers could use our online store.

adventured 9y ago
Out of curiosity - roughly what year was that and what percentage of the customer base would you say was still on Windows XP at the time?
gatlinnewhouse 9y ago
> We didn't think this would be a problem because the current version of the software wasn't compatible with WinXP any longer. > There was some debate internally whether the better fix was to including the legacy encryption protocols or just leave the HTTP version of the site running and use Strict-Transport-Security to move capable browsers to HTTPS. Where can I read about this? Is there any way to display a special "Your browser is outdated" page for the users on WinXP? Sorry if these seem like basic questions. I am just curious and would like to hear some expert advice.
- shortimer 9y ago
  At our place, we put a redirect on the front end networking device that detected if a browser couldn't support more modern encryption protocols, and sent them to an HTTP information page (instead of to the application itself) if so. This allowed us to update the core app to force newer protocols, while still providing some sort of UX for those left behind. We used Piwik to track the hits on the redirect page to get a sense for how many users were left behind.

tomschlick 9y ago

Just a reminder, HTTPS isn't enough. Be sure to turn the other security knobs with headers... https://securityheaders.io/?q=https%3A%2F%2Fstackoverflow.com&followRedirects=on https://securityheaders.io/?q=https%3A%2F%2Fstackoverflow.co...

mixmastamyk 9y ago
Helpful site, but all these headers will slow down a site that doesn't need them. Too bad they aren't defaults. Hopefully http2 mitigates that enough.
- tomschlick 9y ago
  The only header that I can think of that might slow down a site is Content-Security-Policy. Even that is negligible as long as you don't have 1000 entries.
- hdhzy 9y ago
  Http2 compresses headers so it they are not changing the amount of overhead is negligible (see HPACK).
  - Nick-Craver 9y ago
    Well, yes and no - it depends on the length. Let's take 3 common examples. Here's GitHub's relevant headers (that we don't have): Content-Security-Policy:default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src render.githubusercontent.com; connect-src 'self' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-user-asset-79cafe.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; img-src 'self' data: assets-cdn.github.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' assets-cdn.github.com Public-Key-Pins:max-age=5184000; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="RRM1dGqnDFsCJXBTHky16vi1obOlCgFFn/yOhI/y+ho="; pin-sha256="k2v657xBsOVe1PQRwOsHsw3bsGT2VzIqz5K+59sNQws="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="IQBnNBEiFuhj+8x6X8XLgh01V9Ic5/V3IRQLNFFc7v4="; pin-sha256="iie1VXtL7HzAMF+/PVPR9xzT80kQxdZeJ+zduCB3uj0="; pin-sha256="LvRiGEjRqfzurezaWuj8Wie2gyHMrW5Q06LspMnox7A="; includeSubDomains Those are 1220 bytes. I'm not sure what they'll compress down to, but it's still non-trivial and not near 0 (anyone want to run the numbers?). The same pair of headers are 969 bytes for facebook.com and 2,772 for gmail.com. I don't know what ours would be - since we're open-ended on the image domain side it's a bit apples-to-oranges compared to the big players. When you take into account that you can only send 10 packets down the first response (in almost all cases today) due to TCP congestion window specifications (google: CWND), they get more expensive as a percentage of what you can send. It may be that you can't send enough of the page to render, or the browser isn't getting to a critical stylesheet link until the second wave of packets after the ACK. This can greatly affect load times. Does HPACK affect this? Yeah absolutely, but I disagree on "negligible". It depends, and if something critical gets pushed to that 11th packet as a result, you can drastically increase actual page render time for users. If it helps, I did a blog post with some details about this a while back: https://nickcraver.com/blog/2015/03/24/optimization-considerations/ https://nickcraver.com/blog/2015/03/24/optimization-consider...

janwillemb 9y ago

Wow, I didn't expect this ("switching" to HTTPS) to be so hard.

Ajedi32 9y ago
It very much depends on the complexity and scale of your site. StackOverflow is a bit of an extreme case. For example, if instead of having hundreds of domains serving millions of users with tons of user-generated content you're just serving static content from a single server on a small site, the entire process for you might actually be as simple as just running `certbot-auto` on the production server. I suspect the difficulty of switching for most sites will fall somewhere between these two extremes.
irrational 9y ago
Yeah, we've been working on this for about a year (not continually, but as we have time to try to work through the problems). We do use subdomains though, so that is part of the problem. We keep feeling like we are getting close, but then we run into another issue. It's like a rabbit hole that has no bottom.
- skyisblue 9y ago
  > We keep feeling like we are getting close, but then we run into another issue. It's like a rabbit hole that has no bottom. That's exactly what we experienced migrating a bunch of sites to https. There were so many things that we didn't anticipate.

user5994461 9y ago

Funny how the main reason for lack of SSL is said to be the lack of support from 3rd party services... and the first service quoted is ads. https://nickcraver.com/blog/2013/04/23/stackoverflow-com-the-road-to-ssl/ https://nickcraver.com/blog/2013/04/23/stackoverflow-com-the...

therealdrag0 9y ago
Funny how?

quintin 9y ago

Has anyone tried running Fastly behind Cloudflare? Are the tradeoffs worth it?

tyingq 9y ago
Is there some reason other than cost to do that? Curious.
- quintin 9y ago
  Mitigate attacks much better than Fastly, flatten CNAME etc.
  - bpicolo 9y ago
    > Mitigate attacks much better than Fastly if that's the concern, probably just better to configure switching than put both in front all the time.
pbarnes_1 9y ago
Why would you want to double your CDN costs for negligible benefit?

tomschlick 9y ago

Side question: any plans for IPv6?

astrodust 9y ago
Given the scale of Stack Overflow, you'd think they could set up AAAA records that point to a proper TLS 1.3+ server and leave the peasants on IPv4 going to one that's more...accommodating.
- Nick-Craver 9y ago
  We could - but the network side isn't the problem. There's a lot of logging, user banning, etc. pieces that need IPv6 love first. We just haven't had the time yet. There are network bits we'd have to evaluate heavily as well, e.g. firewall rules - basically the very limited benefits don't make it a priority, yet. When things change there, we'll do it.
alienth 9y ago
Not any immediate plans. Decent amount of development is necessary there. There are so many places in our various systems that work with IP addresses, and many of them don't support v6 addresses.
- ec109685 9y ago
  One thing that places do is support IPv6 on the outside and translate to IPv4 calls in the next hop beyond the servers that sit beyond the edge.
tepmoc 9y ago
Here recent detailed answer from Nick https://meta.stackoverflow.com/questions/348223/stack-overflow-not-reachable-via-ipv6/348224#348224 https://meta.stackoverflow.com/questions/348223/stack-overfl...

jontro 9y ago

Regarding the section "Mistakes: APIs and .internal" Why wouldn't they use split horizon DNS for this? Seems like the perfect use case

Nick-Craver 9y ago
Split horizon would point you at the same data center, rather than the writeable one. So that's more of a .local than a .internal. We discussed this, but ultimately the AD version we're on (pre-2016 Geo-DNS) it's not actually supported the way you'd need, and it's a nightmare to debug. We'd consider it for a .local, when the support it properly there in 2016. Even subnet prioritization is busted internally, so that's a bit of an issue. Evidently no one tried to use a wildcard with dual records on 2 subnets before (we prioritize the /16, which is a data center) and it's totally busted. Microsoft has simply said this isn't supported and won't be fixed. A records work, unless they're a wildcard. So specifically, the <star>.stackexchange.com record which we mirror internally at <star>.stackexchange.com.internal for that IP set is particularly problematic. TL;DR: Microsoft AD DNS is busted and they have no intention of fixing it. It's not worth it to try and work around it.
- jontro 9y ago
  Interesting, thanks!

fareesh 9y ago

Despite the "Google gives a boost to https" reasoning, which comes from Google itself, in practice I've read several first-hand accounts of how traffic (non XP) dropped significantly right after the switch.

gbrayut 9y ago

If you like working on these kinds of projects, the SRE team at Stack Overflow is hiring and we allow remote work full time! https://stackoverflow.com/jobs/143725/site-reliability-engineer-generalist-stack-overflow https://stackoverflow.com/jobs/143725/site-reliability-engin...

dzdt 9y ago

Stack Exchange is no longer available from my workplace due to this change. We have a strict no-posting-code-fragments policy, and SE was viewed as too risky to allow without some restriction in place to make it read only. Before HTTPS, the IT department had worked out such a read-only restriction by blocking the SE login with firewall rules. But with HTTPS that kludge is no longer possible, so the site is blocked.

maddyboo 9y ago
Wow, that sounds ridiculous. What's the reasoning behind that policy?
- zcdziura 9y ago
  Seems obvious that someone high up on the corporate ladder, with no practical knowledge in how the nitty-gritty work gets done, made the decision. Probably to "minimize IP theft".
  - astrodust 9y ago
    "What do you mean our competitor is using 'for' loops? We invented those!"
Shoop 9y ago
What sort of company do you work at? Why can't everyone just be told not to post code?
- dogruck 9y ago
  What sort of company do you work at where every employee obeys every directive?
  - sqren 9y ago
    A company that trust their employees. There are so many ways to get around this anyway so it doesn't make sense to try to enforce it in the first place (considering the issues that follows).

bullen 9y ago

I said it before and I'll say it again: HTTPS is a waste of electricity.

merb 9y ago

sadly that haproxy (which stack overflow uses) does not support http/2 directly, you need to terminate it via nginx or anything else.

jcadam 9y ago

I work at a government facility. Stack Overflow and github are now both blocked (in addition to all social media and webmail). But Hacker News is apparently ok.

souenzzo 9y ago

How many questions on stackoverflow to these migration?

gub09 9y ago

It would be better if scripts like jquery were not encrypted. This forces users to use e.g. a google service instead of caching/hosting the scripts themselves or getting them from another CDN. I do not understand why so many people do not consider the privacy implications of every single webpage requiring calls to google services. There are ways to avoid this, but it gets a lot more complicated when that requires MITM methods for SSL. Please: use a non-tracking CDN, host it yourself, or at least leave it HTTP.