8 ms·

How I learned about corporate firewalls

139 points by ValCanBuild 4y ago

tanepiper 4y ago
If I want to push to GitHub when I am in the office, I have to VPN out of the office connection because Port 22 is blocked. And they wonder why I prefer to work from home?
- ValCanBuild 4y ago
  Oh god this is horrible! Yeah, I can't believe how stupidly locked down some of these networks are. I once had an employer said they needed a "whitelist" of websites we wanted to visit instead of a "blacklist" of ones we shouldn't. That was an interesting day...
  - davewasthere 4y ago
    I had exactly this. We run a Saas and someone wrote an email saying that our server was down, and when we'd expect it to be up. Not having had a notification, I double checked from a couple of geographic locations that our application was indeed up and responding. After a bit of investigation, it turns out that they have to whitelist every unique address with their corporate IT. And had only whitelisted our primary client-app URL (talks to a couple of different API endpoints), hence the strange error message. It's been a long time since I've worked somewhere with whitelisting.
    - chasil 4y ago
      I actually deployed several "FireWall ToolKit" firewalls in the '90s for various organizations. I still maintain one on an internal QA network. It allows whitelists. https://www.fwtk.org/ https://www.fwtk.org/ https://avolio.com/fwtk-history/ https://avolio.com/fwtk-history/ https://www.ranum.com/security/computer_security/editorials/codetools/ https://www.ranum.com/security/computer_security/editorials/...
    - richbell 4y ago
      > We run a Saas and someone wrote an email saying that our server was down, and when we'd expect it to be up. Not having had a notification, I double checked from a couple of geographic locations that our application was indeed up and responding. A tale as old as time: https://www.youtube.com/watch?v=uRGljemfwUE https://www.youtube.com/watch?v=uRGljemfwUE
  - shireboy 4y ago
    I’m dealing with this now. Company got hacked and so now are over the top locking down everything to the point it’s unusable. I told them the other day that the most secure thing they could do is just turn it all off.
- Kim_Bruning 4y ago
  My solution was to run an sshd on port 443. I currently no longer need to do so right this minute, but sometimes people do keep asking me why I still have that. --- Not sure if this still works on modern corporate networks. These days I tether to a mobile phone with unlimited internet; which is all-around easier to work with.
  - iso1631 4y ago
    My wireguard UDP endpoints are available on a high port, 443 and 53. I've often have one of them blocked, very rare to have them all blocked.
  - hackernudes 4y ago
    That worked for me for awhile but then the proxy started checking that all traffic was HTTP. Eventually I used ssh over websocket.
  - aaronax 4y ago
    As an example, best practice for Palo Alto firewall setup is to create a rule that allows the "application" known as "SSL" and then use "application-default" as the setting for which ports to allow it on. This would inspect the traffic to determine that it is SSL (actually TLS in most cases I guess) and then allow it if on port 443. If you don't have other relevant allow rules, your sshd traffic would just be dropped, regardless of port. If the firewall administrator does things poorly, they will create an allow rule for port 443 and your sshd traffic on port 443 would be allowed (no inspection of traffic to determine if it is SSL or SSH). BTW this is inspection, not decryption. Two very different things. The business of developing algorithms to effectively detect various applications must be very interesting. You can see all the different "applications" here: https://applipedia.paloaltonetworks.com/ https://applipedia.paloaltonetworks.com/
- gsu2 4y ago
  For anybody running into a similar problem: most git hosting services set up a subdomain that will allow SSH traffic over port 443, e.g. ssh.github.com, altssh.bitbucket.org, altssh.gitlab.com, etc.
- xani_ 4y ago
  You can just push using port 443 -> ᛯ ssh -T -p 443 git@ssh.github.com Warning: Permanently added '[ssh.github.com]:443' (ED25519) to the list of known hosts. Hi XANi! You've successfully authenticated, but GitHub does not provide shell access.
- _fat_santa 4y ago
  I worked for a place where they did wired stuff like this. Ended up that to install dependencies for a Node app, you had to: 1. Disconnect from VPN and run `npm install` until it failed 2. Connect to VPN "Profile 1" and run the command again until it fails 3. Connect to VPN "Profile 2" and run the command again until it fails. 4. Disconnect from VPN and run the command another time to finish installing all dependencies. 5. Reconnect to VPN to actually run the app.
hyperman1 4y ago
We once had to fight for stackoverflow access. Security responds: you devs should only require the manual provided by the vendor(in this case: Oracle javadocs)?
- BizarroLand 4y ago
  I would have taken that as my queue to start finding another job. Not that I can't puzzle everything out from scratch every single time I need to do anything, but why should I reinvent the wheel when off-the-shelf is both faster and higher quality?
  - thedougd 4y ago
    These types of policies and mismanagement drives out the best talent and leaves the organization filled with coasters who love any excuse to not do their job.
  - nightpool 4y ago
    (cue, as in "a signal (such as a word, phrase, or bit of stage business) to a performer to begin a specific speech or action", e.g. "That last line is your cue to exit the stage". See https://www.merriam-webster.com/dictionary/cue https://www.merriam-webster.com/dictionary/cue)
    - BizarroLand 4y ago
      Rats. I should have known it was the wrong version when my brain made me slow down to spell it out. Thanks for the correction!
      - nightpool 4y ago
        haha, no worries. glad it helped!
- deathanatos 4y ago
  A previous employer of my blocked the XTerm escape sequence reference. Like, okay, may your terminal output be plain and boring, I guess. Also, the link to "request an exception" lead to a 404 and the IT team responsible for the blocking didn't respond to email.
szszrk 4y ago
I'm currently in my very first job where running a local silent NTLM proxy is not a vital skill of survival. For similar reasons I somehow always made an opensource sonatype nexus that is doing pypy proxy or similar, so that operators team can actually do meaningful work without triggering security teams daily.
- criddell 4y ago
  Can you expand on this a bit? I googled "ntlm proxy" and "sonatype nexus" and still have no idea what it is you've done but I'd like to understand.
  - richbell 4y ago
    Nexus Repository (commonly referred to as just "Nexus", which is confusing because Sonatype has several products called "Nexus $name") is a local artifact repository. Running it locally allows you to cache artifacts from external repositories like pypi.org or repo.maven.org, which is beneficial because it cuts down on the amount of outbound traffic required to install dependencies.
    - folmar 4y ago
      The biggest win is not the bandwidth, it's that you have exactly what was used before in case you need it (audit/postmortem), or origin goes away.
      - richbell 4y ago
        In the context of GP's comment bandwidth + proxies seems to have been their motivation. Not to mention that external package registries HATE how many large organizations hammer them with the exact same traffic every time a build is running, due to the lack of the local cache. But you are correct as well; that is an uncommon yet hugely beneficial reason to have an internal artifact repository.
    - criddell 4y ago
      Okay, that makes sense and I can certainly see why that often makes sense. It seems that running the proxy leaves the original problem uncorrected. I'd be inclined to exercise a bit of malicious compliance to increase pressure for changes to the security configuration.
sh4un 4y ago
Neil44 4y ago
Yep not to mention pushing their SSL root CA to all the clients so they can scan everything without SSL errors.
- raxxorraxor 4y ago
  Sensible for the users that really download and execute attachments from the most obvious spam mails. The only protection you have is to put these high threat users in a separate subnet and use some antivirus to scan everything they download. At least that offers some protection. Not possible to scan downloads with TLS. Although I still think that breaking it up is a very bad idea in general and it is appalling that this became common practice. Especially because there are exceptions where it fails and you train users to just disregard TLS errors. Even worse, the IT security industry shamelessly uses the data to spy on employees. For that alone it deserves its bad reputation. Still, there is no real solution to shield data from the the most careless users.
  - denton-scratch 4y ago
    > put these high threat users in a separate subnet Ideally a subnet belonging to one of your competitors? I thought that nowadays only very ignorant people follow links or open attachments in spam emails. Certainly all the spam I've seen for a few years has been as plain as the nose on your face: only an ignorant person would mistake it for ham.
    - dhosek 4y ago
      I did almost get caught in a scam—email appeared to come from CEO in my medium-sized company (so it wouldn’t have been out of place to hear from him). First email simply said, do you have a moment to chat, second was, fortunately, an obvious scam request—“can you buy some gift cards for a client?” but everything was disguised enough that I might have gotten caught with a better-conceived spear phishing attack.
    - raxxorraxor 4y ago
      Companies get pretty sophisticated spam. You only need one compromised supplier and they have your names and usual mail format and just sneak edit some links to lead to compromised sites. But yes, some users also fall for the pretty obvious crap.
cheschire 4y ago
"Aha, so an overzealous IT network decided to block the request before it even reached my server." What classifies this as an "overzealous" act of network configuration? There may be a subjectively legitimate reason the user's network was configured this way. "I had no idea I was ever going to get anything different." There's an entire list of HTTP status codes. That was your clue that you would get something different. You made a decision to not have handling for them all. Not implementing handling for 418 is understandable, but forbidden and service unavailable responses are common enough.
- jjoonathan 4y ago
  There's a subjectively legitimate reason to consider blocking POST (but not GET) requests ruder things than "overzealous."
  - adev_ 4y ago
    > There's a subjectively legitimate reason to consider blocking POST (but not GET) No, just no. In a world where many website use GraphQL (POST request with content) (or gRPC) that's complete garbage decision. - This kind of brain-dead admin decision is exactly what bring protocol abuse: people would just use GET query with a ton of parameters and violate semantic just to avoid stupid middle box problems. Same goes with TLS which is used everywhere (even in VPN) just to bypass the crappiness of corporate firewall and stupid managerial decisions.
    - jwolfe 4y ago
      The rest of the sentence that you left off in your quote is saying that blocking POST requests is worse than overzealous. You are in agreement with them.
    - deleted 4y ago
      [deleted]
  - concordDance 4y ago
    What kind of reason? You can have plenty of communication via GET requests.
gabesullice 4y ago
> It helpfully spits out this HTML response in return but, of course, my frontend code was expecting a JSON response. I had no idea I was ever going to get anything different. I wish more front-end devs recognized that they're building HTTP clients whenever they make HTTP requests. There's a whole specification written about how to do that well so one doesn't have to learn things like this the hard way. Specs may look old and esoteric, but following them bakes hard-earned wisdom into your apps for free.
- PcChip 4y ago
  Isn't there some kind of library to handle this without having to code it each time?
  - gabesullice 4y ago
    Yes and no. A library might check for the status codes, but then what? Will it throw an exception? You'll have to catch them. Will it fail silently? If so, is your code prepared for null to be returned instead of parsed JSON objects. You'll have to check for that. Using a library isn't going to save you from handling all the error states and unexpected response bodies. It'll just change the documentation you're reading and the name of the abstractions you're dealing with (e.g. status codes -> exceptions)
  - raxxorraxor 4y ago
    There is a HTTP status code in any response and a 403 like in this case should already inform you about the problem. Pretty high level already. I prefer to let the users actually see such errors, although that seems to be an anti-pattern today. Usually any message receiver should first check the status code and only proceed if it is 2xx and handle errors in any other case. But such edge case errors (a 403 usually isn't an edge case) getting swallowed still happens on the most prominent on thoroughly tested sites. Had similar stuff on Amazon and Microsoft pages for example. Saw the errors in the console but they weren't displayed to the user.
- ValCanBuild 4y ago
  I've found often in my career that, sadly, learning things the hard way is usually the best way to remember the lessons.
gwillz 4y ago
I have a government client that has locked down all outgoing access for a web server except though a socks proxy. It makes simple things really hard - like a links checker, package dependencies, remote servers or integrations with Google. We can't even run test scenarios on the machine because we're also locked _out_ of the server. Instead, we rely on their IT department to run test scripts that we send them via email. We were debugging an elastic server connection for 2 weeks that was working perfectly fine in their "QA". It's a horrible existence.
- roflyear 4y ago
  Hopefully you can bill them for all of this, but yeah, totally ridiculous and costing the taxpayer a ton of money. Reminds me of a friend who started a government job, and they went 6m before they were fully onboarded and able to work. ????
- dijit 4y ago
  Used to work at Ubisoft and they had this same policy, they used an authenticated http proxy, so you either expose your entire SSO credentials to your environment (HTTP_PROXY=http://user:password@proxy:3128 http://user:password@proxy:3128) or you don't get access to the internet for all your console applications. Even then, if you were using certificate pinning, it wouldn't work as the HTTP proxy would serve a "are you sure you want to continue" HTML page, which is of course not expected. SSH is out of the question. it's amazing what "simple" things break; like kubectl, gcloud, go get. So frustrating. Countless development hours lost to bypasses.
- bheadmaster 4y ago
  > I have a government client that has locked down all outgoing access for a web server except though a socks proxy. If you're running Linux, there's a utility called "tsocks" which wraps any other command and redirects all network servers through a SOCKS proxy defined in /etc/tsocks.conf, e.g.: tsocks pip install somepackage One downside is that since it relies on some linker magic, it doesn't work for static binaries. But for most common usage, it served me just fine.
  - 4y ago
mxuribe 4y ago
Ah, yes, the old "the most secure device is one that is not working", or in this case that is blocked from working. :-)
- dhosek 4y ago
  Think inside the box!
jve 4y ago
I wonder what _showHtmlPage_ does? Did he just write something, something that allows 3rd party (corporate firewalls) inject HTML under his domain within TLS protected connection? Cannot judge by not knowing how he displays errors. But a question to HN public: Is opening unknown HTML under my domain within another window safe? Or is there any possibility to strip down any "permissions" to cookies, requests, resources etc for that dedicated page?
- ValCanBuild 4y ago
  OP here - I'm open to advice about how best to handle this! I'm currently just opening a new window and writing the HTML to it. What's the safest way to handle this? Open it in an iframe?
  - jve 4y ago
    Just look under this thread, I wrote one possible solution of using .innerText from constructed DOM. (Or maybe open window in another domain). However @repiret may be right - corporate proxy is already invasive enough, that means the users are already in mercy of those. But still, I'd go with safer practices. Even in the slightly unlikely case someone manages to hack 3rd party (Stripe) and send your users arbitrary HTML for some periods of time... :)
    - ValCanBuild 4y ago
      Thanks, appreciate the advice!
- shireboy 4y ago
  Came here to point this out. For non-trivial implementations of showHtmlPage, this is a vulnerability. A malicious user could set up a 403 response with a fake “please re-enter your card to verify” form that sends to the attacker, or possibly even script to scrape the card number. Probably low risk of this actually happening in this scenario, but I’m pretty sure this fix is a bad idea. Better to show a generic error and log.
pilif 4y ago
My favourite issue caused by a corporate firewall was when it altered an AJAX request to replace a specific combination of digits (in a long product ID) by asterisks. Turns out that a substring of that product ID matched the client company's phone number and their security theatre intercepting proxy was replacing all occurrences of "sensitive" strings sent to the internet with asterisks. The irony is, of course, that as the people running the site, I didn't know (and would never have wanted to know) the user's phone number until this incident. How I loathe security theatre.
- deleted 4y ago
  [deleted]
- HPsquared 4y ago
  Did you let them know? They of course need to patch this vulnerability by blocking anything containing 11 consecutive digits.
  - pstuart 4y ago
    That's crazy! The right thing would be to add a lookup function to first verify the phone number is in use and then call the number to ask for permission to use it; followed by a webhook to send a confirmation back to the database to cache that info because this needs to be efficient! /s
    - alexvoda 4y ago
      How do you propose to tackle the cache invalidation challange? /s
      - deleted 4y ago
        [deleted]
- deepsun 4y ago
  Now the other site knows the phone number (they know what was sent, and they see what was replaced by asterisks). And now they can exfiltrate all the sensitive phone numbers -- just sending clients (you) long strings of numbers, and see what was replaced.
collinvandyck76 4y ago
I've seen so many engineer hours burned because of a reluctance to gate behavior on http response codes.
coldcode 4y ago
I worked in a financial company in mid 2000s where the network head did not believe in internal firewalls so that all internal users were on the same network as all the web app servers and database servers. If someone was downloading a movie then customer web access slowed; since everyone used Windows everyone was required to run virus scanners on their computers and that included the app and database server machines since they were not isolated from the rest of the network. If a vendor came to demo something they were unable to since there was no way to isolate their laptop from everyone else so they could not access the internet. Good thing I never put any of my money in the company accounts...
- deepsun 4y ago
  You won't believe it, but the "one network" came back nowadays. It's called "zero trust", basically treating your internal network as public.
  - slt2021 4y ago
    that's right, except for all traffic is TLS encrypted, all authN/Z is at least two-factor, all services are least privilege/white-list, even intercepting traffic/session keys/or even user/pass credentials wont give you anything important
  - PLG88 4y ago
    Zero Trust pricniples may implie having a flat underlay but explicitly access to applications and services should be microsegmented, least privilege, and authenticate/authorised on strong identity before any connectivity can be established - i.e., the overlay is closed by default and does not trust the underlay. Ideally you put ZT inside an application so you do not need to have any inbound ports, public DNS, etc etc.
- icedchai 4y ago
  This sounds like most of the early ISPs I worked at. No firewalls, and switches weren't popular yet, so we had hubs. The "backbone" of the ISP network was the same as the main office network. Any employee could just tcpdump all the traffic. Actually, we had a couple of customer-owned servers that were colocated, that could also dump all the traffic. Eventually someone set up a firewall (Linux box with dual ethernets) to segment the colo traffic.
mikessoft_gmail 4y ago
huy-nguyen 4y ago
I work at a government agency and here are my tales. 1) They install a root certificate on all machines and use that to MITM all TLS connections using a firewall appliance. They turn this MITM on one day without notifying any developer. Overnight, all our builds (run on-prem) fail because npm install, pip install etc fail and we spent a long time trying to figure it out. They are still failing to this day and I have to get off the VPN every time I need to run these simple commands. IT absolutely doesn't give a flying **** about developers. 2) They ban all non-Chrome browsers from being installed. As in, if you install such a browser and try to launch it, the system will say "browser X is banned. Contact IT." They would have banned Safari too had it not been part of the OS. Furthermore, they also disabled private browsing in Chrome (probably the ability to do this is why they allow Chrome). I think they're preventing people from hiding their internet browsing.
- bombcar 4y ago
  That first one indicates something is being injected and the checksums are failing, that's ... worrying. Or npm and pip use their own certificate stacks and refuse the firewall's cert, which is ... good I guess.
  - thayne 4y ago
    > Or npm and pip use their own certificate stacks and refuse the firewall's cert, which is ... good I guess. Combined with the fact that chrome is the only allowed browser, I suspect it is the other way around. Chrome uses its own certificate stack, and I would guess IT only added the MITM certificate to the chrome trusted CA list, not the system one.
  - marcosdumay 4y ago
    It could be just the problem of the certificate being invalid for those tools because the MITM one was installed only for Chrome up to the firewall replacing all of the files with html pages of some antivirus with internal links where the user can download them. Corporate middleboxes come in all shades of stupid.
  - Karunamon 4y ago
hathym 4y ago
tldr: do error handling.
- justin_oaks 4y ago
  Yes, it's more about error handling than it is about corporate firewalls. There are lots of reasons why the request would fail and returning a 403 or 503 from a corporate firewall is just one of them. What happens if the user's wifi is flaky and the HTTP request is canceled? What happens if the connection is slow and the request times out? What if, heaven forbid, the destination server is down or unreachable temporarily? As a web developer, never let a user's action lead to nothing happening. Always give feedback. Whenever sending background HTTP requests, always provide a visible error message to the user when you encounter unexpected results or HTTP/network errors.
thayne 4y ago
I was expecting another section about all the other ways a corporate firewall can cause issues. Not all firewalls will give you a 403. Sometimes it will be a 200 with the error in the body. Sometimes you'll just never get a response at all. Sometimes you will get an SSL certificate error, because the error response is signed using the certificate for the firewall vendor's domain instead of yours. And etc.
- ValCanBuild 4y ago
  Oh god - I hope I don't have to write a follow-up to this. There's probably a bunch of hidden other firewalls I don't handle but most often I've found 403 and 503 to be the most common.
flumpcakes 4y ago
Firewalls from security vendors with L7 decryption (using MITM root certificates from a company-wide PKI) is pretty standard in any business that needs to care about "cyber security". I always hear people cry and moan about this but having worked on that side of the fence I would like you to know that I know of instances where people have been downloading illegal material (involving children) and running tor. That's not to mention the 75% of staff who willing give details during phishing campaigns. Saying that, I find 60%+ of cyber businesses to be a waste of time at best, and at worse just frauds. Core firewalls with L7 capabilities from vendors such as Palo Alto and CheckPoint are legitimate security devices, especially suited for enterprise networks. I do think it's pretty pointless running those in the cloud though, unless you have admin VMs on vnets for your production resources. But that way lies madness anyway.
- jsmith45 4y ago
  The problem with those is that they are often poorly configured. Take for example the scenario in question here. Is it really legitimate to allow GET requests to a domain but block all POST requests? That sounds questionable at best. How many sites is it safe to view pages, download files, etc from, but POSTing to them is dangerous? There may be a few, but it is not particularly common. Far more common is sites where any request could be harmful. (Malware, sites spoofing other sites, etc). I get fully blocking a domain. That can be reasonable sensible, especially for domains in a known blocklist of porn, malware, etc. I can get inspecting content and blocking if there is clear evidence of maliciousness (but this must be done carefully, since false positives can cause a lot of headache!), but for other content-matching scenarios, you may well be better off generating an alert to be reviewed manually, rather than blocking things. There have been cases where these system incorrectly block business critical functionality, causing a company to completely shut down, losing huge sums of money while figuring out what is breaking things, before getting it sorted.
- sillystuff 4y ago
  The correct solution to phishing is to stop users from receiving phishing email in the first place. Yes, blocking phishing mails can be impossible with some hosted providers' spam filtering. But, here the solution should be to push back on e.g., Microsoft to fix their dumpster fire spam filtering, or switch the organization to a different product that works. I don't think IT should be pretending at being police. It isn't their job. And, any infrastructure that can be used to catch "criminals" can be used to abuse employees. Also, there is absolutely nothing wrong with using tor. I've used it often, at work, to test things as if from off-site. I believe the role of IT is to respectfully facilitate users to safely get their work done. This involves a balance of security measures that do not invade the users' privacy, pushing back against management when appropriate to protect the users from managerial overreach, and sometimes just allowing something that could be dangerous because the alternative is worse e.g., MiM provides limited protection from exfiltration, but also enables horrible abuse by management and should be pushed back against.
bornfreddy 4y ago
While I am sympathetic with the developer, a large part of fault lies with them. Firewall actually behaved very nicely. Always check status codes. Don't assume that backend (even if it is your own server) behaves as you think it should - complain when the response is not what you would have expected. This is why I hate those error responses that encode the error message into JSON and return status 200. Gee, thanks - your backend is so special that it is an honor to write custom error handling for it. /s Glad OP solved it in the end, but I would suggest reacting to all 4xx and 5xx statuses. It's a standard, if you get 418 you know what "your" backend is saying.
ultrahax 4y ago
I work on something that requires a reasonably cooperative NAT and unmolested real-time UDP traffic. I've seen varied failure-modes from corporate firewalls over the years - from simple NAT table overflow causing rapid source port switching, to the firewall appliance downloading an update and deciding UDP packets of a certain size ( and ONLY of a certain size.. ) were bittorrent and hence were to be blackholed. That was an interesting one to track down. I've also seen it block diagnostic GETs to varied bits of cloud infra, due to someone at some point in the distant past hosting porn on that particular IP. Not to mention just good old strict NATs..
Aulig 4y ago
You really need something like sentry to report all unhandled exceptions. I learned that lesson when I realized everyone on Safari couldn't use my website (submit forms) because a feature I used wasn't supported on Safari. Easy way to drop your conversion rate by 30%.
deleted 4y ago
[deleted]
Fredej 4y ago
> Sorry, you don't have permission to visit this site. > Website blocked > Not allowed to browse Shareware Download category > You tried to visit:https://www.valcanbuild.tech/handling-corporate-firewalls/ https://www.valcanbuild.tech/handling-corporate-firewalls/ The irony.
- ValCanBuild 4y ago
  Hah, of course!

139 points by ValCanBuild 4y ago

If I want to push to GitHub when I am in the office, I have to VPN out of the office connection because Port 22 is blocked. And they wonder why I prefer to work from home?

ValCanBuild 4y ago
Oh god this is horrible! Yeah, I can't believe how stupidly locked down some of these networks are. I once had an employer said they needed a "whitelist" of websites we wanted to visit instead of a "blacklist" of ones we shouldn't. That was an interesting day...
- davewasthere 4y ago
  I had exactly this. We run a Saas and someone wrote an email saying that our server was down, and when we'd expect it to be up. Not having had a notification, I double checked from a couple of geographic locations that our application was indeed up and responding. After a bit of investigation, it turns out that they have to whitelist every unique address with their corporate IT. And had only whitelisted our primary client-app URL (talks to a couple of different API endpoints), hence the strange error message. It's been a long time since I've worked somewhere with whitelisting.
  - chasil 4y ago
    I actually deployed several "FireWall ToolKit" firewalls in the '90s for various organizations. I still maintain one on an internal QA network. It allows whitelists. https://www.fwtk.org/ https://www.fwtk.org/ https://avolio.com/fwtk-history/ https://avolio.com/fwtk-history/ https://www.ranum.com/security/computer_security/editorials/codetools/ https://www.ranum.com/security/computer_security/editorials/...
  - richbell 4y ago
    > We run a Saas and someone wrote an email saying that our server was down, and when we'd expect it to be up. Not having had a notification, I double checked from a couple of geographic locations that our application was indeed up and responding. A tale as old as time: https://www.youtube.com/watch?v=uRGljemfwUE https://www.youtube.com/watch?v=uRGljemfwUE
- shireboy 4y ago
  I’m dealing with this now. Company got hacked and so now are over the top locking down everything to the point it’s unusable. I told them the other day that the most secure thing they could do is just turn it all off.
Kim_Bruning 4y ago
My solution was to run an sshd on port 443. I currently no longer need to do so right this minute, but sometimes people do keep asking me why I still have that. --- Not sure if this still works on modern corporate networks. These days I tether to a mobile phone with unlimited internet; which is all-around easier to work with.
- iso1631 4y ago
  My wireguard UDP endpoints are available on a high port, 443 and 53. I've often have one of them blocked, very rare to have them all blocked.
- hackernudes 4y ago
  That worked for me for awhile but then the proxy started checking that all traffic was HTTP. Eventually I used ssh over websocket.
- aaronax 4y ago
  As an example, best practice for Palo Alto firewall setup is to create a rule that allows the "application" known as "SSL" and then use "application-default" as the setting for which ports to allow it on. This would inspect the traffic to determine that it is SSL (actually TLS in most cases I guess) and then allow it if on port 443. If you don't have other relevant allow rules, your sshd traffic would just be dropped, regardless of port. If the firewall administrator does things poorly, they will create an allow rule for port 443 and your sshd traffic on port 443 would be allowed (no inspection of traffic to determine if it is SSL or SSH). BTW this is inspection, not decryption. Two very different things. The business of developing algorithms to effectively detect various applications must be very interesting. You can see all the different "applications" here: https://applipedia.paloaltonetworks.com/ https://applipedia.paloaltonetworks.com/
gsu2 4y ago
For anybody running into a similar problem: most git hosting services set up a subdomain that will allow SSH traffic over port 443, e.g. ssh.github.com, altssh.bitbucket.org, altssh.gitlab.com, etc.
xani_ 4y ago
You can just push using port 443 -> ᛯ ssh -T -p 443 git@ssh.github.com Warning: Permanently added '[ssh.github.com]:443' (ED25519) to the list of known hosts. Hi XANi! You've successfully authenticated, but GitHub does not provide shell access.
_fat_santa 4y ago
I worked for a place where they did wired stuff like this. Ended up that to install dependencies for a Node app, you had to: 1. Disconnect from VPN and run `npm install` until it failed 2. Connect to VPN "Profile 1" and run the command again until it fails 3. Connect to VPN "Profile 2" and run the command again until it fails. 4. Disconnect from VPN and run the command another time to finish installing all dependencies. 5. Reconnect to VPN to actually run the app.

hyperman1 4y ago

We once had to fight for stackoverflow access. Security responds: you devs should only require the manual provided by the vendor(in this case: Oracle javadocs)?

BizarroLand 4y ago
I would have taken that as my queue to start finding another job. Not that I can't puzzle everything out from scratch every single time I need to do anything, but why should I reinvent the wheel when off-the-shelf is both faster and higher quality?
- thedougd 4y ago
  These types of policies and mismanagement drives out the best talent and leaves the organization filled with coasters who love any excuse to not do their job.
- nightpool 4y ago
  (cue, as in "a signal (such as a word, phrase, or bit of stage business) to a performer to begin a specific speech or action", e.g. "That last line is your cue to exit the stage". See https://www.merriam-webster.com/dictionary/cue https://www.merriam-webster.com/dictionary/cue)
  - BizarroLand 4y ago
    Rats. I should have known it was the wrong version when my brain made me slow down to spell it out. Thanks for the correction!
    - nightpool 4y ago
      haha, no worries. glad it helped!
deathanatos 4y ago
A previous employer of my blocked the XTerm escape sequence reference. Like, okay, may your terminal output be plain and boring, I guess. Also, the link to "request an exception" lead to a 404 and the IT team responsible for the blocking didn't respond to email.

szszrk 4y ago

I'm currently in my very first job where running a local silent NTLM proxy is not a vital skill of survival. For similar reasons I somehow always made an opensource sonatype nexus that is doing pypy proxy or similar, so that operators team can actually do meaningful work without triggering security teams daily.

criddell 4y ago
Can you expand on this a bit? I googled "ntlm proxy" and "sonatype nexus" and still have no idea what it is you've done but I'd like to understand.
- richbell 4y ago
  Nexus Repository (commonly referred to as just "Nexus", which is confusing because Sonatype has several products called "Nexus $name") is a local artifact repository. Running it locally allows you to cache artifacts from external repositories like pypi.org or repo.maven.org, which is beneficial because it cuts down on the amount of outbound traffic required to install dependencies.
  - folmar 4y ago
    The biggest win is not the bandwidth, it's that you have exactly what was used before in case you need it (audit/postmortem), or origin goes away.
    - richbell 4y ago
      In the context of GP's comment bandwidth + proxies seems to have been their motivation. Not to mention that external package registries HATE how many large organizations hammer them with the exact same traffic every time a build is running, due to the lack of the local cache. But you are correct as well; that is an uncommon yet hugely beneficial reason to have an internal artifact repository.
  - criddell 4y ago
    Okay, that makes sense and I can certainly see why that often makes sense. It seems that running the proxy leaves the original problem uncorrected. I'd be inclined to exercise a bit of malicious compliance to increase pressure for changes to the security configuration.

sh4un 4y ago

Neil44 4y ago

Yep not to mention pushing their SSL root CA to all the clients so they can scan everything without SSL errors.

raxxorraxor 4y ago
Sensible for the users that really download and execute attachments from the most obvious spam mails. The only protection you have is to put these high threat users in a separate subnet and use some antivirus to scan everything they download. At least that offers some protection. Not possible to scan downloads with TLS. Although I still think that breaking it up is a very bad idea in general and it is appalling that this became common practice. Especially because there are exceptions where it fails and you train users to just disregard TLS errors. Even worse, the IT security industry shamelessly uses the data to spy on employees. For that alone it deserves its bad reputation. Still, there is no real solution to shield data from the the most careless users.
- denton-scratch 4y ago
  > put these high threat users in a separate subnet Ideally a subnet belonging to one of your competitors? I thought that nowadays only very ignorant people follow links or open attachments in spam emails. Certainly all the spam I've seen for a few years has been as plain as the nose on your face: only an ignorant person would mistake it for ham.
  - dhosek 4y ago
    I did almost get caught in a scam—email appeared to come from CEO in my medium-sized company (so it wouldn’t have been out of place to hear from him). First email simply said, do you have a moment to chat, second was, fortunately, an obvious scam request—“can you buy some gift cards for a client?” but everything was disguised enough that I might have gotten caught with a better-conceived spear phishing attack.
  - raxxorraxor 4y ago
    Companies get pretty sophisticated spam. You only need one compromised supplier and they have your names and usual mail format and just sneak edit some links to lead to compromised sites. But yes, some users also fall for the pretty obvious crap.

cheschire 4y ago

"Aha, so an overzealous IT network decided to block the request before it even reached my server." What classifies this as an "overzealous" act of network configuration? There may be a subjectively legitimate reason the user's network was configured this way. "I had no idea I was ever going to get anything different." There's an entire list of HTTP status codes. That was your clue that you would get something different. You made a decision to not have handling for them all. Not implementing handling for 418 is understandable, but forbidden and service unavailable responses are common enough.

jjoonathan 4y ago
There's a subjectively legitimate reason to consider blocking POST (but not GET) requests ruder things than "overzealous."
- adev_ 4y ago
  > There's a subjectively legitimate reason to consider blocking POST (but not GET) No, just no. In a world where many website use GraphQL (POST request with content) (or gRPC) that's complete garbage decision. - This kind of brain-dead admin decision is exactly what bring protocol abuse: people would just use GET query with a ton of parameters and violate semantic just to avoid stupid middle box problems. Same goes with TLS which is used everywhere (even in VPN) just to bypass the crappiness of corporate firewall and stupid managerial decisions.
  - jwolfe 4y ago
    The rest of the sentence that you left off in your quote is saying that blocking POST requests is worse than overzealous. You are in agreement with them.
  - deleted 4y ago
    [deleted]
- concordDance 4y ago
  What kind of reason? You can have plenty of communication via GET requests.

gabesullice 4y ago

> It helpfully spits out this HTML response in return but, of course, my frontend code was expecting a JSON response. I had no idea I was ever going to get anything different. I wish more front-end devs recognized that they're building HTTP clients whenever they make HTTP requests. There's a whole specification written about how to do that well so one doesn't have to learn things like this the hard way. Specs may look old and esoteric, but following them bakes hard-earned wisdom into your apps for free.

PcChip 4y ago
Isn't there some kind of library to handle this without having to code it each time?
- gabesullice 4y ago
  Yes and no. A library might check for the status codes, but then what? Will it throw an exception? You'll have to catch them. Will it fail silently? If so, is your code prepared for null to be returned instead of parsed JSON objects. You'll have to check for that. Using a library isn't going to save you from handling all the error states and unexpected response bodies. It'll just change the documentation you're reading and the name of the abstractions you're dealing with (e.g. status codes -> exceptions)
- raxxorraxor 4y ago
  There is a HTTP status code in any response and a 403 like in this case should already inform you about the problem. Pretty high level already. I prefer to let the users actually see such errors, although that seems to be an anti-pattern today. Usually any message receiver should first check the status code and only proceed if it is 2xx and handle errors in any other case. But such edge case errors (a 403 usually isn't an edge case) getting swallowed still happens on the most prominent on thoroughly tested sites. Had similar stuff on Amazon and Microsoft pages for example. Saw the errors in the console but they weren't displayed to the user.
ValCanBuild 4y ago
I've found often in my career that, sadly, learning things the hard way is usually the best way to remember the lessons.

gwillz 4y ago

I have a government client that has locked down all outgoing access for a web server except though a socks proxy. It makes simple things really hard - like a links checker, package dependencies, remote servers or integrations with Google. We can't even run test scenarios on the machine because we're also locked _out_ of the server. Instead, we rely on their IT department to run test scripts that we send them via email. We were debugging an elastic server connection for 2 weeks that was working perfectly fine in their "QA". It's a horrible existence.

roflyear 4y ago
Hopefully you can bill them for all of this, but yeah, totally ridiculous and costing the taxpayer a ton of money. Reminds me of a friend who started a government job, and they went 6m before they were fully onboarded and able to work. ????
dijit 4y ago
Used to work at Ubisoft and they had this same policy, they used an authenticated http proxy, so you either expose your entire SSO credentials to your environment (HTTP_PROXY=http://user:password@proxy:3128 http://user:password@proxy:3128) or you don't get access to the internet for all your console applications. Even then, if you were using certificate pinning, it wouldn't work as the HTTP proxy would serve a "are you sure you want to continue" HTML page, which is of course not expected. SSH is out of the question. it's amazing what "simple" things break; like kubectl, gcloud, go get. So frustrating. Countless development hours lost to bypasses.
bheadmaster 4y ago
> I have a government client that has locked down all outgoing access for a web server except though a socks proxy. If you're running Linux, there's a utility called "tsocks" which wraps any other command and redirects all network servers through a SOCKS proxy defined in /etc/tsocks.conf, e.g.: tsocks pip install somepackage One downside is that since it relies on some linker magic, it doesn't work for static binaries. But for most common usage, it served me just fine.
- 4y ago

mxuribe 4y ago

Ah, yes, the old "the most secure device is one that is not working", or in this case that is blocked from working. :-)

dhosek 4y ago
Think inside the box!

jve 4y ago

I wonder what _showHtmlPage_ does? Did he just write something, something that allows 3rd party (corporate firewalls) inject HTML under his domain within TLS protected connection? Cannot judge by not knowing how he displays errors. But a question to HN public: Is opening unknown HTML under my domain within another window safe? Or is there any possibility to strip down any "permissions" to cookies, requests, resources etc for that dedicated page?

ValCanBuild 4y ago
OP here - I'm open to advice about how best to handle this! I'm currently just opening a new window and writing the HTML to it. What's the safest way to handle this? Open it in an iframe?
- jve 4y ago
  Just look under this thread, I wrote one possible solution of using .innerText from constructed DOM. (Or maybe open window in another domain). However @repiret may be right - corporate proxy is already invasive enough, that means the users are already in mercy of those. But still, I'd go with safer practices. Even in the slightly unlikely case someone manages to hack 3rd party (Stripe) and send your users arbitrary HTML for some periods of time... :)
  - ValCanBuild 4y ago
    Thanks, appreciate the advice!
shireboy 4y ago
Came here to point this out. For non-trivial implementations of showHtmlPage, this is a vulnerability. A malicious user could set up a 403 response with a fake “please re-enter your card to verify” form that sends to the attacker, or possibly even script to scrape the card number. Probably low risk of this actually happening in this scenario, but I’m pretty sure this fix is a bad idea. Better to show a generic error and log.

pilif 4y ago

My favourite issue caused by a corporate firewall was when it altered an AJAX request to replace a specific combination of digits (in a long product ID) by asterisks. Turns out that a substring of that product ID matched the client company's phone number and their security theatre intercepting proxy was replacing all occurrences of "sensitive" strings sent to the internet with asterisks. The irony is, of course, that as the people running the site, I didn't know (and would never have wanted to know) the user's phone number until this incident. How I loathe security theatre.

deleted 4y ago
[deleted]
HPsquared 4y ago
Did you let them know? They of course need to patch this vulnerability by blocking anything containing 11 consecutive digits.
- pstuart 4y ago
  That's crazy! The right thing would be to add a lookup function to first verify the phone number is in use and then call the number to ask for permission to use it; followed by a webhook to send a confirmation back to the database to cache that info because this needs to be efficient! /s
  - alexvoda 4y ago
    How do you propose to tackle the cache invalidation challange? /s
    - deleted 4y ago
      [deleted]
deepsun 4y ago
Now the other site knows the phone number (they know what was sent, and they see what was replaced by asterisks). And now they can exfiltrate all the sensitive phone numbers -- just sending clients (you) long strings of numbers, and see what was replaced.

collinvandyck76 4y ago

I've seen so many engineer hours burned because of a reluctance to gate behavior on http response codes.

coldcode 4y ago

I worked in a financial company in mid 2000s where the network head did not believe in internal firewalls so that all internal users were on the same network as all the web app servers and database servers. If someone was downloading a movie then customer web access slowed; since everyone used Windows everyone was required to run virus scanners on their computers and that included the app and database server machines since they were not isolated from the rest of the network. If a vendor came to demo something they were unable to since there was no way to isolate their laptop from everyone else so they could not access the internet. Good thing I never put any of my money in the company accounts...

deepsun 4y ago
You won't believe it, but the "one network" came back nowadays. It's called "zero trust", basically treating your internal network as public.
- slt2021 4y ago
  that's right, except for all traffic is TLS encrypted, all authN/Z is at least two-factor, all services are least privilege/white-list, even intercepting traffic/session keys/or even user/pass credentials wont give you anything important
- PLG88 4y ago
  Zero Trust pricniples may implie having a flat underlay but explicitly access to applications and services should be microsegmented, least privilege, and authenticate/authorised on strong identity before any connectivity can be established - i.e., the overlay is closed by default and does not trust the underlay. Ideally you put ZT inside an application so you do not need to have any inbound ports, public DNS, etc etc.
icedchai 4y ago
This sounds like most of the early ISPs I worked at. No firewalls, and switches weren't popular yet, so we had hubs. The "backbone" of the ISP network was the same as the main office network. Any employee could just tcpdump all the traffic. Actually, we had a couple of customer-owned servers that were colocated, that could also dump all the traffic. Eventually someone set up a firewall (Linux box with dual ethernets) to segment the colo traffic.

mikessoft_gmail 4y ago

huy-nguyen 4y ago

I work at a government agency and here are my tales. 1) They install a root certificate on all machines and use that to MITM all TLS connections using a firewall appliance. They turn this MITM on one day without notifying any developer. Overnight, all our builds (run on-prem) fail because npm install, pip install etc fail and we spent a long time trying to figure it out. They are still failing to this day and I have to get off the VPN every time I need to run these simple commands. IT absolutely doesn't give a flying **** about developers. 2) They ban all non-Chrome browsers from being installed. As in, if you install such a browser and try to launch it, the system will say "browser X is banned. Contact IT." They would have banned Safari too had it not been part of the OS. Furthermore, they also disabled private browsing in Chrome (probably the ability to do this is why they allow Chrome). I think they're preventing people from hiding their internet browsing.

bombcar 4y ago
That first one indicates something is being injected and the checksums are failing, that's ... worrying. Or npm and pip use their own certificate stacks and refuse the firewall's cert, which is ... good I guess.
- thayne 4y ago
  > Or npm and pip use their own certificate stacks and refuse the firewall's cert, which is ... good I guess. Combined with the fact that chrome is the only allowed browser, I suspect it is the other way around. Chrome uses its own certificate stack, and I would guess IT only added the MITM certificate to the chrome trusted CA list, not the system one.
- marcosdumay 4y ago
  It could be just the problem of the certificate being invalid for those tools because the MITM one was installed only for Chrome up to the firewall replacing all of the files with html pages of some antivirus with internal links where the user can download them. Corporate middleboxes come in all shades of stupid.
- Karunamon 4y ago

hathym 4y ago

tldr: do error handling.

justin_oaks 4y ago
Yes, it's more about error handling than it is about corporate firewalls. There are lots of reasons why the request would fail and returning a 403 or 503 from a corporate firewall is just one of them. What happens if the user's wifi is flaky and the HTTP request is canceled? What happens if the connection is slow and the request times out? What if, heaven forbid, the destination server is down or unreachable temporarily? As a web developer, never let a user's action lead to nothing happening. Always give feedback. Whenever sending background HTTP requests, always provide a visible error message to the user when you encounter unexpected results or HTTP/network errors.

thayne 4y ago

I was expecting another section about all the other ways a corporate firewall can cause issues. Not all firewalls will give you a 403. Sometimes it will be a 200 with the error in the body. Sometimes you'll just never get a response at all. Sometimes you will get an SSL certificate error, because the error response is signed using the certificate for the firewall vendor's domain instead of yours. And etc.

ValCanBuild 4y ago
Oh god - I hope I don't have to write a follow-up to this. There's probably a bunch of hidden other firewalls I don't handle but most often I've found 403 and 503 to be the most common.

flumpcakes 4y ago

Firewalls from security vendors with L7 decryption (using MITM root certificates from a company-wide PKI) is pretty standard in any business that needs to care about "cyber security". I always hear people cry and moan about this but having worked on that side of the fence I would like you to know that I know of instances where people have been downloading illegal material (involving children) and running tor. That's not to mention the 75% of staff who willing give details during phishing campaigns. Saying that, I find 60%+ of cyber businesses to be a waste of time at best, and at worse just frauds. Core firewalls with L7 capabilities from vendors such as Palo Alto and CheckPoint are legitimate security devices, especially suited for enterprise networks. I do think it's pretty pointless running those in the cloud though, unless you have admin VMs on vnets for your production resources. But that way lies madness anyway.

jsmith45 4y ago
The problem with those is that they are often poorly configured. Take for example the scenario in question here. Is it really legitimate to allow GET requests to a domain but block all POST requests? That sounds questionable at best. How many sites is it safe to view pages, download files, etc from, but POSTing to them is dangerous? There may be a few, but it is not particularly common. Far more common is sites where any request could be harmful. (Malware, sites spoofing other sites, etc). I get fully blocking a domain. That can be reasonable sensible, especially for domains in a known blocklist of porn, malware, etc. I can get inspecting content and blocking if there is clear evidence of maliciousness (but this must be done carefully, since false positives can cause a lot of headache!), but for other content-matching scenarios, you may well be better off generating an alert to be reviewed manually, rather than blocking things. There have been cases where these system incorrectly block business critical functionality, causing a company to completely shut down, losing huge sums of money while figuring out what is breaking things, before getting it sorted.
sillystuff 4y ago
The correct solution to phishing is to stop users from receiving phishing email in the first place. Yes, blocking phishing mails can be impossible with some hosted providers' spam filtering. But, here the solution should be to push back on e.g., Microsoft to fix their dumpster fire spam filtering, or switch the organization to a different product that works. I don't think IT should be pretending at being police. It isn't their job. And, any infrastructure that can be used to catch "criminals" can be used to abuse employees. Also, there is absolutely nothing wrong with using tor. I've used it often, at work, to test things as if from off-site. I believe the role of IT is to respectfully facilitate users to safely get their work done. This involves a balance of security measures that do not invade the users' privacy, pushing back against management when appropriate to protect the users from managerial overreach, and sometimes just allowing something that could be dangerous because the alternative is worse e.g., MiM provides limited protection from exfiltration, but also enables horrible abuse by management and should be pushed back against.

bornfreddy 4y ago

While I am sympathetic with the developer, a large part of fault lies with them. Firewall actually behaved very nicely. Always check status codes. Don't assume that backend (even if it is your own server) behaves as you think it should - complain when the response is not what you would have expected. This is why I hate those error responses that encode the error message into JSON and return status 200. Gee, thanks - your backend is so special that it is an honor to write custom error handling for it. /s Glad OP solved it in the end, but I would suggest reacting to all 4xx and 5xx statuses. It's a standard, if you get 418 you know what "your" backend is saying.

ultrahax 4y ago

I work on something that requires a reasonably cooperative NAT and unmolested real-time UDP traffic. I've seen varied failure-modes from corporate firewalls over the years - from simple NAT table overflow causing rapid source port switching, to the firewall appliance downloading an update and deciding UDP packets of a certain size ( and ONLY of a certain size.. ) were bittorrent and hence were to be blackholed. That was an interesting one to track down. I've also seen it block diagnostic GETs to varied bits of cloud infra, due to someone at some point in the distant past hosting porn on that particular IP. Not to mention just good old strict NATs..

Aulig 4y ago

You really need something like sentry to report all unhandled exceptions. I learned that lesson when I realized everyone on Safari couldn't use my website (submit forms) because a feature I used wasn't supported on Safari. Easy way to drop your conversion rate by 30%.

deleted 4y ago

[deleted]

Fredej 4y ago

> Sorry, you don't have permission to visit this site. > Website blocked > Not allowed to browse Shareware Download category > You tried to visit:https://www.valcanbuild.tech/handling-corporate-firewalls/ https://www.valcanbuild.tech/handling-corporate-firewalls/ The irony.

ValCanBuild 4y ago
Hah, of course!