4 ms·

How my Apache server became a malicious free internet proxy

53 points by MariuszGalus 11y ago

bognition 11y ago
This is a perfect example of why most people should not run their own hardware. Don't get me wrong its really fun to build and configure your own server and I openly encourage people to learn but I also remind them that its extremely difficult (for a novice) to do securely. Additionally connecting a misconfigured server to the internet doesn't just hurt the server owner but the entire network is affected, as you are providing another piece of hardware that malicious actors can use to execute their attacks.
- burnte 11y ago
  I ran a machine in a colo for almost four years, public facing web server for a dedicated webapp. I managed to escape any attacks or infections, but I'm positive it was 80% of what I did and 20% that it's a low volume app that never got attention by malicious actors. And I've been in the industry for a long time, not a noob.
  - Canada 11y ago
    Everything is attacked. It's all automated, not personal.
    - tomschlick 11y ago
      Yup install something like fail2ban & setup email alerts. You'll get a new notice every 5 minutes or so for the first month from all the china based bots trying to login via common ssh credentials.
      - A010 11y ago
        I run snort and from its alerts, the bots come from all around the world not all from CN. Though the CN sources are largest.
        protomyth 11y ago
        I was rather surprised when I had a week where most of the attacks came from Las Vegas and New Hampshire.
    - burnte 11y ago
      Well, true, we did get spam attacks on the app, so I wound up banning huge swaths of the world. I moved SSH to a very high port, only ran apache, sshd, and an SMTP agent (the machine never received email, inbound was handled by GMail). Still, had someone wanted to exploit the PHP app, I'd be surprised if they couldn't find a weakness. It was custom written, but nearly a decade age.
      - Canada 11y ago
        Elite attackers can compromise almost any target, but even they won't bother finding one-off vulnerabilities when the goal is botnet host acquisition. For that only widely deployed vulnerabilities are worth exploiting. In practice that means you can get away with something as crazy as opening up a password-free root shell on some random high port, but you can't get away with weak SSH logins or unpatched Wordpress sites.
- rtehfm 11y ago
  Better question would be what, assumingly, LAMP stack configuration were they using.
- codezero 11y ago
  I was going to say the opposite. It's awesome that we can spin up boxes and host our own servers, and on top of that, learn from our mistakes. I doubt there's a person here who's never been host to malware, spam or some other malady as a result of some of the experimentation they've done as they learned more about computer systems.
  - bognition 11y ago
    This is exactly my point. One of the major reasons that malware and malicious actors have been able to do as much as they have is because of the large number of misconfigured devices on the internet. The internet has evolved beyond a network cobbled together by a bunch of academics and engineers -- its a critical piece of infrastructure.
    - codezero 11y ago
      Seems like the problem is vulnerable by default options in software, not experimenting with it. More people have hosted malware by just running Windows than anyone has by experimenting with Apache, then fixing it and learning something useful that can now help other newbies.
    - rlpb 11y ago
      The Internet is kept running by a bunch of people who experimented in this way. If they didn't do it any more, in a couple of generations we wouldn't have any competent Internet engineers any more.
      - codezero 11y ago
        Thanks, this puts what I was trying to say in a much more concise and clear context :)
- vog 11y ago
  > should not run their own hardware. Did you mean "should not run their own dedicated server"? The article doesn't mention anything related to hardware. It could have happened with a rented dedicated server, and even within a virtual machine.
jawshie 11y ago
Any idea what the actual vulnerability was?
- pki 11y ago
  my guess would be mod_proxy that didn't explicitly only define a single ip port to proxy to (ala upstream) therefore allowing a http connect?
  - MariuszGalus 11y ago
    I know that they can spoof the referrer. I have mine set to allow '.atrament.net'. I was thinking of using my IP in the setting, but I did not have any problems with '.atrament.net' just yet.
- Sanddancer 11y ago
  It's really, really easy to misconfigure mod_proxy and set yourself up as an open proxy. The ProxyRequests directive sounds like it should be needed for any sort of proxying, but is only really needed if you're allowing your apache instance to act as a forward proxy, not as a reverse proxy. For reverse proxying, which is what you want most of the time, you really want ProxyPass and ProxyPassReverse .
  - scintill76 11y ago
    The phrase "[my blog] was being hosted on another port because apache was taking up the internet http port 80" sounds like the reason they were trying to set up a reverse-proxy. Apache docs have an obvious warning about ProxyRequests and security: https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxyrequests https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxyre... . This config snippet looks like it was copied/modified without understanding: <Proxy *> AddDefaultCharset off Order deny,allow Allow from .example.com </Proxy> Example.com? If you read the docs on Order (https://httpd.apache.org/docs/2.2/mod/mod_authz_host.html#order https://httpd.apache.org/docs/2.2/mod/mod_authz_host.html#or...), you see that Deny,Allow defaults to allow, so that's why it's an open proxy. Above that, there is a comment "turning ProxyRequests on and allowing proxying from all may allow spammers to use your proxy to send email", so I guess it was somewhat safe originally, until ProxyRequests was changed to On without reading and understanding the comment.
nostalgiac 11y ago
So you got to the end solution of... uninstalling fail2ban to fix it? You didn't bother to check WHY it was maxing out the cpu? Glad you got the issue resolved though and didn't fork over the $10 because you would've just run into the same issue in the future if you didn't get to the root cause of it (misconfigured Apache).
- rtkwe 11y ago
  At this point fail2ban wasn't really needed anymore since the author installed it to stop the people using the server as an open proxy. After they solved the proxy issue fail2ban was just causing problems and wasn't needed.
  - nostalgiac 11y ago
    I understand it wasn't needed after the fact. But the entire point of the original post/resolution was to understand what was causing the problem (page time outs) - yet when faced with a second problem, he chose the 'easy' route of just uninstalling it (unalike his resilience to pay for his IP change or wiping the server).
wiradikusuma 11y ago
Honest question from a developer perspective: Why there isn't any "best practice/hardened by default" wizard-style configuration, something people can do right after they install their OS? E.g.: Welcome to Best Practice Linux. Click Next to continue. Which http server you want (httpd/lightttpd/...). Click Next to continue. (you get the idea). Something like apt-get but with best-practice defaults.
- snuxoll 11y ago
  Something that proper config management already solves, the default apache module for puppet places sane defaults on everything, from there it's your responsibility to know what you are doing.
- newjersey 11y ago
  This discussion about otto (from the makers of Vagrant) has some ideas about why people think a one-size-fits-all "default" is a bad/good idea. https://news.ycombinator.com/item?id=10291778 https://news.ycombinator.com/item?id=10291778
- bryogenic 11y ago
  Something link this might be a start: https://github.com/hardening-io/ansible-os-hardening https://github.com/hardening-io/ansible-os-hardening
- AgentME 11y ago
  Apache doesn't default to acting as an open proxy. It already has safe defaults! He specifically configured it this way.
- hackaflocka 11y ago
  This would be a really really good idea. However, in my experience (as an Apache noob), the Apache community consists of experts who are so far ahead of the noobs that they can't see the issue from the perspective of noobs. When I configured my first web box, I couldn't believe that in 2 days it was hacked open and taken over (by some hackers in China apparently -- those guys are scary good). My host (DO) couldn't provide any advice / support on what exactly had happened. I reset everything and set it up again, and again, 2 days later, the box was completely taken over (again by peeps in cn). Finally, I did a couple of tiny tweaks in how I logged in (I disabled root login, and configured SSH keys to log-in, and changed the log-in port), and I was never hacked again. If these 3 little tweaks could be made defaults, there'd be a whole lot less hacking going on.
userbinator 11y ago
As someone who's used open proxies to get around geo-IP-tracking/restrictions/censoring, I get the point about excessive bandwidth usage (you can apply per-IP ratelimiting for that), but it does make me a bit sad that open proxies are now considered "malicious"...
- MariuszGalus 11y ago
  I used the word malicious because I saw people and have a list of credentials now from compromised accounts and spambots. Mainly from Russia. Also, all the sport betting websites that were being hit. I think there must have been something shady there. I was also used for ad-click fraud. :|
SignMeTheHELLUp 11y ago
"How my Apache server became a malicious free internet proxy" tl;dr: Negligence, and failing to RTFM. What really horrifies me is the author doesn't seem to understand the magnitude of their error. The final quip at the end illustrates this. "Ha! someone searched manslaughter over my proxy! I had a lot of fun reading my open proxy logs..." I wonder how many stolen credit card transactions were done over his proxy, causing headaches for many innocent people? Or worse?
- MariuszGalus 11y ago
  (:
- Laforet 11y ago
  I've had similar issues with an open source project in which a simple proxy was established with FiddlerCore to tap traffic to a web browser, pretty tame stuff and nothing malicious. Problem is that by default it was configured to listen on 0.0.0.0:80, making it an open HTTP proxy that everybody on the same LAN could connect to. The only real threat so far is that somebody could send in a large volume of traffic to crash the proxy, but wait and behold.... ...some users were running it from hosts that are either a) directly connected to the public IP space without a firewall and b) behind NAT, but with lazy DMZ/port forwarding configuration that exposes their port 80 to the internet anyway. For about a year people have been obliviously hosting open HTTP proxies from home. Eventually somebody found out and it took another couple of months of back and forth issue reporting and PR tugging battles to get it properly patched. Opsec is hard.
- duskwuff 11y ago
  Bonus TL;DR: By setting "ProxyRequests On", which is only required to use Apache as a forward web proxy. (There's even a big red warning in the documentation about this!)
mkhpalm 11y ago
As a long time Apache user I've never understood using its proxy modules for stuff like this. I've always felt like its much cleaner to just use a small daemon process built solely for the task of reverse proxy or balancing. e.g. haproxy, pound, etc.

53 points by MariuszGalus 11y ago

This is a perfect example of why most people should not run their own hardware. Don't get me wrong its really fun to build and configure your own server and I openly encourage people to learn but I also remind them that its extremely difficult (for a novice) to do securely. Additionally connecting a misconfigured server to the internet doesn't just hurt the server owner but the entire network is affected, as you are providing another piece of hardware that malicious actors can use to execute their attacks.

burnte 11y ago
I ran a machine in a colo for almost four years, public facing web server for a dedicated webapp. I managed to escape any attacks or infections, but I'm positive it was 80% of what I did and 20% that it's a low volume app that never got attention by malicious actors. And I've been in the industry for a long time, not a noob.
- Canada 11y ago
  Everything is attacked. It's all automated, not personal.
  - tomschlick 11y ago
    Yup install something like fail2ban & setup email alerts. You'll get a new notice every 5 minutes or so for the first month from all the china based bots trying to login via common ssh credentials.
    - A010 11y ago
      I run snort and from its alerts, the bots come from all around the world not all from CN. Though the CN sources are largest.
      - protomyth 11y ago
        I was rather surprised when I had a week where most of the attacks came from Las Vegas and New Hampshire.
  - burnte 11y ago
    Well, true, we did get spam attacks on the app, so I wound up banning huge swaths of the world. I moved SSH to a very high port, only ran apache, sshd, and an SMTP agent (the machine never received email, inbound was handled by GMail). Still, had someone wanted to exploit the PHP app, I'd be surprised if they couldn't find a weakness. It was custom written, but nearly a decade age.
    - Canada 11y ago
      Elite attackers can compromise almost any target, but even they won't bother finding one-off vulnerabilities when the goal is botnet host acquisition. For that only widely deployed vulnerabilities are worth exploiting. In practice that means you can get away with something as crazy as opening up a password-free root shell on some random high port, but you can't get away with weak SSH logins or unpatched Wordpress sites.
rtehfm 11y ago
Better question would be what, assumingly, LAMP stack configuration were they using.
codezero 11y ago
I was going to say the opposite. It's awesome that we can spin up boxes and host our own servers, and on top of that, learn from our mistakes. I doubt there's a person here who's never been host to malware, spam or some other malady as a result of some of the experimentation they've done as they learned more about computer systems.
- bognition 11y ago
  This is exactly my point. One of the major reasons that malware and malicious actors have been able to do as much as they have is because of the large number of misconfigured devices on the internet. The internet has evolved beyond a network cobbled together by a bunch of academics and engineers -- its a critical piece of infrastructure.
  - codezero 11y ago
    Seems like the problem is vulnerable by default options in software, not experimenting with it. More people have hosted malware by just running Windows than anyone has by experimenting with Apache, then fixing it and learning something useful that can now help other newbies.
  - rlpb 11y ago
    The Internet is kept running by a bunch of people who experimented in this way. If they didn't do it any more, in a couple of generations we wouldn't have any competent Internet engineers any more.
    - codezero 11y ago
      Thanks, this puts what I was trying to say in a much more concise and clear context :)
vog 11y ago
> should not run their own hardware. Did you mean "should not run their own dedicated server"? The article doesn't mention anything related to hardware. It could have happened with a rented dedicated server, and even within a virtual machine.

jawshie 11y ago

Any idea what the actual vulnerability was?

pki 11y ago
my guess would be mod_proxy that didn't explicitly only define a single ip port to proxy to (ala upstream) therefore allowing a http connect?
- MariuszGalus 11y ago
  I know that they can spoof the referrer. I have mine set to allow '.atrament.net'. I was thinking of using my IP in the setting, but I did not have any problems with '.atrament.net' just yet.
Sanddancer 11y ago
It's really, really easy to misconfigure mod_proxy and set yourself up as an open proxy. The ProxyRequests directive sounds like it should be needed for any sort of proxying, but is only really needed if you're allowing your apache instance to act as a forward proxy, not as a reverse proxy. For reverse proxying, which is what you want most of the time, you really want ProxyPass and ProxyPassReverse .
- scintill76 11y ago
  The phrase "[my blog] was being hosted on another port because apache was taking up the internet http port 80" sounds like the reason they were trying to set up a reverse-proxy. Apache docs have an obvious warning about ProxyRequests and security: https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxyrequests https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxyre... . This config snippet looks like it was copied/modified without understanding: <Proxy *> AddDefaultCharset off Order deny,allow Allow from .example.com </Proxy> Example.com? If you read the docs on Order (https://httpd.apache.org/docs/2.2/mod/mod_authz_host.html#order https://httpd.apache.org/docs/2.2/mod/mod_authz_host.html#or...), you see that Deny,Allow defaults to allow, so that's why it's an open proxy. Above that, there is a comment "turning ProxyRequests on and allowing proxying from all may allow spammers to use your proxy to send email", so I guess it was somewhat safe originally, until ProxyRequests was changed to On without reading and understanding the comment.

nostalgiac 11y ago

So you got to the end solution of... uninstalling fail2ban to fix it? You didn't bother to check WHY it was maxing out the cpu? Glad you got the issue resolved though and didn't fork over the $10 because you would've just run into the same issue in the future if you didn't get to the root cause of it (misconfigured Apache).

rtkwe 11y ago
At this point fail2ban wasn't really needed anymore since the author installed it to stop the people using the server as an open proxy. After they solved the proxy issue fail2ban was just causing problems and wasn't needed.
- nostalgiac 11y ago
  I understand it wasn't needed after the fact. But the entire point of the original post/resolution was to understand what was causing the problem (page time outs) - yet when faced with a second problem, he chose the 'easy' route of just uninstalling it (unalike his resilience to pay for his IP change or wiping the server).

wiradikusuma 11y ago

Honest question from a developer perspective: Why there isn't any "best practice/hardened by default" wizard-style configuration, something people can do right after they install their OS? E.g.: Welcome to Best Practice Linux. Click Next to continue. Which http server you want (httpd/lightttpd/...). Click Next to continue. (you get the idea). Something like apt-get but with best-practice defaults.

snuxoll 11y ago
Something that proper config management already solves, the default apache module for puppet places sane defaults on everything, from there it's your responsibility to know what you are doing.
newjersey 11y ago
This discussion about otto (from the makers of Vagrant) has some ideas about why people think a one-size-fits-all "default" is a bad/good idea. https://news.ycombinator.com/item?id=10291778 https://news.ycombinator.com/item?id=10291778
bryogenic 11y ago
Something link this might be a start: https://github.com/hardening-io/ansible-os-hardening https://github.com/hardening-io/ansible-os-hardening
AgentME 11y ago
Apache doesn't default to acting as an open proxy. It already has safe defaults! He specifically configured it this way.
hackaflocka 11y ago
This would be a really really good idea. However, in my experience (as an Apache noob), the Apache community consists of experts who are so far ahead of the noobs that they can't see the issue from the perspective of noobs. When I configured my first web box, I couldn't believe that in 2 days it was hacked open and taken over (by some hackers in China apparently -- those guys are scary good). My host (DO) couldn't provide any advice / support on what exactly had happened. I reset everything and set it up again, and again, 2 days later, the box was completely taken over (again by peeps in cn). Finally, I did a couple of tiny tweaks in how I logged in (I disabled root login, and configured SSH keys to log-in, and changed the log-in port), and I was never hacked again. If these 3 little tweaks could be made defaults, there'd be a whole lot less hacking going on.

userbinator 11y ago

As someone who's used open proxies to get around geo-IP-tracking/restrictions/censoring, I get the point about excessive bandwidth usage (you can apply per-IP ratelimiting for that), but it does make me a bit sad that open proxies are now considered "malicious"...

MariuszGalus 11y ago
I used the word malicious because I saw people and have a list of credentials now from compromised accounts and spambots. Mainly from Russia. Also, all the sport betting websites that were being hit. I think there must have been something shady there. I was also used for ad-click fraud. :|

SignMeTheHELLUp 11y ago

"How my Apache server became a malicious free internet proxy" tl;dr: Negligence, and failing to RTFM. What really horrifies me is the author doesn't seem to understand the magnitude of their error. The final quip at the end illustrates this. "Ha! someone searched manslaughter over my proxy! I had a lot of fun reading my open proxy logs..." I wonder how many stolen credit card transactions were done over his proxy, causing headaches for many innocent people? Or worse?

MariuszGalus 11y ago
(:
Laforet 11y ago
I've had similar issues with an open source project in which a simple proxy was established with FiddlerCore to tap traffic to a web browser, pretty tame stuff and nothing malicious. Problem is that by default it was configured to listen on 0.0.0.0:80, making it an open HTTP proxy that everybody on the same LAN could connect to. The only real threat so far is that somebody could send in a large volume of traffic to crash the proxy, but wait and behold.... ...some users were running it from hosts that are either a) directly connected to the public IP space without a firewall and b) behind NAT, but with lazy DMZ/port forwarding configuration that exposes their port 80 to the internet anyway. For about a year people have been obliviously hosting open HTTP proxies from home. Eventually somebody found out and it took another couple of months of back and forth issue reporting and PR tugging battles to get it properly patched. Opsec is hard.
duskwuff 11y ago
Bonus TL;DR: By setting "ProxyRequests On", which is only required to use Apache as a forward web proxy. (There's even a big red warning in the documentation about this!)

mkhpalm 11y ago

As a long time Apache user I've never understood using its proxy modules for stuff like this. I've always felt like its much cleaner to just use a small daemon process built solely for the task of reverse proxy or balancing. e.g. haproxy, pound, etc.