7 ms·

pov of sysadmin: - reverse proxy to backend webserver - run checks on each request on the reverse proxy - close all ports except what needs to be open - onl

pov of sysadmin: - reverse proxy to backend webserver - run checks on each request on the reverse proxy - close all ports except what needs to be open - only allow access to what needs to be seen - perform frequent security updates (script it, automate it, run it hourly) - never install more software then strictly required - never allow SSH directly to your public address - for ssh, use custom ports, cert (4096b) with passwords - install logwatch or something else that will inform you about logs (elk stack) - automate checks in logs for tell-tale signs of scriptkiddies (attempted access to /wp-admin or phpmyadmin for example) => automate adding source IP's to firewalls - frequently check for malicious code (clamav, rkhunter, ...) - only use stable and tested software - only use stable and tested OS (Centos/redhat. DO NOT run production on anything else, especially ubuntu.) - do not use ubuntu for production. it is not suited. - install failtoban from dev pov: - implement measures to prevent sql injection - don't require production to install dev libraries - don't require production to install packages only available on ubuntu - log strange behaviour to a file, make sure actions can be automated (iptable drop) I'm no dev, but those are things I'd like dev's to do with their software.

no1youknowz 9y ago
> - reverse proxy to backend webserver Can you give more clarification? Do you mean LB -> Firewall -> Web Server? > - run checks on each request on the reverse proxy Can you give more information on how to do this please. Thanks
- jonssons 9y ago
  > - reverse proxy to backend webserver Using a reverse proxy allows you to cache static content, decreasing load. I'm not talking about load balancers. These days you need them when your website pulls _a lot_ of requests (1000/s) Your reverse proxy will also stop unwanted requests and indeed will have a firewall incorperated. > - run checks on each request on the reverse proxy post => clamav post => sqlinjection get => blacklisted address? any => iptables (let any connection go through your firewall. It's a check) don't terminate SSL on reverse proxy though. It's a bad idea, unless you are a bad person and like to read your visitors data they think is secure. There are plenty of other ways of using security on a reverse proxy. Have a look around on google.
corobo 9y ago
No backups? Security involves the ability to recover from a breach too! :)
- jonssons 9y ago
  if you wanna go there. I use a custom script to make backups, encrypt them using gpg and store them on public ftp boxes. Automated retrieval of said backup in case of problems and automated server recovery in case of disaster.
  - deleted 9y ago
    [deleted]
ge96 9y ago
Curious if you're not a dev how do you know this (no offense meant). Surprised about Ubuntu, what about Debian? Share same... Hmm... A lot of these I have to tick off.
- rpod 9y ago
  As a sysadmin, you'd have to have first-hand experience on what can go wrong on the dev side. Also curious as to why Ubuntu isn't suitable for production.
  - ge96 9y ago
    Maybe I have been lucky I am using nginx now on top of centos for one site but the reverse proxy for example not using that. I am on an Amazon EC2 in this particular case, also curious that Apache was also on there but only nginx is running. Had a long TTFB was trying to figure out what was wrong. Upon restart (new instance) I guess I got lucky, now back to regular site speed around 2 seconds load (full page). Sorry rambling, cool stuff. Been on VPS, I guess EC2 is still the same thing but next would be serverless I guess.
- jonssons 9y ago
  I am a sysadmin with approx 15 years of experience running linux systems and open source software. When I set things up, I tend to form things my way. I usually have little resources for monitoring or staff, so automation is the best to do. A server serves, it may produce errors and you can act upon that (error 404 for not finding phpmyadmin (which should _never_ be public => add ip to drop table). There are scripts that can do this for you. However, the application which is being served, may produce different errors (failed login attempt, attempt for sql injection, cross site attack, ...). Some things are better logged by the application, Then the application needs to produce these logs. An awful lot of applications do nothing when presented with any of these type of attacks, making them extremely vulnerable. It's the dev's job to provide logging. Debian: fine if you like pain :) it's stable.
  - ge96 9y ago
Jonnax 9y ago
Why is Ubuntu not suited to production? I keep hearing it but it's starting to sound like Red Hat marketing like Macs don't get viruses.
- kusmi 9y ago
  I think it can be used in production but it's more work. Red hat and CentOS can be hardened following STIG security profile on first boot from fresh install which includes SElinux checks, firewall and service configurations, etc. I'm sure there's more to it, but reading documentation on these alone is enough to overwhelm me.
- yellowapple 9y ago
  An LTS Ubuntu release should be fine, though I'd hold off on using it in production until a year or so after release (to give Canonical time to shake out bugs). A non-LTS Ubuntu release should be avoided, since they tend to be less stable. If you're using something in the Debian family, Debian itself is probably the much better choice unless you're looking to use Canonical's enterprise products.
- jonssons 9y ago
  In my experience: - Ubuntu is not stable - it is not hardened out of the box - it can contain dangerous configuration out of the box - it calls home - it will tempt you to install dev packages or other packages unfit for production (because devs dev on ubuntu and they require the dev packages) But please, don't mind me. Other people are better at explaining this. I'm biassed.
LinuxBender 9y ago
For my own sites, I do all of the things jonssons said, but I also use haproxy stick table to limit the number of errors a user may generate. I also route certain URI patterns to a tarpit. At work, I show developers, ops and infrastructure folks these things and let them decide if they want to test and implement any of it.
atmosx 9y ago
If you don't use FIM there is no way in hell the above list will protect you from targeted attacks.
- jonssons 9y ago
  rkhunter does fim in a limited way. I hope OP is not talking about sites with sensitive data. ossec is usually great for storing more sensitive data. The above list will help you from most scriptkiddy attacks, which are 99% of the attacks I have experienced. It's a good idea not to piss off anybody on the net or do it with a sufficiently obscure account.
  - atmosx 9y ago
    Agreed. I would add "auditd" and "acct on" to your list.
DKnoll 9y ago
> - only use stable and tested OS (Centos/redhat. DO NOT run production on anything else, especially ubuntu.) I'd tend to agree about Ubuntu because... why aren't you just using Debian? I would generally opt for CentOS for a prod server, but I am now starting to think about using OpenBSD in production environments. Ultimately it comes down to the way you configure the server, not the distro you use. The only benefit of Enterprise Linux (CentOS, Red Hat, Fedora, Oracle) over Debian out-of-the-box (that I am aware of) is the presence of SELinux, which you can still set up on Debian. All have insecure configs by default. I will say that the live kernel patching in Ubuntu Server now is cool. I used KSplice on CentOS before it was discontinued (only on Oracle Linux now).
- jonssons 9y ago
  I don't use debian because I don't like apt. The most vulnerable moment of my systems is when the ISP sends the password in plain text over e-mail. Until the password has been changed and the system has been sanitized(), I consider it compromised. () updated, rkhunter, clamav, ...
- deleted 9y ago
  [deleted]

no1youknowz 9y ago

> - reverse proxy to backend webserver Can you give more clarification? Do you mean LB -> Firewall -> Web Server? > - run checks on each request on the reverse proxy Can you give more information on how to do this please. Thanks

jonssons 9y ago
> - reverse proxy to backend webserver Using a reverse proxy allows you to cache static content, decreasing load. I'm not talking about load balancers. These days you need them when your website pulls _a lot_ of requests (1000/s) Your reverse proxy will also stop unwanted requests and indeed will have a firewall incorperated. > - run checks on each request on the reverse proxy post => clamav post => sqlinjection get => blacklisted address? any => iptables (let any connection go through your firewall. It's a check) don't terminate SSL on reverse proxy though. It's a bad idea, unless you are a bad person and like to read your visitors data they think is secure. There are plenty of other ways of using security on a reverse proxy. Have a look around on google.

corobo 9y ago

No backups? Security involves the ability to recover from a breach too! :)

jonssons 9y ago
if you wanna go there. I use a custom script to make backups, encrypt them using gpg and store them on public ftp boxes. Automated retrieval of said backup in case of problems and automated server recovery in case of disaster.
- deleted 9y ago
  [deleted]

ge96 9y ago

Curious if you're not a dev how do you know this (no offense meant). Surprised about Ubuntu, what about Debian? Share same... Hmm... A lot of these I have to tick off.

rpod 9y ago
As a sysadmin, you'd have to have first-hand experience on what can go wrong on the dev side. Also curious as to why Ubuntu isn't suitable for production.
- ge96 9y ago
  Maybe I have been lucky I am using nginx now on top of centos for one site but the reverse proxy for example not using that. I am on an Amazon EC2 in this particular case, also curious that Apache was also on there but only nginx is running. Had a long TTFB was trying to figure out what was wrong. Upon restart (new instance) I guess I got lucky, now back to regular site speed around 2 seconds load (full page). Sorry rambling, cool stuff. Been on VPS, I guess EC2 is still the same thing but next would be serverless I guess.
jonssons 9y ago
I am a sysadmin with approx 15 years of experience running linux systems and open source software. When I set things up, I tend to form things my way. I usually have little resources for monitoring or staff, so automation is the best to do. A server serves, it may produce errors and you can act upon that (error 404 for not finding phpmyadmin (which should _never_ be public => add ip to drop table). There are scripts that can do this for you. However, the application which is being served, may produce different errors (failed login attempt, attempt for sql injection, cross site attack, ...). Some things are better logged by the application, Then the application needs to produce these logs. An awful lot of applications do nothing when presented with any of these type of attacks, making them extremely vulnerable. It's the dev's job to provide logging. Debian: fine if you like pain :) it's stable.
- ge96 9y ago

Jonnax 9y ago

Why is Ubuntu not suited to production? I keep hearing it but it's starting to sound like Red Hat marketing like Macs don't get viruses.

kusmi 9y ago
I think it can be used in production but it's more work. Red hat and CentOS can be hardened following STIG security profile on first boot from fresh install which includes SElinux checks, firewall and service configurations, etc. I'm sure there's more to it, but reading documentation on these alone is enough to overwhelm me.
yellowapple 9y ago
An LTS Ubuntu release should be fine, though I'd hold off on using it in production until a year or so after release (to give Canonical time to shake out bugs). A non-LTS Ubuntu release should be avoided, since they tend to be less stable. If you're using something in the Debian family, Debian itself is probably the much better choice unless you're looking to use Canonical's enterprise products.
jonssons 9y ago
In my experience: - Ubuntu is not stable - it is not hardened out of the box - it can contain dangerous configuration out of the box - it calls home - it will tempt you to install dev packages or other packages unfit for production (because devs dev on ubuntu and they require the dev packages) But please, don't mind me. Other people are better at explaining this. I'm biassed.

LinuxBender 9y ago

For my own sites, I do all of the things jonssons said, but I also use haproxy stick table to limit the number of errors a user may generate. I also route certain URI patterns to a tarpit. At work, I show developers, ops and infrastructure folks these things and let them decide if they want to test and implement any of it.

atmosx 9y ago

If you don't use FIM there is no way in hell the above list will protect you from targeted attacks.

jonssons 9y ago
rkhunter does fim in a limited way. I hope OP is not talking about sites with sensitive data. ossec is usually great for storing more sensitive data. The above list will help you from most scriptkiddy attacks, which are 99% of the attacks I have experienced. It's a good idea not to piss off anybody on the net or do it with a sufficiently obscure account.
- atmosx 9y ago
  Agreed. I would add "auditd" and "acct on" to your list.

DKnoll 9y ago

> - only use stable and tested OS (Centos/redhat. DO NOT run production on anything else, especially ubuntu.) I'd tend to agree about Ubuntu because... why aren't you just using Debian? I would generally opt for CentOS for a prod server, but I am now starting to think about using OpenBSD in production environments. Ultimately it comes down to the way you configure the server, not the distro you use. The only benefit of Enterprise Linux (CentOS, Red Hat, Fedora, Oracle) over Debian out-of-the-box (that I am aware of) is the presence of SELinux, which you can still set up on Debian. All have insecure configs by default. I will say that the live kernel patching in Ubuntu Server now is cool. I used KSplice on CentOS before it was discontinued (only on Oracle Linux now).

jonssons 9y ago
I don't use debian because I don't like apt. The most vulnerable moment of my systems is when the ISP sends the password in plain text over e-mail. Until the password has been changed and the system has been sanitized(), I consider it compromised. () updated, rkhunter, clamav, ...
deleted 9y ago
[deleted]