6 ms·

Bastion hosts

62 points by slem 12y ago

slem 12y ago
Bastion host(s) are a useful and important component of a system management infrastructure. A bastion-host, in this context, is actually more properly, but more obscurely, called a jump server. In this post I will simply use the term bastion host. It is the most commonly used term for the system's function: a server, which has undergone security hardening steps, that is the operational and administrative control point for systems and hosts in a datacenter (or AWS Region).
neohaven 12y ago
Direct quote from the page at the other end of this link : "You appear to be using a browser that may not be able to display our site correctly. Generally, this is beause it's an older browser that doesn't support a lot of the newer, awesome features in open Web standards. Rather than provide an untested and possibly subpar experience of our site, may we suggest downloading one of these fine browsers? (We think you'll find your overall experience of the Web will improve if you do.)" This is followed with links for Firefox and Chrome. I am running Firefox 24.4 from CentOS repos. This is unskippable, and there's no simple way to bypass it on the page. On a browser from March of this year. Urgh.
- GrinningFool 12y ago
  Yeah. Welcome to the world of an HTML 'standard' that is 'living'.
- tedunangst 12y ago
  Since it's only a plaintext page, you can read it with curl. Or you could, except the server is broken and sends back gzipped data even if the client doesn't request it. curl https://www.pandastrike.com/posts/20141113-bastion-hosts | gunzip | more
  - neohaven 12y ago
    I shouldn't have to grab a page curl to see a website. A disclaimer of "Hey we notice $X is your browser, maybe switch to those?" with a "No thanks, I'll deal with it" option on some modal JS dialog.
- deleted 12y ago
  [deleted]
- cdubz 12y ago
  TBH, > 6 months old is hopelessly outdated. Welcome to the future, grampa.
  - voltagex_ 12y ago
    Some of us are using browsers imposed upon us by $EMPLOYER, and may not be able to update of our own volition.
  - ryan-c 12y ago
    I think Firefox 24 is still supported for a few more weeks, as it is a "long term" release.
  - vacri 12y ago
    The point of those LTS browsers is so that an employer doesn't have to watch for UI changes in their browser every 6 weeks, and then retrain all their staff when the browser moves things around or makes things look different.
tptacek 12y ago
Jump boxes are a good idea. A better idea, if you're in AWS, is to use a VPC and just VPN into your environment; the VPN terminator in the VPC serves the same purpose as a jump box. If OpenVPN scares you, you can still do layer-4 filtering to minimize the impact of losing the OpenVPN server (ie, to get it to "no worse than if we hadn't used VPCs in the first place). "Single-packet authentication" is, I think, silly, and I've never recommended it to anyone. It's hard to think of a case where hand-rolled authentication in front of SSH has saved anyone with a proper SSH configuration (keys-only-no-passwords). You can skip that step.
- mahmoudimus 12y ago
  Do you have a good reference to administering an OpenVPN server? Setting it up is easy, setting it up with HA is also no problem, but administrating OpenVPN for a startup that doesn't have a dedicated sysadmin would be a very useful guide.
- phil21 12y ago
  > A better idea, if you're in AWS, is to use a VPC and just VPN into your environment; Is it though? Is having your (and your co-workers) desktops/laptops/and possibly more directly able to access production servers at a packet level a good idea? I would posit that it is not. A bastion host can be used to lock down more than firewalls, and reduces the attack surface living on your network considerably. > VPN terminator in the VPC serves the same purpose as a jump box. It certainly serves the same network separation purposes a jump box does, but is not able to dictate the access environment nearly as much. If only your jump box has access to servers (and production servers properly filtered), every bit of remote access must go through that single ingress/egress point. And that point can have security policy built-in as well for multi-person teams. For example: I can enforce certain key types being utilized on remote hosts, logging of ssh sessions, etc. I also agree the "single packet authentication" is silly though. Most bastion hosts these days will be behind VPN already. Just another riff on port knocking really.
  - jpgvm 12y ago
    Bastion hosts are generally a bad idea. There is not a single implementation of a bastion host that doesn't have significant security holes. This is doubly true if you want to do proper group based access control. Most notable of these issues is the nature of either needing SSH agent forwarding enabled or storing a copy of the private keys on the bastion host (please don't do that). There are ofcourse ways of dealing with this but they aren't worth it. It's fundamentally much better to just harden every system and implement proper federated authentication and authorization with LDAP. It also makes it much nicer to implement things like MFA and device level authentication. Remember, it's very rarely that an attacker is going to be able to successfully walk in or break down the front door. Your security holes are very unlikely to be in your administrative channels like SSH/RDP but rather crappy applications or services you may be running on your systems. Those crappy bits of software + insider threat are the 2 main things you generally need to deal with. Ensuring systems are sufficiently isolated from one another helps with both of these and bastion hosts just add another layer of shared stuff between workloads that should stay entirely separate.
cthalupa 12y ago
Jumphosts have their uses, but in this case, I really don't see it. As others have pointed out, better yet is VPN only access. As for the port knocking silliness, well, I don't see any reason for it. http://bsdly.blogspot.com/2012/04/why-not-use-port-knocking.html http://bsdly.blogspot.com/2012/04/why-not-use-port-knocking....
- akerl_ 12y ago
  It's worth noting that single packet auth is distinct from port knocking. fwknop isn't relying on "secret" port combinations: it's using GPG to sign/encrypt requests.
  - cthalupa 12y ago
    It's not meaningfully different - most of the same flaws still apply. Say you have a bad actor behind the same NAT'ed IP. Good User A signs their SPA, and the firewall opens up for 30 seconds... during which Malicious User B has access. What real advantage are you gaining vs. just having restricted firewall rules in the first place? Lowering the window of time an attacker has access? If you're not using password auth that could be brute forced, who cares about the window? Either they have the keys or an exploit needed to get in, or they don't.
    - sophacles 12y ago
      The advantage over vanilla restrictive firewalls is ability to come from many different IPs without cluttering logs. In an enterprise setting, this is probably not helpful. At home or for private servers, this is a nice feature. (Also at home, it helps reduce stupid ISP imposed rules on not hosting ssh... their scanners just assume "no server here" but I can still get to my home network)
- rsync 12y ago
  I've been doing this sysadmin thing for a long, long time, and have built some large, high profile platforms,[1][2] and I can tell you that few things have made me happier than port knocking. Yes, it is a password with a tiny number of "bits". But nobody suggested using only port knocking, so that's not significant. Yes, if my attacker is sharing the exact same NAT'd IP as me then they have a window to attack ... my fully hardened server, which was no worse than what was presented to them before. Finally (and this is what makes me happiest) my sshd is not exposed to whatever new exploit might be discovered tomorrow that doesn't require user credentials. It's not exposed to being DOS'd by a buffer overflow vuln. It's not filling my logs with brute forcers all day and night. I love port knocking. [1] JohnCompanies (first VPS provider) [2] rsync.net
kev009 12y ago
This article goes from mediocre to bad when it starts advocating security by obscurity.
- the_french 12y ago
  Security by obscurity is perfectly fine when it's part of a layered approach. It will help keep attackers from learning the details of your architecture and will also stop skiddies in their tracks.
  - rsync 12y ago
    Exactly. (Your most secure server) is not as secure as (your most secure server with some obscurity added).
  - arde 12y ago
    This. Obscurity has too much bad press. It's not to be depended on, but it can be a useful addition sometimes.
lmm 12y ago
Bastion hosts are the wrong approach, as are VPNs. They might have had some merit in the days of handcrafted servers, where "hardening" was something you could do to a single machine. But nowadays, especially on AWS, there's no point in a configuration that isn't automatically reproducible - in which case, why not harden all your machines? Then you avoid having a single point of failure. If and when a single SSH key is compromised, there are a limited number of hosts that key had access to - and unless that's the log server, you have secured logs so you can figure out what happened. Different keys can have different capabilities, rather than a one-size-fits-all policy on the bastion. It makes it harder for an attacker to know where to get the boss's keys. And you can expand your system geographically without the performance compromises of a VPN; your servers already make encrypted, secured calls to each other (over SSH, SSL or a very small number of vetted protocols), expose only those ports that are actively used. Your attacker could steal your entire system documentation and it wouldn't make any difference, because the system is truly secured, not relying on obscurity.
- jpgvm 12y ago
  Better yet. Store the authorized public keys in LDAP or similar system. (AuthorizedKeysCommand makes this super simple these days). That way you can rotate them easily if the private keys become compromised. I recently built a system using this technique and synced the keys to S3, very effective.
- rudolf0 12y ago
  You're describing an overly optimistic scenario. "Why not just harden all your machines?" Because it's more complicated than that. Yes, you can automate basic hardening (configuration of access controls, limited use of SSH keys), but how do you automate hardening of RPC between one server and another, or hardening of those 3 internal administration web apps which may or may not have some local file disclosure and crypto flaws? Defense-in-depth is still the only sensible way to handle security. You do what you can on the host side, but bastion hosts, jump hosts, VPNs, etc. provide an additional layer to make lateral movement much more difficult for an attacker.
peterwwillis 12y ago
In the Enterprise world, jump boxes are more typically just a hack implemented by IT workers who don't want to go about properly setting up a firewall and 802.1x/vlan/vpn/ipsec/etc to allow strict control to a protected network resource. (Or because they don't have the hardware capability to do it, or their network admin decided against it). The only two proper excuses for ever using a bastion host are "I don't have the money to do it right" and "I don't have the time to do it right."
corford 12y ago
I'm a fan of the VPN terminator as jump point approach. OpenVPN with tls-auth switched on is a pretty robust front door. If you combine that with enforced key only auth for ssh, ensure all internal machines run their own firewall and logically split different bits of your internal network off in to their own subnets you've got a fairly secure foundation to build upon. Another easy win is switching internal services over to TLS/SSL only (where it makes sense/is possible to do so).
nullc 12y ago
You appear to be using a browser that may not be able to display our site correctly. Generally, this is beause it's an older browser that doesn't support a lot of the newer, awesome features in open Web standards. Rather than provide an untested and possibly subpar experience of our site, may we suggest downloading one of these fine browsers? (We think you'll find your overall experience of the Web will improve if you do.)

62 points by slem 12y ago

slem 12y ago

Bastion host(s) are a useful and important component of a system management infrastructure. A bastion-host, in this context, is actually more properly, but more obscurely, called a jump server. In this post I will simply use the term bastion host. It is the most commonly used term for the system's function: a server, which has undergone security hardening steps, that is the operational and administrative control point for systems and hosts in a datacenter (or AWS Region).

neohaven 12y ago

Direct quote from the page at the other end of this link : "You appear to be using a browser that may not be able to display our site correctly. Generally, this is beause it's an older browser that doesn't support a lot of the newer, awesome features in open Web standards. Rather than provide an untested and possibly subpar experience of our site, may we suggest downloading one of these fine browsers? (We think you'll find your overall experience of the Web will improve if you do.)" This is followed with links for Firefox and Chrome. I am running Firefox 24.4 from CentOS repos. This is unskippable, and there's no simple way to bypass it on the page. On a browser from March of this year. Urgh.

GrinningFool 12y ago
Yeah. Welcome to the world of an HTML 'standard' that is 'living'.
tedunangst 12y ago
Since it's only a plaintext page, you can read it with curl. Or you could, except the server is broken and sends back gzipped data even if the client doesn't request it. curl https://www.pandastrike.com/posts/20141113-bastion-hosts | gunzip | more
- neohaven 12y ago
  I shouldn't have to grab a page curl to see a website. A disclaimer of "Hey we notice $X is your browser, maybe switch to those?" with a "No thanks, I'll deal with it" option on some modal JS dialog.
deleted 12y ago
[deleted]
cdubz 12y ago
TBH, > 6 months old is hopelessly outdated. Welcome to the future, grampa.
- voltagex_ 12y ago
  Some of us are using browsers imposed upon us by $EMPLOYER, and may not be able to update of our own volition.
- ryan-c 12y ago
  I think Firefox 24 is still supported for a few more weeks, as it is a "long term" release.
- vacri 12y ago
  The point of those LTS browsers is so that an employer doesn't have to watch for UI changes in their browser every 6 weeks, and then retrain all their staff when the browser moves things around or makes things look different.

tptacek 12y ago

Jump boxes are a good idea. A better idea, if you're in AWS, is to use a VPC and just VPN into your environment; the VPN terminator in the VPC serves the same purpose as a jump box. If OpenVPN scares you, you can still do layer-4 filtering to minimize the impact of losing the OpenVPN server (ie, to get it to "no worse than if we hadn't used VPCs in the first place). "Single-packet authentication" is, I think, silly, and I've never recommended it to anyone. It's hard to think of a case where hand-rolled authentication in front of SSH has saved anyone with a proper SSH configuration (keys-only-no-passwords). You can skip that step.

mahmoudimus 12y ago
Do you have a good reference to administering an OpenVPN server? Setting it up is easy, setting it up with HA is also no problem, but administrating OpenVPN for a startup that doesn't have a dedicated sysadmin would be a very useful guide.
phil21 12y ago
> A better idea, if you're in AWS, is to use a VPC and just VPN into your environment; Is it though? Is having your (and your co-workers) desktops/laptops/and possibly more directly able to access production servers at a packet level a good idea? I would posit that it is not. A bastion host can be used to lock down more than firewalls, and reduces the attack surface living on your network considerably. > VPN terminator in the VPC serves the same purpose as a jump box. It certainly serves the same network separation purposes a jump box does, but is not able to dictate the access environment nearly as much. If only your jump box has access to servers (and production servers properly filtered), every bit of remote access must go through that single ingress/egress point. And that point can have security policy built-in as well for multi-person teams. For example: I can enforce certain key types being utilized on remote hosts, logging of ssh sessions, etc. I also agree the "single packet authentication" is silly though. Most bastion hosts these days will be behind VPN already. Just another riff on port knocking really.
- jpgvm 12y ago
  Bastion hosts are generally a bad idea. There is not a single implementation of a bastion host that doesn't have significant security holes. This is doubly true if you want to do proper group based access control. Most notable of these issues is the nature of either needing SSH agent forwarding enabled or storing a copy of the private keys on the bastion host (please don't do that). There are ofcourse ways of dealing with this but they aren't worth it. It's fundamentally much better to just harden every system and implement proper federated authentication and authorization with LDAP. It also makes it much nicer to implement things like MFA and device level authentication. Remember, it's very rarely that an attacker is going to be able to successfully walk in or break down the front door. Your security holes are very unlikely to be in your administrative channels like SSH/RDP but rather crappy applications or services you may be running on your systems. Those crappy bits of software + insider threat are the 2 main things you generally need to deal with. Ensuring systems are sufficiently isolated from one another helps with both of these and bastion hosts just add another layer of shared stuff between workloads that should stay entirely separate.

cthalupa 12y ago

Jumphosts have their uses, but in this case, I really don't see it. As others have pointed out, better yet is VPN only access. As for the port knocking silliness, well, I don't see any reason for it. http://bsdly.blogspot.com/2012/04/why-not-use-port-knocking.html http://bsdly.blogspot.com/2012/04/why-not-use-port-knocking....

akerl_ 12y ago
It's worth noting that single packet auth is distinct from port knocking. fwknop isn't relying on "secret" port combinations: it's using GPG to sign/encrypt requests.
- cthalupa 12y ago
  It's not meaningfully different - most of the same flaws still apply. Say you have a bad actor behind the same NAT'ed IP. Good User A signs their SPA, and the firewall opens up for 30 seconds... during which Malicious User B has access. What real advantage are you gaining vs. just having restricted firewall rules in the first place? Lowering the window of time an attacker has access? If you're not using password auth that could be brute forced, who cares about the window? Either they have the keys or an exploit needed to get in, or they don't.
  - sophacles 12y ago
    The advantage over vanilla restrictive firewalls is ability to come from many different IPs without cluttering logs. In an enterprise setting, this is probably not helpful. At home or for private servers, this is a nice feature. (Also at home, it helps reduce stupid ISP imposed rules on not hosting ssh... their scanners just assume "no server here" but I can still get to my home network)
rsync 12y ago
I've been doing this sysadmin thing for a long, long time, and have built some large, high profile platforms,[1][2] and I can tell you that few things have made me happier than port knocking. Yes, it is a password with a tiny number of "bits". But nobody suggested using only port knocking, so that's not significant. Yes, if my attacker is sharing the exact same NAT'd IP as me then they have a window to attack ... my fully hardened server, which was no worse than what was presented to them before. Finally (and this is what makes me happiest) my sshd is not exposed to whatever new exploit might be discovered tomorrow that doesn't require user credentials. It's not exposed to being DOS'd by a buffer overflow vuln. It's not filling my logs with brute forcers all day and night. I love port knocking. [1] JohnCompanies (first VPS provider) [2] rsync.net

kev009 12y ago

This article goes from mediocre to bad when it starts advocating security by obscurity.

the_french 12y ago
Security by obscurity is perfectly fine when it's part of a layered approach. It will help keep attackers from learning the details of your architecture and will also stop skiddies in their tracks.
- rsync 12y ago
  Exactly. (Your most secure server) is not as secure as (your most secure server with some obscurity added).
- arde 12y ago
  This. Obscurity has too much bad press. It's not to be depended on, but it can be a useful addition sometimes.

lmm 12y ago

Bastion hosts are the wrong approach, as are VPNs. They might have had some merit in the days of handcrafted servers, where "hardening" was something you could do to a single machine. But nowadays, especially on AWS, there's no point in a configuration that isn't automatically reproducible - in which case, why not harden all your machines? Then you avoid having a single point of failure. If and when a single SSH key is compromised, there are a limited number of hosts that key had access to - and unless that's the log server, you have secured logs so you can figure out what happened. Different keys can have different capabilities, rather than a one-size-fits-all policy on the bastion. It makes it harder for an attacker to know where to get the boss's keys. And you can expand your system geographically without the performance compromises of a VPN; your servers already make encrypted, secured calls to each other (over SSH, SSL or a very small number of vetted protocols), expose only those ports that are actively used. Your attacker could steal your entire system documentation and it wouldn't make any difference, because the system is truly secured, not relying on obscurity.

jpgvm 12y ago
Better yet. Store the authorized public keys in LDAP or similar system. (AuthorizedKeysCommand makes this super simple these days). That way you can rotate them easily if the private keys become compromised. I recently built a system using this technique and synced the keys to S3, very effective.
rudolf0 12y ago
You're describing an overly optimistic scenario. "Why not just harden all your machines?" Because it's more complicated than that. Yes, you can automate basic hardening (configuration of access controls, limited use of SSH keys), but how do you automate hardening of RPC between one server and another, or hardening of those 3 internal administration web apps which may or may not have some local file disclosure and crypto flaws? Defense-in-depth is still the only sensible way to handle security. You do what you can on the host side, but bastion hosts, jump hosts, VPNs, etc. provide an additional layer to make lateral movement much more difficult for an attacker.

peterwwillis 12y ago

In the Enterprise world, jump boxes are more typically just a hack implemented by IT workers who don't want to go about properly setting up a firewall and 802.1x/vlan/vpn/ipsec/etc to allow strict control to a protected network resource. (Or because they don't have the hardware capability to do it, or their network admin decided against it). The only two proper excuses for ever using a bastion host are "I don't have the money to do it right" and "I don't have the time to do it right."

corford 12y ago

I'm a fan of the VPN terminator as jump point approach. OpenVPN with tls-auth switched on is a pretty robust front door. If you combine that with enforced key only auth for ssh, ensure all internal machines run their own firewall and logically split different bits of your internal network off in to their own subnets you've got a fairly secure foundation to build upon. Another easy win is switching internal services over to TLS/SSL only (where it makes sense/is possible to do so).

nullc 12y ago

You appear to be using a browser that may not be able to display our site correctly. Generally, this is beause it's an older browser that doesn't support a lot of the newer, awesome features in open Web standards. Rather than provide an untested and possibly subpar experience of our site, may we suggest downloading one of these fine browsers? (We think you'll find your overall experience of the Web will improve if you do.)