7 ms·

Jump Servers

100 points by andriosr 4y ago

yjftsjthsd-h 4y ago
Er, so most of those supposed problems sound like you could fix them by just using unprivileged users jumping through the jump server via `ssh -J`/ProxyJump? Or a VPN, which is pretty close to what it sounds like the product they're trying to sell is anyways. And if you're going to try and sell a Teleport competitor, you really should do a better job convincing me that it's going to actually be secure in the first place. I don't see source code or audits anywhere on this website.
- POPOSYS 4y ago
  But it is written on the website: "without security compromise" - so it must be secure!!! Also there is a lot of low contrast text on that website - this shows that they really know what they are doing and you can trust them. I am sure it is a very good company and you should give all the login credentials of your org to them and also install their binary on all servers. Isn´t "trust in company" the reason why you are using Linux servers?
SomaticPirate 4y ago
Google Cloud’s IAP tunneling has basically made this a non-issue. All access is logged and its extremely easy to get around without any public IPs. Also easy to port-forward instances to your local machine. I’m always surprised that other clouds don’t have this.
- drowsspa 4y ago
  Isn't it basically AWS SSM? you can even configure it in the .ssh/config
berkle4455 4y ago
AWS Session Manager (aws specific) or Tailscale (anything) has solved this problem for me
- threeseed 4y ago
  Jump hosts seem like an anti-pattern in the era of AWS SSM and Tailscale. It is far too easy to misconfigure network policies and grant them access to infrastructure that they shouldn't. And with Tailscale you can run the agent within SaaS products like Github Actions or Terraform Cloud to securely manage their access into your systems.
  - theteapot 4y ago
    > anti-pattern Is wearing blue denim jacket and jeans an anti-pattern? Is vegemite an anti-pattern? Can't we just say "obsolete", or "bad idea"?
    - muppetman 4y ago
      You should "reach out" to him/her and sort this out. [I've always found "reach out" to be hilarious, (what, with my arms?) but it's so common now I'm the oddball]
      - theteapot 4y ago
        > I've always found "reach out" to be hilarious, (what, with my arms?) ... Using the expression "reach out" is such an anti-pattern.
      - deathanatos 4y ago
        Oh, the one that kills me now is "let's take that offline" used in a meeting to refer to "let's move this to a Slack conversation". (Or, more cynically, "This problem is thorny and I wish to bury it rather than put in the time and work to fix it.")
    - kevsim 4y ago
      > Is wearing blue denim jacket and jeans an anti-pattern? The Canadian tuxedo is clearly a best practice.
    - dspillett 4y ago
phamilton 4y ago
I've spent the last 7 years using SSH to get onto a prod box a few dozen times. Less than once a month. When I do have to get on the box, we use signed keys that expire in 24 hours (a manager or SRE is required to sign the key if you need to get on a box). The article is talking about the scaling issues of SSH access via jump boxes or bastions. I'd argue that a better solution to scaling SSH access is to invest in tooling that makes SSH access unnecessary for most of your team. Centralized logs, error reporting, distributed traces, etc. are fairly well solved for most installations. Interactive access to prod (e.g. running DB migrations) requires a little more investment, but tools like ECS Exec make that fairly accessible without requiring SSH access.
- vasco 4y ago
  DB migrations should definitely not be done interactively and definitely not in an SSH session. Write them beforehand, have them reviewed with the rest of the code, and have your deployment process run them.
  - PedroBatista 4y ago
    Agree, but in the real world that works 100% most of the time.
    - ranting-moth 4y ago
      100% most of the time. One of those rare cases when I laugh out loud at a HN comment. I'm definitely stealing this phrase!
  - phamilton 4y ago
    Just to clarify our process: DB migrations are part of code review, they run as a script, but we don't have that script run automatically as part of CD. We've been bit by enough migration surprises that we require someone watching and able to interrupt and cancel the migration if needed. But that's the extent of action required. Run this command, Ctrl-C if necessary. Definitely not YOLO'ing in a psql in prod.
mbreese 4y ago
Even when using a jump server, I've solved the issue of revoking access in the past by using LDAP to control access to servers. Instead of adding user accounts directly to server, the account information is stored in LDAP -- including public SSH keys. If you want to revoke a user's access across the entire infrastructure, then you can do so in one fell swoop. I also have set this up to restrict SSH access to particular hosts based on the LDAP record. The problem they are describing isn't a problem with Jump servers... it's a problem with distributed authorization.
- debarshri 4y ago
  I haven't seen many organisations actually set SSH PAM via LDAP or another delegated authentication system. You are right about the problem statement. It is distributed authorization problem. And it is a very hard problem to solve or visualize for a fast moving company unless it is a problem.
  - POPOSYS 4y ago
    Do you have some data about how many orgs actually set SSH PAM via LDAP? Where can I see this data? I would like to check your statements. We are on the internet, so would you please like to add the source of knowledge to your statements - it is a very basic and good feature called "URL", please use it! Or do you just want to say "I have not seen many orgs with SSH + LDAP in my career because I have never worked in one and from that I conclude that the whole world works like this"?
    - debarshri 4y ago
      I don't think neither me nor anyone reading that statement would come to a conclusion that the works the way I suggested in that argument. I can talk about me interviewing and empirical data of talking to dozens of companies from seed to series B and how they have been managing access to servers. But I won't, I would rather urge you to do basic trend search either on google or your favorite platform for SSH PAM via LDAP or SSH LDAP and see it for yourself where the world is heading [1]. [1] https://trends.google.com/trends/explore?date=today%205-y&q=ssh%20ldap https://trends.google.com/trends/explore?date=today%205-y&q=...
vbernat 4y ago
Use of SSH agent forwarding is dangerous as it allows an attacker to gain access to more key materials to access more servers. Using it casually in an article about SSH security is a bit worrying.
- pritambaral 4y ago
  Not with the confirm option of ssh-add. I've had agent forwarding on for every host (trusted and untrusted) for a decade now, without worry, because my ssh agent confirms with me each use of any ssh key.
  - remram 4y ago
    Interesting. However in practice, I don't ssh-add my keys, they get loaded on first use by the ssh client. Is there a way to make ssh load keys into the agent with that option set?
    - pritambaral 4y ago
      Yes: https://man.openbsd.org/ssh_config#AddKeysToAgent https://man.openbsd.org/ssh_config#AddKeysToAgent Source: I don't ssh-add my keys either.
fpanzer 4y ago
I'd rather patch a bunch of openssh servers than a bunch of proprietary software agents that essentially keep reverse ssh tunnels open for me. Or did I miss something?
debarshri 4y ago
I am a bit confused about this product. I had once seen a product called Runops [1], the customer list and product testimonials are exactly same as this product Hoop.dev [2]. [1] https://runops.io/ https://runops.io/ [2] https://hoop.dev/ https://hoop.dev/
- renewiltord 4y ago
  At this moment, 6 minutes after your post, the top of runops.io reads: > Announcing a new phase of Runops. Launching hoop.dev with a link to this https://hoop.dev/blog/launching-hoop/ https://hoop.dev/blog/launching-hoop/
_joel 4y ago
Umm what's wrong with ssh -AJ ?
- deleted 4y ago
  [deleted]
- larschdk 4y ago
  You need to fully trust your target server, so you need to manage your known_hosts diligently and make sure you trust the host you connect to. If you just accept the host key without checking, you allow any host to use your SSH key for authentication. Any SSH server can accept your private key as authentication. Also, if the target host is infiltrated, it can use your private SSH key for authentication elsewhere without your knowledge.
  - jesprenj 4y ago
    Why even use -A? Isn't -J enough?
    - _joel 4y ago
      Well if you want to pass your keys to the endpoint without keeping them on the jump hosts, yea, it's kinda useful.
      - deathanatos 4y ago
        You don't need -A with -J. You need -A if you're running ssh on the intermediate host, i.e., on the jump host in this case. But -J doesn't run ssh on the intermediate host, it more or less runs two ssh's on your local host. The first from local to the jump, the second from local to the eventual target, through a tunnel forwarding the connecting through the jump[1]. But because all the SSH processing is always local, it always has access to the local ssh-agent: you don't need -A. And, as someone points out upthread, you need to fully trust the remote machine to pass -A. You usually shouldn't, in most cases that I think people using jump hosts in corporate settings would be interacting with jump hosts: it permits other employees to impersonate you, by abusing your forwarded ssh-agent, if they have sufficient access on the jump host. [1]: -J in ssh(1) documents this
BohdanPetryshyn 4y ago
AWS Session Manager is great. However, to connect to an RDS/Aurora/Elasticache instance, you must still create an intermediate EC2 instance to run SSM commands against. We use Basti (https://github.com/BohdanPetryshyn/basti/issues https://github.com/BohdanPetryshyn/basti/issues) to set up and manage the jump host. The tool automatically starts/stops the instance, which is excellent for irregular access.
- migf 4y ago
  This is what I was looking for. It seems weird to always have a jump box sitting up, waiting for someone to come mess with it. Part of the tooling should be to spin up / boot an ephemeral instance.
- TheHappyOddish 4y ago
  It's probably also worth mentioning that you _build_ Basti as well as using it, otherwise you may come across as shilling. That said, thanks, Basti looks very useful!
ilyt 4y ago
> Jump Servers must be able to reach to a certain private network and this requires specific configuration for each environment; Yup. Use VPNs and firewall ACLs. Jump servers are leftovers of bad practices where good practices were too hard to implement. > Burden of managing SSH keys of users throughout all nodes. Rotation is required when someone leaves or enter the organization; That is extremely trivial if you have (you should) any sensible configuration management in place. We just store them in LDAP with user data and distribute where neede (gitlab, servers) > Role management requires managing sudoers files, making sure file system permissions are properly configured and users are within their proper groups; Ah yes, managing a text file, so fucking hard /s > Nodes must be updated with the tooling necessary to interact with internal services. >Keep a list of updated services (DNS) available to interact with it see the point about CM > Usually, infrastructure enginners are a scarce team and keeping all these components updated are hard to tackle. Over time, these nodes will onboard more users and tooling, which will increase the complexity over managing these resources. Which is why you write it once and use automation. I don't think we touched our sudoers or ssh key management module in years, it was written once then had some small changes but that's about it

100 points by andriosr 4y ago

Er, so most of those supposed problems sound like you could fix them by just using unprivileged users jumping through the jump server via `ssh -J`/ProxyJump? Or a VPN, which is pretty close to what it sounds like the product they're trying to sell is anyways. And if you're going to try and sell a Teleport competitor, you really should do a better job convincing me that it's going to actually be secure in the first place. I don't see source code or audits anywhere on this website.

POPOSYS 4y ago
But it is written on the website: "without security compromise" - so it must be secure!!! Also there is a lot of low contrast text on that website - this shows that they really know what they are doing and you can trust them. I am sure it is a very good company and you should give all the login credentials of your org to them and also install their binary on all servers. Isn´t "trust in company" the reason why you are using Linux servers?

SomaticPirate 4y ago

Google Cloud’s IAP tunneling has basically made this a non-issue. All access is logged and its extremely easy to get around without any public IPs. Also easy to port-forward instances to your local machine. I’m always surprised that other clouds don’t have this.

drowsspa 4y ago
Isn't it basically AWS SSM? you can even configure it in the .ssh/config

berkle4455 4y ago

AWS Session Manager (aws specific) or Tailscale (anything) has solved this problem for me

threeseed 4y ago
Jump hosts seem like an anti-pattern in the era of AWS SSM and Tailscale. It is far too easy to misconfigure network policies and grant them access to infrastructure that they shouldn't. And with Tailscale you can run the agent within SaaS products like Github Actions or Terraform Cloud to securely manage their access into your systems.
- theteapot 4y ago
  > anti-pattern Is wearing blue denim jacket and jeans an anti-pattern? Is vegemite an anti-pattern? Can't we just say "obsolete", or "bad idea"?
  - muppetman 4y ago
    You should "reach out" to him/her and sort this out. [I've always found "reach out" to be hilarious, (what, with my arms?) but it's so common now I'm the oddball]
    - theteapot 4y ago
      > I've always found "reach out" to be hilarious, (what, with my arms?) ... Using the expression "reach out" is such an anti-pattern.
    - deathanatos 4y ago
      Oh, the one that kills me now is "let's take that offline" used in a meeting to refer to "let's move this to a Slack conversation". (Or, more cynically, "This problem is thorny and I wish to bury it rather than put in the time and work to fix it.")
  - kevsim 4y ago
    > Is wearing blue denim jacket and jeans an anti-pattern? The Canadian tuxedo is clearly a best practice.
  - dspillett 4y ago

phamilton 4y ago

I've spent the last 7 years using SSH to get onto a prod box a few dozen times. Less than once a month. When I do have to get on the box, we use signed keys that expire in 24 hours (a manager or SRE is required to sign the key if you need to get on a box). The article is talking about the scaling issues of SSH access via jump boxes or bastions. I'd argue that a better solution to scaling SSH access is to invest in tooling that makes SSH access unnecessary for most of your team. Centralized logs, error reporting, distributed traces, etc. are fairly well solved for most installations. Interactive access to prod (e.g. running DB migrations) requires a little more investment, but tools like ECS Exec make that fairly accessible without requiring SSH access.

vasco 4y ago
DB migrations should definitely not be done interactively and definitely not in an SSH session. Write them beforehand, have them reviewed with the rest of the code, and have your deployment process run them.
- PedroBatista 4y ago
  Agree, but in the real world that works 100% most of the time.
  - ranting-moth 4y ago
    100% most of the time. One of those rare cases when I laugh out loud at a HN comment. I'm definitely stealing this phrase!
- phamilton 4y ago
  Just to clarify our process: DB migrations are part of code review, they run as a script, but we don't have that script run automatically as part of CD. We've been bit by enough migration surprises that we require someone watching and able to interrupt and cancel the migration if needed. But that's the extent of action required. Run this command, Ctrl-C if necessary. Definitely not YOLO'ing in a psql in prod.

mbreese 4y ago

Even when using a jump server, I've solved the issue of revoking access in the past by using LDAP to control access to servers. Instead of adding user accounts directly to server, the account information is stored in LDAP -- including public SSH keys. If you want to revoke a user's access across the entire infrastructure, then you can do so in one fell swoop. I also have set this up to restrict SSH access to particular hosts based on the LDAP record. The problem they are describing isn't a problem with Jump servers... it's a problem with distributed authorization.

debarshri 4y ago
I haven't seen many organisations actually set SSH PAM via LDAP or another delegated authentication system. You are right about the problem statement. It is distributed authorization problem. And it is a very hard problem to solve or visualize for a fast moving company unless it is a problem.
- POPOSYS 4y ago
  Do you have some data about how many orgs actually set SSH PAM via LDAP? Where can I see this data? I would like to check your statements. We are on the internet, so would you please like to add the source of knowledge to your statements - it is a very basic and good feature called "URL", please use it! Or do you just want to say "I have not seen many orgs with SSH + LDAP in my career because I have never worked in one and from that I conclude that the whole world works like this"?
  - debarshri 4y ago
    I don't think neither me nor anyone reading that statement would come to a conclusion that the works the way I suggested in that argument. I can talk about me interviewing and empirical data of talking to dozens of companies from seed to series B and how they have been managing access to servers. But I won't, I would rather urge you to do basic trend search either on google or your favorite platform for SSH PAM via LDAP or SSH LDAP and see it for yourself where the world is heading [1]. [1] https://trends.google.com/trends/explore?date=today%205-y&q=ssh%20ldap https://trends.google.com/trends/explore?date=today%205-y&q=...

vbernat 4y ago

Use of SSH agent forwarding is dangerous as it allows an attacker to gain access to more key materials to access more servers. Using it casually in an article about SSH security is a bit worrying.

pritambaral 4y ago
Not with the confirm option of ssh-add. I've had agent forwarding on for every host (trusted and untrusted) for a decade now, without worry, because my ssh agent confirms with me each use of any ssh key.
- remram 4y ago
  Interesting. However in practice, I don't ssh-add my keys, they get loaded on first use by the ssh client. Is there a way to make ssh load keys into the agent with that option set?
  - pritambaral 4y ago
    Yes: https://man.openbsd.org/ssh_config#AddKeysToAgent https://man.openbsd.org/ssh_config#AddKeysToAgent Source: I don't ssh-add my keys either.

fpanzer 4y ago

I'd rather patch a bunch of openssh servers than a bunch of proprietary software agents that essentially keep reverse ssh tunnels open for me. Or did I miss something?

debarshri 4y ago

I am a bit confused about this product. I had once seen a product called Runops [1], the customer list and product testimonials are exactly same as this product Hoop.dev [2]. [1] https://runops.io/ https://runops.io/ [2] https://hoop.dev/ https://hoop.dev/

renewiltord 4y ago
At this moment, 6 minutes after your post, the top of runops.io reads: > Announcing a new phase of Runops. Launching hoop.dev with a link to this https://hoop.dev/blog/launching-hoop/ https://hoop.dev/blog/launching-hoop/

_joel 4y ago

Umm what's wrong with ssh -AJ ?

deleted 4y ago
[deleted]
larschdk 4y ago
You need to fully trust your target server, so you need to manage your known_hosts diligently and make sure you trust the host you connect to. If you just accept the host key without checking, you allow any host to use your SSH key for authentication. Any SSH server can accept your private key as authentication. Also, if the target host is infiltrated, it can use your private SSH key for authentication elsewhere without your knowledge.
- jesprenj 4y ago
  Why even use -A? Isn't -J enough?
  - _joel 4y ago
    Well if you want to pass your keys to the endpoint without keeping them on the jump hosts, yea, it's kinda useful.
    - deathanatos 4y ago
      You don't need -A with -J. You need -A if you're running ssh on the intermediate host, i.e., on the jump host in this case. But -J doesn't run ssh on the intermediate host, it more or less runs two ssh's on your local host. The first from local to the jump, the second from local to the eventual target, through a tunnel forwarding the connecting through the jump[1]. But because all the SSH processing is always local, it always has access to the local ssh-agent: you don't need -A. And, as someone points out upthread, you need to fully trust the remote machine to pass -A. You usually shouldn't, in most cases that I think people using jump hosts in corporate settings would be interacting with jump hosts: it permits other employees to impersonate you, by abusing your forwarded ssh-agent, if they have sufficient access on the jump host. [1]: -J in ssh(1) documents this

BohdanPetryshyn 4y ago

AWS Session Manager is great. However, to connect to an RDS/Aurora/Elasticache instance, you must still create an intermediate EC2 instance to run SSM commands against. We use Basti (https://github.com/BohdanPetryshyn/basti/issues https://github.com/BohdanPetryshyn/basti/issues) to set up and manage the jump host. The tool automatically starts/stops the instance, which is excellent for irregular access.

migf 4y ago
This is what I was looking for. It seems weird to always have a jump box sitting up, waiting for someone to come mess with it. Part of the tooling should be to spin up / boot an ephemeral instance.
TheHappyOddish 4y ago
It's probably also worth mentioning that you _build_ Basti as well as using it, otherwise you may come across as shilling. That said, thanks, Basti looks very useful!

ilyt 4y ago

> Jump Servers must be able to reach to a certain private network and this requires specific configuration for each environment; Yup. Use VPNs and firewall ACLs. Jump servers are leftovers of bad practices where good practices were too hard to implement. > Burden of managing SSH keys of users throughout all nodes. Rotation is required when someone leaves or enter the organization; That is extremely trivial if you have (you should) any sensible configuration management in place. We just store them in LDAP with user data and distribute where neede (gitlab, servers) > Role management requires managing sudoers files, making sure file system permissions are properly configured and users are within their proper groups; Ah yes, managing a text file, so fucking hard /s > Nodes must be updated with the tooling necessary to interact with internal services. >Keep a list of updated services (DNS) available to interact with it see the point about CM > Usually, infrastructure enginners are a scarce team and keeping all these components updated are hard to tackle. Over time, these nodes will onboard more users and tooling, which will increase the complexity over managing these resources. Which is why you write it once and use automation. I don't think we touched our sudoers or ssh key management module in years, it was written once then had some small changes but that's about it