6 ms·

Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment wi

Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. In my view there are emerging best practices in this area. There are two ways to reduce this risk and both are controversial: 1. Force developers to only develop software using an SSH terminal (by first connecting to a developer VPN via 2FA and then sshing into their secure development environment where may use tools like tmux, vi, and their programming language of choice to get the job done). In this scheme copying source codes or security credentials to a developer laptop is considered a violation and becomes a fireable offense no questions asked. 2. Require all developers to run a private USB-bootable linux desktop shell which is known to be clean. In this case they remain free to utilize modern desktop editors and code emulators (such as android simulator). It's even possible to setup secure persistence in these environments so that the developer's browser configuration, network/vpn config, dotfiles, apt installs, etc are stored on an encrypted filesystem on USB device. The reason why a USB image is preferred is because it's annoying to ask a new employee to repartiation her personal harddrive. My suspicion is that as more of the tools developers need to rely on are cloud-based: (example: github, cloud9, jenkins, etc) we will eventually see these modern best practices against client-side attacks being adopted more broadly. The quality and reliability of hot-bootable ultra secure cloud operating systems has gone thru the roof over the past couple of years and I assume this trend will only accellerate due to the fact that Google Chrome OS continues to penerate more of the market and consumers are getting used to it. TLDR: ssh keys existing on developer harddrives is an info-sec anti-pattern, they should only ever exist in system memory or in an encrypted partition on a USB stick.

EvanPlaice 11y ago
3. Never grand direct SSH to access the the staging/production environment. Automate the build and deployment environment on a system that developers don't have direct access. Instead of pushing changes, the bot pulls a specified release branch, builds, tests, and deploys the code. All without human interaction. If malicious code were somehow introduced from a developer's environment, it would be recorded and reflected in the commit history. AFAIK, that's how GitHub manages deployments via HubBot. See https://www.youtube.com/watch?v=NST3u-GjjFw https://www.youtube.com/watch?v=NST3u-GjjFw. ----- To take things a step further, public-facing environments should be made immutable wherever possible. With the entire system being built and released as a whole. Docker alleviates some of the complexity and overhead but I think this space is where Unikernels have a lot of potential to shine. There's a very good talk about how the Wunderlist team used chaos and frequent destruction of their envronments to overcome fear and uncertainty here. https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop. Emphasis being placed on the the frequent disposal and recreation of environments rather than building long-running persistent environments. This setup probably won't work for long-lived systems (ex databases). In those cases, access via a transient environment like a USB bootable OS would be ideal to prevent persistent viruses/trojans. Ironically, the best current options are security-focused distros like KaliLinux that put a special emphasis on avoiding persistent state. Maybe one day soon we'll see an admin-focused OS that better fits this role.
- jvehent 11y ago
  > 3. Never grand direct SSH to access the the staging/production environment. Most complex systems fail in unexpected ways, and the best way to debug is to grant the devs access to production machines. It can be temporary, monitored and through a secure channel, but it's still something most organizations can hardly live without.
  - shawn-furyan 11y ago
    I think that this issue is potentially moot if you've transitioned to immutable deploys. Under such a system, arbitrage steps would tend to change from the standard have an SA/Dev ssh in and muck around until the issue is found. Instead, first step may be just redeploying (in case of transient/intermittent issues that are disrupting service), and then checking out the production system locally to do root cause analysis. So yeah, most organizations couldn't do this today, because they don't have the deploy process in place. But when an organization has made the switch, then it is much more realistic to assume that it can get along without granting ad hoc ssh access to production systems.
    - EvanPlaice 11y ago
      Exactly. It's also possible to capture stack traces remotely from a running system and/or after a crash. http://techblog.netflix.com/2015/12/debugging-nodejs-in-production.html http://techblog.netflix.com/2015/12/debugging-nodejs-in-prod.... Hopefully, this practice -- and the tools required to make it happen -- get better with time.
jvehent 11y ago
> Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. It's also the least likely to happen. Attackers will have broken deep into your database due to poor webapp security before getting a hold of your ssh private key. Not that SSH security doesn't matter. It's easy enough and carries enough risks that putting some effort into it is worthwhile.
- hackercomplex 11y ago
  I think that unfortunately you're mistaken. If all you do is sit and look at log files all day you can come off with this impression because you see so many scans go by, but that creates a bias in your thinking which doesn't line up with the facts. A motivated attacker will always tend to try the "low tech" approaches first to rule them out as a first step because it it doesn't require as much sophisticated technical attention and in addition it's a well known fact that human beings are more often than not the weakest link in the chain of security. It goes something like this: 1. enumerate the employees.. in other words build a list of all their work emails and try to obtain their personal emails and those of their spouses and children. 2. investigate the background of executives so that a compelling phishing email message can be crafted which looks like it originated from high up within the organization. 3. Send targeted phishing emails designed to bait people within the organization to visit webpages that exploit Adobe Flash attack vectors or other browser based vulnerabilities, or perhaps even lure them into installing a trojan directly. 4. after someone in the organization falls victim to the client-side attack, read their emails to learn more about the organization's structure so that your next phising attack can be more refined, specific, and compelling and can possibly utilize a real corporate email address. 5. Rinse and repeat until you eventually gain access to a developer laptop where you can grab production environment SSH keys. Now you own the network without even having to scan a single server and without leaving a trace in the log files.
  - jvehent 11y ago
    You're describing the workflow of an APT attack. Those are extremely costly, take several months to complete and definitely involve highly technical skills. I agree they are a real threat, but not to startups, to Fortune 500 companies. For startups, the #1 risk is a vulnerability in some web app, or a forgotten admin panel, that leaks the entire database to an attacker. Not a complex attack conducted by a nation-sponsored offensive team. But again, SSH security matters, you should take it seriously.
yeukhon 11y ago
I argue leaking SSH key is not the worst nightmare but definitely one of the top five on anyone's list. I am more worried about source code and database leaks through development cycle. Copied production data to dev environment and clone source code to your laptop. How does Google/Facebook handle this? I can't imagine anyone cloning their 30G or whatever size single-branch repository down to laptop (they may do shallow clone but doesn't matter).
superuser2 11y ago
(Disclaimer, unicorn employee.) I'm confused. Are you implying people write company code on personally owned laptops? That's insane! 1. Company walks you through setting up FileVault (with key escrow) and VPN client, generating and uploading SSH public key immediately after unboxing laptop. 2. OneLogin + Duo (or pick your SSO/2FA scheme) for everything - internal webapps, GMail, etc. 3. SSH keys managed by Puppet. 4. 2FA verification of SSH logins (using pam-interactive), through a bastion (provided .ssh file makes it transparent), with OSX's default SSH agent, with pretty much all of those best practices configured except smartcards/certs. 5. Engineers have SSH access only to utility/development boxes. You can deploy code to production through a webapp (identifying the commit ID) but you can't get a shell as your application's user, and certainly not root. 6. Webapps moving behind a VPN. In the rare case that you need to debug in production beyond what you can get from metrics/logs, you pair with a "blessed"/senior sysadmin type.
- hackercomplex 11y ago
  Thank you for this. I suppose I should have included the caveot that my rant applies mostly to early stage shops which are aren't ready to build out and ship a shiny new dev machine for every employee. Another problem for smaller startups is that if you're not careful almost everybody ends up being "blessed". And simply being a senior sysadmin type doesn't mean that you're automatically immune from all forms of social engineering attacks. I suppose your point is that even very early companies should opt for standardized hardware, I think you're right. I <3 Duo Security.
- fanf2 11y ago
  If you are using an ssh agent, it is a good idea to delete the keys when you lock the screen or sleep your laptop. On Mac OS X you can do this with Hammerspoon: http://fanf.livejournal.com/139925.html http://fanf.livejournal.com/139925.html

EvanPlaice 11y ago

3. Never grand direct SSH to access the the staging/production environment. Automate the build and deployment environment on a system that developers don't have direct access. Instead of pushing changes, the bot pulls a specified release branch, builds, tests, and deploys the code. All without human interaction. If malicious code were somehow introduced from a developer's environment, it would be recorded and reflected in the commit history. AFAIK, that's how GitHub manages deployments via HubBot. See https://www.youtube.com/watch?v=NST3u-GjjFw https://www.youtube.com/watch?v=NST3u-GjjFw. ----- To take things a step further, public-facing environments should be made immutable wherever possible. With the entire system being built and released as a whole. Docker alleviates some of the complexity and overhead but I think this space is where Unikernels have a lot of potential to shine. There's a very good talk about how the Wunderlist team used chaos and frequent destruction of their envronments to overcome fear and uncertainty here. https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop. Emphasis being placed on the the frequent disposal and recreation of environments rather than building long-running persistent environments. This setup probably won't work for long-lived systems (ex databases). In those cases, access via a transient environment like a USB bootable OS would be ideal to prevent persistent viruses/trojans. Ironically, the best current options are security-focused distros like KaliLinux that put a special emphasis on avoiding persistent state. Maybe one day soon we'll see an admin-focused OS that better fits this role.

jvehent 11y ago
> 3. Never grand direct SSH to access the the staging/production environment. Most complex systems fail in unexpected ways, and the best way to debug is to grant the devs access to production machines. It can be temporary, monitored and through a secure channel, but it's still something most organizations can hardly live without.
- shawn-furyan 11y ago
  I think that this issue is potentially moot if you've transitioned to immutable deploys. Under such a system, arbitrage steps would tend to change from the standard have an SA/Dev ssh in and muck around until the issue is found. Instead, first step may be just redeploying (in case of transient/intermittent issues that are disrupting service), and then checking out the production system locally to do root cause analysis. So yeah, most organizations couldn't do this today, because they don't have the deploy process in place. But when an organization has made the switch, then it is much more realistic to assume that it can get along without granting ad hoc ssh access to production systems.
  - EvanPlaice 11y ago
    Exactly. It's also possible to capture stack traces remotely from a running system and/or after a crash. http://techblog.netflix.com/2015/12/debugging-nodejs-in-production.html http://techblog.netflix.com/2015/12/debugging-nodejs-in-prod.... Hopefully, this practice -- and the tools required to make it happen -- get better with time.

jvehent 11y ago

> Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. It's also the least likely to happen. Attackers will have broken deep into your database due to poor webapp security before getting a hold of your ssh private key. Not that SSH security doesn't matter. It's easy enough and carries enough risks that putting some effort into it is worthwhile.

hackercomplex 11y ago
I think that unfortunately you're mistaken. If all you do is sit and look at log files all day you can come off with this impression because you see so many scans go by, but that creates a bias in your thinking which doesn't line up with the facts. A motivated attacker will always tend to try the "low tech" approaches first to rule them out as a first step because it it doesn't require as much sophisticated technical attention and in addition it's a well known fact that human beings are more often than not the weakest link in the chain of security. It goes something like this: 1. enumerate the employees.. in other words build a list of all their work emails and try to obtain their personal emails and those of their spouses and children. 2. investigate the background of executives so that a compelling phishing email message can be crafted which looks like it originated from high up within the organization. 3. Send targeted phishing emails designed to bait people within the organization to visit webpages that exploit Adobe Flash attack vectors or other browser based vulnerabilities, or perhaps even lure them into installing a trojan directly. 4. after someone in the organization falls victim to the client-side attack, read their emails to learn more about the organization's structure so that your next phising attack can be more refined, specific, and compelling and can possibly utilize a real corporate email address. 5. Rinse and repeat until you eventually gain access to a developer laptop where you can grab production environment SSH keys. Now you own the network without even having to scan a single server and without leaving a trace in the log files.
- jvehent 11y ago
  You're describing the workflow of an APT attack. Those are extremely costly, take several months to complete and definitely involve highly technical skills. I agree they are a real threat, but not to startups, to Fortune 500 companies. For startups, the #1 risk is a vulnerability in some web app, or a forgotten admin panel, that leaks the entire database to an attacker. Not a complex attack conducted by a nation-sponsored offensive team. But again, SSH security matters, you should take it seriously.

yeukhon 11y ago

I argue leaking SSH key is not the worst nightmare but definitely one of the top five on anyone's list. I am more worried about source code and database leaks through development cycle. Copied production data to dev environment and clone source code to your laptop. How does Google/Facebook handle this? I can't imagine anyone cloning their 30G or whatever size single-branch repository down to laptop (they may do shallow clone but doesn't matter).

superuser2 11y ago

(Disclaimer, unicorn employee.) I'm confused. Are you implying people write company code on personally owned laptops? That's insane! 1. Company walks you through setting up FileVault (with key escrow) and VPN client, generating and uploading SSH public key immediately after unboxing laptop. 2. OneLogin + Duo (or pick your SSO/2FA scheme) for everything - internal webapps, GMail, etc. 3. SSH keys managed by Puppet. 4. 2FA verification of SSH logins (using pam-interactive), through a bastion (provided .ssh file makes it transparent), with OSX's default SSH agent, with pretty much all of those best practices configured except smartcards/certs. 5. Engineers have SSH access only to utility/development boxes. You can deploy code to production through a webapp (identifying the commit ID) but you can't get a shell as your application's user, and certainly not root. 6. Webapps moving behind a VPN. In the rare case that you need to debug in production beyond what you can get from metrics/logs, you pair with a "blessed"/senior sysadmin type.

hackercomplex 11y ago
Thank you for this. I suppose I should have included the caveot that my rant applies mostly to early stage shops which are aren't ready to build out and ship a shiny new dev machine for every employee. Another problem for smaller startups is that if you're not careful almost everybody ends up being "blessed". And simply being a senior sysadmin type doesn't mean that you're automatically immune from all forms of social engineering attacks. I suppose your point is that even very early companies should opt for standardized hardware, I think you're right. I <3 Duo Security.
fanf2 11y ago
If you are using an ssh agent, it is a good idea to delete the keys when you lock the screen or sleep your laptop. On Mac OS X you can do this with Hammerspoon: http://fanf.livejournal.com/139925.html http://fanf.livejournal.com/139925.html