6 ms·

- Scan containers and Pods for vulnerabilities or misconfigurations. - Run containers and Pods with the least privileges possible. - Use network separation to

- Scan containers and Pods for vulnerabilities or misconfigurations. - Run containers and Pods with the least privileges possible. - Use network separation to control the amount of damage a compromise can cause. - Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality. - Use strong authentication and authorization to limit user and administrator access as well as to limit the attack surface. - Use log auditing so that administrators can monitor activity and be alerted to potential malicious activity. - Periodically review all Kubernetes settings and use vulnerability scans to help ensure risks are appropriately accounted for and security patches are applied.

haolez 5y ago
Who scans the vulnerability scanners? Genuine question. How does the community/ecosystem solve this problem of auditability?
- tinco 5y ago
  If your threat profile says you need to audit your vulnerability scanners, you audit your vulnerability scanners. There's not really a problem there right?
  - haolez 5y ago
    I've never had to. I wanted feedback from people who have.
  - Pokepokalypse 5y ago
    NIST also says: if your scanner finds a vulnerability, it's up to you to VALIDATE that it's not a false-positive. False-positives abound on these scanners.
- knownjorbist 5y ago
  For anyone who hasn't read it: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
  - Zababa 5y ago
    Are there any people working seriously on this? I'm aware of efforts for OCaml (http://gallium.inria.fr/~scherer/drafts/camlboot.pdf http://gallium.inria.fr/~scherer/drafts/camlboot.pdf), but that's it.
    - LukeShu 5y ago
      https://dwheeler.com/trusting-trust/ https://dwheeler.com/trusting-trust/ 'dwheeler is now the Linux Foundation's Director of Open Source Supply Chain Security.
    - pabs3 5y ago
      The Bootstrappable Builds community (which camlboot is part of) are working on a lot of different efforts in this area. The main one is going from a small amount of machine code to an entire Linux distro, which is in-progress. https://bootstrappable.org/ https://bootstrappable.org/ https://bootstrapping.miraheze.org/ https://bootstrapping.miraheze.org/
    - kibwen 5y ago
      Here's the original resource on Diverse Double Compilation to counter Trusting Trust Attacks: https://dwheeler.com/trusting-trust/ https://dwheeler.com/trusting-trust/ Notably I know the Rust compiler has been verified in this way (or at least certain versions of it have been verified), but it shouldn't be hard to do the same for any language with multiple independent implementations.
- nonameiguess 5y ago
  We deal with this by having multiple vulnerability scanners. Product A and Product B both scan your active environment. Product A scans Product B. Product B scans Product A. Additionally, make the vendors of those products sign NDAs so your threat actors, other than insiders, don't necessarily even know who they are. An attacker then needs to not only compromise both, but figure out who they are in the first place.
  - herodoturtle 5y ago
    To this I'd add what is colloquialy referred to as a "Chinese wall", so that even insiders aren't aware of the full picture.
- jhawk28 5y ago
  that was the issue in the solar winds hack: https://www.npr.org/2021/04/16/985439655/a-worst-nightmare-cyberattack-the-untold-story-of-the-solarwinds-hack https://www.npr.org/2021/04/16/985439655/a-worst-nightmare-c...
throwaway984393 5y ago
> and encryption to protect confidentiality Probably the hardest part about this. Private networks with private domains. Who runs the private CA, updates DNS records, issues certs, revokes, keeps the keys secure, embeds the keychain in every container's cert stack, and enforces validation? That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete. How many teams are actually going to do this, versus just terminating at the load balancer and everything in the cluster running plaintext?
- devy 5y ago
  > That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete. Agree! This is why that "Kubernetes Hardening Guidance" is for NSA, not for startups. Resource needs aside, keeping basic AppSec/InfoSec hygiene is a strong recommendation. Also there are tons of startups that are trying to provide solutions/services to solve that also. A lot of times, it's worth the money.
  - throwaway984393 5y ago
    I wonder about that. What are the attack vectors within a K8s cluster to necessitate inter-cluster transport encryption?
    - smarterclayton 5y ago
      Most (but not all) overlay networks are implemented in kernel. If you compromise one node in a cluster, you can fairly trivially snoop traffic, bias other nodes to send traffic through you, or listen via various mechanisms such that you can intercept traffic flowing between workloads not actually located on the compromised node. So always encrypt everything unless you’re in a very rare environment with central network control that cannot be compromised or intercepted from a given machine.
      - killjoywashere 5y ago
sound1 5y ago
I am mostly non technical person but why do we need to resort to firewalls etc. if we can employ UNIX like file permission system for network access? Wouldn't it be awesome if we can allow any installed software to contact ONLY whitelisted domains? Of course this excludes web browsers but you get the idea. How about our mainstream OSes incorporate that kind of permission system similar to what we have in mobile OSes already have today?
- Meandering 5y ago
  SELinux
  - sound1 5y ago
    I didn't know that, learnt somthing today, Thank You! Again, as a non technical person, why a software needs access to entire internet instead of whitelisted domains specific to its requirements is beyond me, since we already know how UNIX permission system works. Is it so hard to extend that to networks? Especially since everything is file in UNIX? Kindly pardon my ignorance :-)
    - Meandering 5y ago
      You are right. Software doesn't need access to everything and it shouldn't. Unfortunately, it is easier on the consumer end to leave software access somewhat "open ended". The domain for updates may change or it may need to connect to different plugin sources. Unnecessary constrictions on a software's ability to function would fuel software issues. So, more sensitive networks will have administrators define these permissions. However, providing constrictive defaults to a regular consumer wouldn't be worth the customer service burden.
- remram 5y ago
  You're describing a firewall? How could it be more "UNIX-like"?
- zeroxfe 5y ago
  Think of them as a defence-in-depth that protect from accidental misconfiguration, software bugs, local exploits, etc.
gizdan 5y ago
> - Use network separation to control the amount of damage a compromise can cause. > - Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality. Are we still on this? Why isn't anyone pushing for zero trust? A concept made significantly easier to achieve thanks to container orchestration.
graderjs 5y ago
I do all those things in the pro version of my RBI (remote browser isolation) product, but i don't use k8s. - Scan for vulns and misconfigs: I regularly update the underlying distro images, and use security scanning software to monitor dependencies, and regularly update them. - Run with least privilege: I create a separate, temporary user account (no login, no shell) for each browser and service which has no elevated privileges, as well as run that browser and its service in a group and cgroup that restricts disk, bandwidth, CPU, and memory using block quotas, cgroups, tc, iptables and active monitoring and termination. - Use network separation to isolate: RBI is basically a network isolation layer between the client (where the human interacts), and the server (where the browser actually runs.) I also don't have any privileges (service accounts, SSH keys, trusted IPs) on any of the machines and they're all single tenant and run inside GCE. - Use firewalls to lock down connectivity + encryption: I use GCE firewall rules and iptables drop rules to block access to GCP metadata endpoints, as well as to other machines in the subnet. Also, every network request is encrypted (HTTP is https/TLS, WebSocket is wss/TLS, WebRTC is encrypted by default). - Use strong auth to limit user access: For running the processes I use temporary users. For persistent browser sessions I use persistent users (either system native, or in a DB, always with bcrypt salted hashed passwords). For SaaS and resource control I use high entropy random API keys between each service layer. But I could improve my game for keeping secrets out of private git repos and separating code and config, ideally automatically. I could also improve my game to limit administrator access (right now I just have a single role, with God power, but I should create an admin role with power limited to a project, ideally even on a per-customer level). - Use log autditing: I do this, but only manually, using various grepping and inspection of various logs, including last and lastb, as well as the service internal logs. This is likely something I could improve as well. - Review all k8s settings: I don't use k8s or docker, just run services in this custom sandbox on GCE instances. I see that as both a way to limit attack service and complexity as well as minimize some overheads for maintenance and performance. In the longer term these things are worth exploring. Thanks a lot for the TLDR. For more info on my RBI work check out https://github.com/i5ik/ViewFinder https://github.com/i5ik/ViewFinder

haolez 5y ago

Who scans the vulnerability scanners? Genuine question. How does the community/ecosystem solve this problem of auditability?

tinco 5y ago
If your threat profile says you need to audit your vulnerability scanners, you audit your vulnerability scanners. There's not really a problem there right?
- haolez 5y ago
  I've never had to. I wanted feedback from people who have.
- Pokepokalypse 5y ago
  NIST also says: if your scanner finds a vulnerability, it's up to you to VALIDATE that it's not a false-positive. False-positives abound on these scanners.
knownjorbist 5y ago
For anyone who hasn't read it: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
- Zababa 5y ago
  Are there any people working seriously on this? I'm aware of efforts for OCaml (http://gallium.inria.fr/~scherer/drafts/camlboot.pdf http://gallium.inria.fr/~scherer/drafts/camlboot.pdf), but that's it.
  - LukeShu 5y ago
    https://dwheeler.com/trusting-trust/ https://dwheeler.com/trusting-trust/ 'dwheeler is now the Linux Foundation's Director of Open Source Supply Chain Security.
  - pabs3 5y ago
    The Bootstrappable Builds community (which camlboot is part of) are working on a lot of different efforts in this area. The main one is going from a small amount of machine code to an entire Linux distro, which is in-progress. https://bootstrappable.org/ https://bootstrappable.org/ https://bootstrapping.miraheze.org/ https://bootstrapping.miraheze.org/
  - kibwen 5y ago
    Here's the original resource on Diverse Double Compilation to counter Trusting Trust Attacks: https://dwheeler.com/trusting-trust/ https://dwheeler.com/trusting-trust/ Notably I know the Rust compiler has been verified in this way (or at least certain versions of it have been verified), but it shouldn't be hard to do the same for any language with multiple independent implementations.
nonameiguess 5y ago
We deal with this by having multiple vulnerability scanners. Product A and Product B both scan your active environment. Product A scans Product B. Product B scans Product A. Additionally, make the vendors of those products sign NDAs so your threat actors, other than insiders, don't necessarily even know who they are. An attacker then needs to not only compromise both, but figure out who they are in the first place.
- herodoturtle 5y ago
  To this I'd add what is colloquialy referred to as a "Chinese wall", so that even insiders aren't aware of the full picture.
jhawk28 5y ago
that was the issue in the solar winds hack: https://www.npr.org/2021/04/16/985439655/a-worst-nightmare-cyberattack-the-untold-story-of-the-solarwinds-hack https://www.npr.org/2021/04/16/985439655/a-worst-nightmare-c...

throwaway984393 5y ago

> and encryption to protect confidentiality Probably the hardest part about this. Private networks with private domains. Who runs the private CA, updates DNS records, issues certs, revokes, keeps the keys secure, embeds the keychain in every container's cert stack, and enforces validation? That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete. How many teams are actually going to do this, versus just terminating at the load balancer and everything in the cluster running plaintext?

devy 5y ago
> That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete. Agree! This is why that "Kubernetes Hardening Guidance" is for NSA, not for startups. Resource needs aside, keeping basic AppSec/InfoSec hygiene is a strong recommendation. Also there are tons of startups that are trying to provide solutions/services to solve that also. A lot of times, it's worth the money.
- throwaway984393 5y ago
  I wonder about that. What are the attack vectors within a K8s cluster to necessitate inter-cluster transport encryption?
  - smarterclayton 5y ago
    Most (but not all) overlay networks are implemented in kernel. If you compromise one node in a cluster, you can fairly trivially snoop traffic, bias other nodes to send traffic through you, or listen via various mechanisms such that you can intercept traffic flowing between workloads not actually located on the compromised node. So always encrypt everything unless you’re in a very rare environment with central network control that cannot be compromised or intercepted from a given machine.
    - killjoywashere 5y ago

sound1 5y ago

I am mostly non technical person but why do we need to resort to firewalls etc. if we can employ UNIX like file permission system for network access? Wouldn't it be awesome if we can allow any installed software to contact ONLY whitelisted domains? Of course this excludes web browsers but you get the idea. How about our mainstream OSes incorporate that kind of permission system similar to what we have in mobile OSes already have today?

Meandering 5y ago
SELinux
- sound1 5y ago
  I didn't know that, learnt somthing today, Thank You! Again, as a non technical person, why a software needs access to entire internet instead of whitelisted domains specific to its requirements is beyond me, since we already know how UNIX permission system works. Is it so hard to extend that to networks? Especially since everything is file in UNIX? Kindly pardon my ignorance :-)
  - Meandering 5y ago
    You are right. Software doesn't need access to everything and it shouldn't. Unfortunately, it is easier on the consumer end to leave software access somewhat "open ended". The domain for updates may change or it may need to connect to different plugin sources. Unnecessary constrictions on a software's ability to function would fuel software issues. So, more sensitive networks will have administrators define these permissions. However, providing constrictive defaults to a regular consumer wouldn't be worth the customer service burden.
remram 5y ago
You're describing a firewall? How could it be more "UNIX-like"?
zeroxfe 5y ago
Think of them as a defence-in-depth that protect from accidental misconfiguration, software bugs, local exploits, etc.

gizdan 5y ago

> - Use network separation to control the amount of damage a compromise can cause. > - Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality. Are we still on this? Why isn't anyone pushing for zero trust? A concept made significantly easier to achieve thanks to container orchestration.

graderjs 5y ago

I do all those things in the pro version of my RBI (remote browser isolation) product, but i don't use k8s. - Scan for vulns and misconfigs: I regularly update the underlying distro images, and use security scanning software to monitor dependencies, and regularly update them. - Run with least privilege: I create a separate, temporary user account (no login, no shell) for each browser and service which has no elevated privileges, as well as run that browser and its service in a group and cgroup that restricts disk, bandwidth, CPU, and memory using block quotas, cgroups, tc, iptables and active monitoring and termination. - Use network separation to isolate: RBI is basically a network isolation layer between the client (where the human interacts), and the server (where the browser actually runs.) I also don't have any privileges (service accounts, SSH keys, trusted IPs) on any of the machines and they're all single tenant and run inside GCE. - Use firewalls to lock down connectivity + encryption: I use GCE firewall rules and iptables drop rules to block access to GCP metadata endpoints, as well as to other machines in the subnet. Also, every network request is encrypted (HTTP is https/TLS, WebSocket is wss/TLS, WebRTC is encrypted by default). - Use strong auth to limit user access: For running the processes I use temporary users. For persistent browser sessions I use persistent users (either system native, or in a DB, always with bcrypt salted hashed passwords). For SaaS and resource control I use high entropy random API keys between each service layer. But I could improve my game for keeping secrets out of private git repos and separating code and config, ideally automatically. I could also improve my game to limit administrator access (right now I just have a single role, with God power, but I should create an admin role with power limited to a project, ideally even on a per-customer level). - Use log autditing: I do this, but only manually, using various grepping and inspection of various logs, including last and lastb, as well as the service internal logs. This is likely something I could improve as well. - Review all k8s settings: I don't use k8s or docker, just run services in this custom sandbox on GCE instances. I see that as both a way to limit attack service and complexity as well as minimize some overheads for maintenance and performance. In the longer term these things are worth exploring. Thanks a lot for the TLDR. For more info on my RBI work check out https://github.com/i5ik/ViewFinder https://github.com/i5ik/ViewFinder