6 ms·

Other than fixing the Django source code , is there any OS level mitigation techniques that can detect and prevent such security vulnerabilities? I am thinki

by srcmap 8y ago

Other than fixing the Django source code , is there any OS level mitigation techniques that can detect and prevent such security vulnerabilities? I am thinking something like selinux, docker or chroot - a bit like internal firewall for Django (or any other webapp). Any suggestions on the links to latest best practices?

lmm 8y ago
There's nothing to be done at the OS level, because the OS doesn't know which things are secret and which things aren't. You can sandbox the Django process (and to a certain extent Facebook were doing this at an even higher level, by having the whole server on a separate VLAN that was (supposedly) away from the important stuff) but that only does so much because the process needs to have enough access to do whatever it was intended to do. Ultimately the only thing you can do is not run debug mode in production, not use insecure serialization formats and not leak secret keys. If you want to be a bit more pre-emptive about it, avoid language-specific serialization formats especially in dynamic languages ("serialization" that executes arbitrary code seems more common in those) and use taint checking or, better, a type system that can avoid leaking secrets (you likely need rank-2 types to be able to have values that you can use in certain contexts but never access outside them, a la http://okmij.org/ftp/Computation/resource-aware-prog/region-io.pdf http://okmij.org/ftp/Computation/resource-aware-prog/region-... ). It sounds like Django already has some level of taint-like functionality but this plugin/addon (sentry) then didn't use it properly. So, multiple failures interacting to create this vulnerability - I guess we should take that as a sign of progress compared to the days of basic buffer overflows?
- srcmap 8y ago
  I am hoping a security subsystem that detects call to external programs (such as sleep in this case) , log them and raise alarm if not in a whitelist.
  - syntheticcdo 8y ago
    ThreatStack definitely does this, utilizing the OS-exposed auditing frameworks. There are probably open source alternatives as well, but I rely extensively on TS for server-level threat detection.
  - AgentME 8y ago
    You don't need to make calls to external programs to make good use of an RCE vulnerability like this.
Someone1234 8y ago
Depends on your threat model. For all we know chroot, docker, and SELinux could have been in active usage on this machine. But Facebook may view a compromise on their edge network and potentially one of their trusted servers as serious even if actual damage on that server itself is limited.
jrowley 8y ago
To be clear, this isn't an issue with the Django source, as much it was a miss configured server - having debug mode on enabled the stack trace to be leaked which yielded the secret key. Debug mode shouldn't be used in production, and django probably shouldn't be responsible for snipping every possible value out of debug traces.
jrockway 8y ago
It seems unlikely. The application has a feature to use a secret key to secure everything. The same application then prints out the secret key to anyone that asks. Your OS can't do much about that. Maybe if you tell your OS "never let any data containing this string leave the machine" you can kind of mitigate this, but it's unlikely to work. The HTTP response is probably compressed. Someone probably base64-encoded the thing and stuck it inside some JSON, which is then base64-encoded again. Ultimately, it's about managing complexity. Django and the extension punted: Django says it will print anything and everything it has access to, and the extension has a documentation caveat about how bad that would be. One could imagine an API that tries to be resistant to this sort of thing; when the extension is initialized, it deletes the key from the environment dictionary (only partially possible), stores it in a private attribute, and only provides public EncryptAndSignCookie and DecryptAndVerifyCookie methods. This will be better than a documentation caveat, maybe, but the truly ambitious debug mode will probably get the key and print it out. (I would also point out that I'm not a fan of storing state in encrypted+signed cookies, if only because there is no way to revoke a stolen cookie without revoking every cookie ever. If you have the state on the server to store a revocation list, you might as well just store everything there and never have this problem.)
C14L 8y ago
Doesn't seem to be a Django problem, as the author says > wow, it looks like it’s a sort of Django SECRET-KEY override! Sentry uses its "own" secret key, so Django doesn't know it should be stripped from the stacktrace. A simple `DEBUG = False` in `settings.py` would fix the bug. You don't have Django run in debug mode in production.
tptacek 8y ago
Of course. You can lock the process down so that it can't make unexpected system calls. If you deploy in a modern container environment, you can also use container networking to drastically limit what the application environment can talk to on the network. Though it's a less potent mitigation than seccomp and container isolation (and one you get for free once you deploy in a container), you can also limit filesystem access. If you run the application under a non-privileged uid, these mitigations combined can raise the bar somewhat for privilege escalation and pivoting after compromise. Of course, in a sense, Facebook appears to have accomplished something simpler simply by sacrificing an instance to this application and putting it on a lonely isolated VLAN. The other responses to your question are pretty weird, since it's obvious that there are things you can do to mitigate the possibility of your Django program literally calling execve or whatever.
- hyperpape 8y ago
  It seems like several people are discussing whether you can use the OS to prevent the sentry instance from disclosing its own data. You're talking about using the OS to prevent pivoting from that sentry instance to compromise something else. That said, I'd think it's pretty obvious that you can only address the second part, so it's what's interesting. If your application speaks HTTP, then the OS can't do that much to keep it from doing stupid things over HTTP.
  - tptacek 8y ago
    Ah, if that's the case, this thread makes more sense. Thanks!
- GordonS 8y ago
  > You can lock the process down so that it can't make unexpected system calls Huh, this isn't something I've ever come across before. Off the top of my head, I guess it would be possible on Windows using a kernel mode driver, but that's pretty hardcore, and really easy to get wrong. I know you can easily audit syscalls on Linux with auditd, but haven't seen preventing them before. Is this an option on both Linux and Windows, and is it commonly used? Interested to know more!

by srcmap 8y ago

lmm 8y ago

There's nothing to be done at the OS level, because the OS doesn't know which things are secret and which things aren't. You can sandbox the Django process (and to a certain extent Facebook were doing this at an even higher level, by having the whole server on a separate VLAN that was (supposedly) away from the important stuff) but that only does so much because the process needs to have enough access to do whatever it was intended to do. Ultimately the only thing you can do is not run debug mode in production, not use insecure serialization formats and not leak secret keys. If you want to be a bit more pre-emptive about it, avoid language-specific serialization formats especially in dynamic languages ("serialization" that executes arbitrary code seems more common in those) and use taint checking or, better, a type system that can avoid leaking secrets (you likely need rank-2 types to be able to have values that you can use in certain contexts but never access outside them, a la http://okmij.org/ftp/Computation/resource-aware-prog/region-io.pdf http://okmij.org/ftp/Computation/resource-aware-prog/region-... ). It sounds like Django already has some level of taint-like functionality but this plugin/addon (sentry) then didn't use it properly. So, multiple failures interacting to create this vulnerability - I guess we should take that as a sign of progress compared to the days of basic buffer overflows?

srcmap 8y ago
I am hoping a security subsystem that detects call to external programs (such as sleep in this case) , log them and raise alarm if not in a whitelist.
- syntheticcdo 8y ago
  ThreatStack definitely does this, utilizing the OS-exposed auditing frameworks. There are probably open source alternatives as well, but I rely extensively on TS for server-level threat detection.
- AgentME 8y ago
  You don't need to make calls to external programs to make good use of an RCE vulnerability like this.

Someone1234 8y ago

Depends on your threat model. For all we know chroot, docker, and SELinux could have been in active usage on this machine. But Facebook may view a compromise on their edge network and potentially one of their trusted servers as serious even if actual damage on that server itself is limited.

jrowley 8y ago

To be clear, this isn't an issue with the Django source, as much it was a miss configured server - having debug mode on enabled the stack trace to be leaked which yielded the secret key. Debug mode shouldn't be used in production, and django probably shouldn't be responsible for snipping every possible value out of debug traces.

jrockway 8y ago

It seems unlikely. The application has a feature to use a secret key to secure everything. The same application then prints out the secret key to anyone that asks. Your OS can't do much about that. Maybe if you tell your OS "never let any data containing this string leave the machine" you can kind of mitigate this, but it's unlikely to work. The HTTP response is probably compressed. Someone probably base64-encoded the thing and stuck it inside some JSON, which is then base64-encoded again. Ultimately, it's about managing complexity. Django and the extension punted: Django says it will print anything and everything it has access to, and the extension has a documentation caveat about how bad that would be. One could imagine an API that tries to be resistant to this sort of thing; when the extension is initialized, it deletes the key from the environment dictionary (only partially possible), stores it in a private attribute, and only provides public EncryptAndSignCookie and DecryptAndVerifyCookie methods. This will be better than a documentation caveat, maybe, but the truly ambitious debug mode will probably get the key and print it out. (I would also point out that I'm not a fan of storing state in encrypted+signed cookies, if only because there is no way to revoke a stolen cookie without revoking every cookie ever. If you have the state on the server to store a revocation list, you might as well just store everything there and never have this problem.)

C14L 8y ago

Doesn't seem to be a Django problem, as the author says > wow, it looks like it’s a sort of Django SECRET-KEY override! Sentry uses its "own" secret key, so Django doesn't know it should be stripped from the stacktrace. A simple `DEBUG = False` in `settings.py` would fix the bug. You don't have Django run in debug mode in production.

tptacek 8y ago

Of course. You can lock the process down so that it can't make unexpected system calls. If you deploy in a modern container environment, you can also use container networking to drastically limit what the application environment can talk to on the network. Though it's a less potent mitigation than seccomp and container isolation (and one you get for free once you deploy in a container), you can also limit filesystem access. If you run the application under a non-privileged uid, these mitigations combined can raise the bar somewhat for privilege escalation and pivoting after compromise. Of course, in a sense, Facebook appears to have accomplished something simpler simply by sacrificing an instance to this application and putting it on a lonely isolated VLAN. The other responses to your question are pretty weird, since it's obvious that there are things you can do to mitigate the possibility of your Django program literally calling execve or whatever.

hyperpape 8y ago
It seems like several people are discussing whether you can use the OS to prevent the sentry instance from disclosing its own data. You're talking about using the OS to prevent pivoting from that sentry instance to compromise something else. That said, I'd think it's pretty obvious that you can only address the second part, so it's what's interesting. If your application speaks HTTP, then the OS can't do that much to keep it from doing stupid things over HTTP.
- tptacek 8y ago
  Ah, if that's the case, this thread makes more sense. Thanks!
GordonS 8y ago
> You can lock the process down so that it can't make unexpected system calls Huh, this isn't something I've ever come across before. Off the top of my head, I guess it would be possible on Windows using a kernel mode driver, but that's pretty hardcore, and really easy to get wrong. I know you can easily audit syscalls on Linux with auditd, but haven't seen preventing them before. Is this an option on both Linux and Windows, and is it commonly used? Interested to know more!