5 ms·

Learning from a Year of Security Breaches

327 points by arice 10y ago

tptacek 10y ago
This is the best security article I've read in a long time. If you're at a startup right now, drop most things and take a few minutes to read it carefully.
mikeyk 10y ago
Had the pleasure of working with Ryan when he was at FB--he's one of the best.
- atmosx 10y ago
  You were part 'Red Team' incidents[1]? I can only imagine the panicked sysadmins running around like crazy, but jokes apart his is the best way to train a team's incidence response I've seen. [1] https://medium.com/starting-up-security/red-teams-6faa8d95f602#.kfudlxkgf https://medium.com/starting-up-security/red-teams-6faa8d95f6...
rokosbasilisk 10y ago
Im surprised credential theft is still the lowest hanging fruit. I thought Banks seem to have solved alot of that.
- kevinr 10y ago
  Banks are in the business of managing financial risk for their customers, and they have enough money to eat a lot of risk before it becomes a problem for them. Other business models with less money in them do not have the same kind of resources.
user5994461 10y ago
> I wasn’t roped into a single intrusion this year at any companies with completely role driven environments where secrets were completely managed by a secret store. > This can either mean one of a few things: These environments don’t exist at all, there aren’t many of them, or they don’t see incidents that would warrant involving IR folks like myself. What are these secrets store? Do they exist?
- theluketaylor 10y ago
  There are a number of secret stores. Some more basic ones resemble password managers on steroids, with audit logs of who checked out what and when. Or you can go to a full HSM (hardware security module) that totally isolates secrets (keys) from secret users (actual users, application code etc). HSMs allow you to sign or encrypt without ever having the keys used. It's hard to accidentally leak a secret if you never had it in the first place.
- dsp1234 10y ago
  For example, Hashicorp Vault[0] [0] - https://www.vaultproject.io/intro/index.html https://www.vaultproject.io/intro/index.html
  - user5994461 10y ago
    Then people need secrets to access the secret store and you're back at square one ;)
    - lvh 10y ago
      There are plenty of things a secret store still buys you. - It knows how to encrypt and store secrets securely. Having one specialized application have an opinion on how to do that is much better than having a hundred ones that do it incidentally. The central one will be audited and monitored. The hundreds will invariably mess it up. - It tracks who accessed a secret and when. This is critical information for remediation and ongoing scope reduction. Knowing who accessed what, when gives you the context for why; all three tell you how to further reduce the authority that application has. - It can generate "minimal" credentials on-demand. I.e. a new key that only lets you access what you need and for a limited amount of time. - It can encrypt things on behalf of the requester, such that the requester never sees the key. That is good, because it can be one-way. It is also good because if a service is compromised, the compromise may be detected and remediated (access revoked) before all data is dumped and compromised. Having the secret store lets you do e.g. rate limiting and centralized monitoring, for example. - Secret stores can know how secrets are linked; making it easier to do revocation, and easier to determine the impact of a breach or misuse incident.
lvh 10y ago
This is solid advice. To illustrate a little based on my own experiences and goals this year: - Yes, centralized logging is the biggest thing. What you put into it matters; queriability matters; but nothing matters as much as having that centralized logging pipeline to begin with. Once you have that, you can start adding other relevant metadata, like host config states, API calls, et cetera. - Giving employees a budget to buy the device they want is probably a better idea than BYOD. Strong password policies still matter. If it's BYOD, you probably still want to bring the device into policy. That can include physical rules (only do work work on the VPN or from the office) and software ones (you can use any device you want but it has to be running our osqueryd or whatever). Unfortunately, visibility becomes a double edged sword: there are good legal and ethical reasons for not wanting to see everything on an employee's laptop. (Overall, I think BYOD is a bad idea for most companies.) - 2FA is pretty cool. It doesn't just solve the usual "bad/compromised password" model -- it also typically makes it a lot harder for employees to mismanage their credentials (e.g. re-use the same SSH keys and have their personal box be compromised). For some reason, having that around seems to remind developers that you can make users re-authenticate for important/unusual actions -- you don't just have to count on the ambient authority of a session cookie. - We'd all like to imagine that we're going to be attacked by space alien 0day ninjas. Realistically, the main vector is an employee (rogue or confused deputy). Trainings are boring and don't work. Signature-based detection gets outdated pretty quick. I've done a little work on faster analysis tools -- I'm hoping we get a lot better at unobtrusively protecting people from even spearphishing in the next few years. (The tools we're building at Latacora are ready to beat a lot of attacker tactics right now, but I think we have an arms race ahead of us. Boring domain generation algorithms still aren't detected by most organization, so there's not a lot of evolutionary pressure.) - I have no idea if we'll get better at quantifying metrics for debt and security risk. I did a little bit of research into this, and it's a wide open field. You can get decent high-level reports with a "DEFCON number", but most of these models are not sophisticated in the sense you'd expect actuarial tables to be. And that's what they should be! It's revenue-at-risk! Step one here is fortunately getting all of that data into that centralized logging pipeline, and security professionals seem to mostly agree that's what you do first, so hopefully we get better here.
acidbaseextract 10y ago
Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking at centralized logging stuff it seems like a rabbit hole of problems we're too small to be worrying about, stuff that's not shipping features on my app.
- theluketaylor 10y ago
  If you want a pretty prepackaged solution you could do a lot worse than splunk. They even offer it SaaS https://www.splunk.com/en_us/cloud.html https://www.splunk.com/en_us/cloud.html
- ben_jones 10y ago
  I'm in the same boat. Looking for recommendations on strong, sturdy, buckets, for bailing water.
- d4mi3n 10y ago
  TFA aside, centralized logging is super useful for debugging a variety of issues. There are a number of hosted options, and setting them up isn't too hard. It usually involves configuring you're application's log device to talk to the remote service, or configuring syslog on your app servers to forward logs to said service. See https://logentries.com/ https://logentries.com/ for an example
- lvh 10y ago
  You have a lot of decent options. You could do a lot worse than ELK. If you're on AWS, you can get hosted Elasticsearch. It comes out of the box with Logstash you can hook up to DynamoDB, and it also does Kibana out of the box. There are a number of other vendors; but there are decent reasons for keeping your logs as close as possible. CloudWatch works fine too. CloudWatch comes integrated with AWS services out of the box. It can be more annoying to get your logs into it than ELK (the latter seems overall more popular). Its alerting and the AWS CLI integration pretty slick, though. You should also go turn on CloudTrail right now. It lets you automatically log side-effectful API calls. It is not a replacement for a centralized logging pipeline, but it's great high-signal data to put into one. I appreciate that your complaint (totally valid!) was "this is a rabbit hole", and I just gave you two options, and that might not help your perception that it's a rabbit hole. If you find yourself paralyzed by choice, either choice is much better than deferring the choice! Just pick one. Heck, if you can't pick, let me help: pick AWS hosted Elasticsearch. A lot of people (also in the security space) like Splunk. I find it annoying to deploy (I've heard rsyslog-in-front-of-forwarders as a canonical deployment method for just ingesting syslog more than once because reasons) and overpriced. YMMV. Disclaimer: shameless plug! You're not the only one with your hair on fire. One of the first things we're doing for Latacora customers is setting up a centralized logging pipeline.
mag00 10y ago
Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!
- dsacco 10y ago
  Thank you for writing these. These blog posts are my go-to resources when my client companies want to learn more about what they can do to improve their security posture long term. It's a really great series.
- ThePhysicist 10y ago
  Thanks for writing this, really insightful! A question: What's your advice on how to store secrets on the server-side? Currently, I mainly use a seperate "secrets.yml" file that gets deployed via Ansible and is stored there encrypted using Ansible-Vault with a strong password. Is that a reasonable approach? What is your opinion about storing secrets in environment variables? It seems that some people advise this over storing them in files, but I have seen some cases where environment variables can be exposed to the web client as well.
  - mag00 10y ago
    I don't like the idea of keeping secrets in ENV and limiting it to config, though it's the kind of thing I'd ask other folks about myself to understand any tradeoffs. I see Kubernetes and other things supporting secrets in env variables so unsure how common it is. The big win is simply keeping secrets out of source code, out of an general engineer's copy/paste buffer, and with errors not going to a logging platform with single factor access. Your likelihood of a short term incident decreases dramatically. Especially if those secrets have well segmented access, (IE, not a single AWS key with `AdministratorAccess` everywhere).
    - tptacek 10y ago
jonstewart 10y ago
It's interesting to see press leaks highlighted here as a pattern for insider threat. I don't doubt the author that this is so for the limited scope of organizations considered (SFBA tech companies), but I've worked on several insider cases, had insight into many more, and it's almost always an employee or ex-employee, with an axe to grind, taking trade secret information to a new job at a competitor. In many instances, the competitor has no idea and is pissed when they find out. One piece of advice that I'd give out with such cases is to listen to your Spidey Sense. A lot of organizations will say, after the fact, "well... something didn't seem right with Bob...". If you sense something isn't right, prepare to secure evidence and analyze it. Don't put IT assets back into circulation if there's doubt, and don't sit on it.
bm98 10y ago
Ugh. A good and scary reminder of what's lurking around the corner for any of us at any time - including holidays and vacations (Linode's holiday attack last December comes to mind). IMHO, the emotional impact of breaches on the staff who respond to them is under-discussed. The author touches on it here: > The discovery of a root cause is an important milestone that dictates the emotional environment an incident will take place in, and whether it becomes unhealthy or not. > A grey cloud will hover over a team until a guiding root cause is discovered. This can make people bad to one another. I work very hard to avoid this toxicity with teams. I remember close calls when massive blame, panic, and resignations felt like they were just one tough conversation away.
coldcode 10y ago
Logging everything is a great idea, but only if you read the log data. Target installed a system to monitor for certain kinds of security hacks which wrote to their logs files. The logging was turned off due to a high number of warnings cluttering up the logs. Of course the logging was telling them they were being hacked which they ignored for months, leading to all sorts of business disasters.
weld 10y ago
Great article! For those interested in security debt and how it relates to startups, I wrote this in 2011: https://www.veracode.com/blog/2011/02/application-security-debt-and-application-interest-rates https://www.veracode.com/blog/2011/02/application-security-d... and presented it that year: https://www.youtube.com/watch?v=MKdiiXgvz_U https://www.youtube.com/watch?v=MKdiiXgvz_U This predates by a year the referenced security debt presentation which has much of the same material uncited.

327 points by arice 10y ago

This is the best security article I've read in a long time. If you're at a startup right now, drop most things and take a few minutes to read it carefully.

mikeyk 10y ago

Had the pleasure of working with Ryan when he was at FB--he's one of the best.

atmosx 10y ago
You were part 'Red Team' incidents[1]? I can only imagine the panicked sysadmins running around like crazy, but jokes apart his is the best way to train a team's incidence response I've seen. [1] https://medium.com/starting-up-security/red-teams-6faa8d95f602#.kfudlxkgf https://medium.com/starting-up-security/red-teams-6faa8d95f6...

rokosbasilisk 10y ago

Im surprised credential theft is still the lowest hanging fruit. I thought Banks seem to have solved alot of that.

kevinr 10y ago
Banks are in the business of managing financial risk for their customers, and they have enough money to eat a lot of risk before it becomes a problem for them. Other business models with less money in them do not have the same kind of resources.

user5994461 10y ago

> I wasn’t roped into a single intrusion this year at any companies with completely role driven environments where secrets were completely managed by a secret store. > This can either mean one of a few things: These environments don’t exist at all, there aren’t many of them, or they don’t see incidents that would warrant involving IR folks like myself. What are these secrets store? Do they exist?

theluketaylor 10y ago
There are a number of secret stores. Some more basic ones resemble password managers on steroids, with audit logs of who checked out what and when. Or you can go to a full HSM (hardware security module) that totally isolates secrets (keys) from secret users (actual users, application code etc). HSMs allow you to sign or encrypt without ever having the keys used. It's hard to accidentally leak a secret if you never had it in the first place.
dsp1234 10y ago
For example, Hashicorp Vault[0] [0] - https://www.vaultproject.io/intro/index.html https://www.vaultproject.io/intro/index.html
- user5994461 10y ago
  Then people need secrets to access the secret store and you're back at square one ;)
  - lvh 10y ago
    There are plenty of things a secret store still buys you. - It knows how to encrypt and store secrets securely. Having one specialized application have an opinion on how to do that is much better than having a hundred ones that do it incidentally. The central one will be audited and monitored. The hundreds will invariably mess it up. - It tracks who accessed a secret and when. This is critical information for remediation and ongoing scope reduction. Knowing who accessed what, when gives you the context for why; all three tell you how to further reduce the authority that application has. - It can generate "minimal" credentials on-demand. I.e. a new key that only lets you access what you need and for a limited amount of time. - It can encrypt things on behalf of the requester, such that the requester never sees the key. That is good, because it can be one-way. It is also good because if a service is compromised, the compromise may be detected and remediated (access revoked) before all data is dumped and compromised. Having the secret store lets you do e.g. rate limiting and centralized monitoring, for example. - Secret stores can know how secrets are linked; making it easier to do revocation, and easier to determine the impact of a breach or misuse incident.

lvh 10y ago

This is solid advice. To illustrate a little based on my own experiences and goals this year: - Yes, centralized logging is the biggest thing. What you put into it matters; queriability matters; but nothing matters as much as having that centralized logging pipeline to begin with. Once you have that, you can start adding other relevant metadata, like host config states, API calls, et cetera. - Giving employees a budget to buy the device they want is probably a better idea than BYOD. Strong password policies still matter. If it's BYOD, you probably still want to bring the device into policy. That can include physical rules (only do work work on the VPN or from the office) and software ones (you can use any device you want but it has to be running our osqueryd or whatever). Unfortunately, visibility becomes a double edged sword: there are good legal and ethical reasons for not wanting to see everything on an employee's laptop. (Overall, I think BYOD is a bad idea for most companies.) - 2FA is pretty cool. It doesn't just solve the usual "bad/compromised password" model -- it also typically makes it a lot harder for employees to mismanage their credentials (e.g. re-use the same SSH keys and have their personal box be compromised). For some reason, having that around seems to remind developers that you can make users re-authenticate for important/unusual actions -- you don't just have to count on the ambient authority of a session cookie. - We'd all like to imagine that we're going to be attacked by space alien 0day ninjas. Realistically, the main vector is an employee (rogue or confused deputy). Trainings are boring and don't work. Signature-based detection gets outdated pretty quick. I've done a little work on faster analysis tools -- I'm hoping we get a lot better at unobtrusively protecting people from even spearphishing in the next few years. (The tools we're building at Latacora are ready to beat a lot of attacker tactics right now, but I think we have an arms race ahead of us. Boring domain generation algorithms still aren't detected by most organization, so there's not a lot of evolutionary pressure.) - I have no idea if we'll get better at quantifying metrics for debt and security risk. I did a little bit of research into this, and it's a wide open field. You can get decent high-level reports with a "DEFCON number", but most of these models are not sophisticated in the sense you'd expect actuarial tables to be. And that's what they should be! It's revenue-at-risk! Step one here is fortunately getting all of that data into that centralized logging pipeline, and security professionals seem to mostly agree that's what you do first, so hopefully we get better here.

acidbaseextract 10y ago

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking at centralized logging stuff it seems like a rabbit hole of problems we're too small to be worrying about, stuff that's not shipping features on my app.

theluketaylor 10y ago
If you want a pretty prepackaged solution you could do a lot worse than splunk. They even offer it SaaS https://www.splunk.com/en_us/cloud.html https://www.splunk.com/en_us/cloud.html
ben_jones 10y ago
I'm in the same boat. Looking for recommendations on strong, sturdy, buckets, for bailing water.
d4mi3n 10y ago
TFA aside, centralized logging is super useful for debugging a variety of issues. There are a number of hosted options, and setting them up isn't too hard. It usually involves configuring you're application's log device to talk to the remote service, or configuring syslog on your app servers to forward logs to said service. See https://logentries.com/ https://logentries.com/ for an example
lvh 10y ago
You have a lot of decent options. You could do a lot worse than ELK. If you're on AWS, you can get hosted Elasticsearch. It comes out of the box with Logstash you can hook up to DynamoDB, and it also does Kibana out of the box. There are a number of other vendors; but there are decent reasons for keeping your logs as close as possible. CloudWatch works fine too. CloudWatch comes integrated with AWS services out of the box. It can be more annoying to get your logs into it than ELK (the latter seems overall more popular). Its alerting and the AWS CLI integration pretty slick, though. You should also go turn on CloudTrail right now. It lets you automatically log side-effectful API calls. It is not a replacement for a centralized logging pipeline, but it's great high-signal data to put into one. I appreciate that your complaint (totally valid!) was "this is a rabbit hole", and I just gave you two options, and that might not help your perception that it's a rabbit hole. If you find yourself paralyzed by choice, either choice is much better than deferring the choice! Just pick one. Heck, if you can't pick, let me help: pick AWS hosted Elasticsearch. A lot of people (also in the security space) like Splunk. I find it annoying to deploy (I've heard rsyslog-in-front-of-forwarders as a canonical deployment method for just ingesting syslog more than once because reasons) and overpriced. YMMV. Disclaimer: shameless plug! You're not the only one with your hair on fire. One of the first things we're doing for Latacora customers is setting up a centralized logging pipeline.

mag00 10y ago

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

dsacco 10y ago
Thank you for writing these. These blog posts are my go-to resources when my client companies want to learn more about what they can do to improve their security posture long term. It's a really great series.
ThePhysicist 10y ago
Thanks for writing this, really insightful! A question: What's your advice on how to store secrets on the server-side? Currently, I mainly use a seperate "secrets.yml" file that gets deployed via Ansible and is stored there encrypted using Ansible-Vault with a strong password. Is that a reasonable approach? What is your opinion about storing secrets in environment variables? It seems that some people advise this over storing them in files, but I have seen some cases where environment variables can be exposed to the web client as well.
- mag00 10y ago
  I don't like the idea of keeping secrets in ENV and limiting it to config, though it's the kind of thing I'd ask other folks about myself to understand any tradeoffs. I see Kubernetes and other things supporting secrets in env variables so unsure how common it is. The big win is simply keeping secrets out of source code, out of an general engineer's copy/paste buffer, and with errors not going to a logging platform with single factor access. Your likelihood of a short term incident decreases dramatically. Especially if those secrets have well segmented access, (IE, not a single AWS key with `AdministratorAccess` everywhere).
  - tptacek 10y ago

jonstewart 10y ago

It's interesting to see press leaks highlighted here as a pattern for insider threat. I don't doubt the author that this is so for the limited scope of organizations considered (SFBA tech companies), but I've worked on several insider cases, had insight into many more, and it's almost always an employee or ex-employee, with an axe to grind, taking trade secret information to a new job at a competitor. In many instances, the competitor has no idea and is pissed when they find out. One piece of advice that I'd give out with such cases is to listen to your Spidey Sense. A lot of organizations will say, after the fact, "well... something didn't seem right with Bob...". If you sense something isn't right, prepare to secure evidence and analyze it. Don't put IT assets back into circulation if there's doubt, and don't sit on it.

bm98 10y ago

Ugh. A good and scary reminder of what's lurking around the corner for any of us at any time - including holidays and vacations (Linode's holiday attack last December comes to mind). IMHO, the emotional impact of breaches on the staff who respond to them is under-discussed. The author touches on it here: > The discovery of a root cause is an important milestone that dictates the emotional environment an incident will take place in, and whether it becomes unhealthy or not. > A grey cloud will hover over a team until a guiding root cause is discovered. This can make people bad to one another. I work very hard to avoid this toxicity with teams. I remember close calls when massive blame, panic, and resignations felt like they were just one tough conversation away.

coldcode 10y ago

Logging everything is a great idea, but only if you read the log data. Target installed a system to monitor for certain kinds of security hacks which wrote to their logs files. The logging was turned off due to a high number of warnings cluttering up the logs. Of course the logging was telling them they were being hacked which they ignored for months, leading to all sorts of business disasters.

weld 10y ago

Great article! For those interested in security debt and how it relates to startups, I wrote this in 2011: https://www.veracode.com/blog/2011/02/application-security-debt-and-application-interest-rates https://www.veracode.com/blog/2011/02/application-security-d... and presented it that year: https://www.youtube.com/watch?v=MKdiiXgvz_U https://www.youtube.com/watch?v=MKdiiXgvz_U This predates by a year the referenced security debt presentation which has much of the same material uncited.