5 ms·

Matrix.org Security Incident

74 points by Perceptes 7y ago

arunc 7y ago
Why was Jenkins running on a production server?
- deepwell 7y ago
  it was not, read again
  - Perceptes 7y ago
    But it does seem to be the case that the same SSH key pair that was used to access Jenkins also provided access to the production infrastructure. Unless I'm misunderstanding the nature of the attack.
    - zigara 7y ago
      It seems the issue was developers using SSH agent forwarding which was abused to access the production environment.
iancarroll 7y ago
Did the blog get hacked (again?) in between this being posted and now? It has what looks like password hashes and `uname -a` from every(?) server in their infrastructure. This is about as bad as IR can get: you realize you got hacked, you re-build your entire infrastructure and publicly say it's fixed, and then you get popped again...
- snailmailman 7y ago
  i think so. here is a wayback machine link to the previous blog post https://web.archive.org/web/20190412000400/https://matrix.org/blog/2019/04/11/security-incident/ https://web.archive.org/web/20190412000400/https://matrix.or... it stated "Having fully flushed out the attacker [...]" which i guess turned out to be false :-/ also im getting invalid HTTPS certs on the blog now. for some reason im getting a cert that looks like its for github.com ? edit: now im getting a lets encrypt cert on matrix.org, but a cloudflare SSL error page when i go to www.matrix.org ? the lets encrypt cert looks like it was just issued about an hour ago. edit2: i guess both with and without www. are lets encrypt, but the with www. cert was issued back in february (and gives a cloudflare SSL error page), while without www. was issued today. (and gives the current hacked message)
  - deepwell 7y ago
    Here is the message of the hacker, after the matrix.org admins allegedly cleaned everything up and wrote the original blog post: https://web.archive.org/web/20190412055614/https://matrix.org/blog/2019/04/11/security-incident/ https://web.archive.org/web/20190412055614/https://matrix.or...
ge0rg 7y ago
The most favorable reading of the current defacement page is that the attackers still controls the DNS, but no other parts of the infrastructure. Otherwise, the page probably wouldn't run off github.
localhostdotdev 7y ago
content before it gets fixed: Time for actual transparency. [list of servers, uname -a for each] root@[name]:/var/lib/postgresql# df -h [list of partitions] $ cat users.txt | grep [name] | head -n1 @[name]:matrix.org|[hash] $ wc -l users.txt [~6M users] See you soon. (affects whole site, even https://matrix.org https://matrix.org, site is on jekyll BTW)
- l2dy 7y ago
  Here's the source of that page: https://github.com/matrixnotorg/matrixnotorg.github.io/blob/master/blog/2019/04/11/security-incident/index.html https://github.com/matrixnotorg/matrixnotorg.github.io/blob/....
  - localhostdotdev 7y ago
    account got banned :) (it's on archive.org though)
mkj 7y ago
> As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history That seems like a fairly bad usability/security design?
zigara 7y ago
The attacker seems to have responded: https://github.com/matrix-org/matrix.org/issues/357 https://github.com/matrix-org/matrix.org/issues/357 edit: just saw the rest: https://github.com/matrix-org/matrix.org/issues?utf8=%E2%9C%93&q=is%3Aissue+SECURITY https://github.com/matrix-org/matrix.org/issues?utf8=%E2%9C%... "[SECURITY] SSH Agent Forwarding I noticed in your blog post that you were talking about doing a postmortem and steps you need to take. As someone who is intimately familiar with your entire infrastructure, I thought I could help you out. Complete compromise could have been avoided if developers were prohibited from using ForwardAgent yes or not using -A in their SSH commands. The flaws with agent forwarding are well documented."
- deleted 7y ago
  [deleted]
irgeek 7y ago
Assuming the GitHub issues are from the actual attacker -- and I see no reason to doubt they are -- this is very troubling: https://github.com/matrix-org/matrix.org/issues/363 https://github.com/matrix-org/matrix.org/issues/363 Compromise began well over a month ago Yikes. That's a long time for a compromise to go unnoticed.
- ygjb 7y ago
  Not really, the average in the industry seems to be floating between 70 and 400 days depending on the source of your stats on the topic (different vendors and reports use different stats for this)

74 points by Perceptes 7y ago

arunc 7y ago

Why was Jenkins running on a production server?

deepwell 7y ago
it was not, read again
- Perceptes 7y ago
  But it does seem to be the case that the same SSH key pair that was used to access Jenkins also provided access to the production infrastructure. Unless I'm misunderstanding the nature of the attack.
  - zigara 7y ago
    It seems the issue was developers using SSH agent forwarding which was abused to access the production environment.

iancarroll 7y ago

Did the blog get hacked (again?) in between this being posted and now? It has what looks like password hashes and `uname -a` from every(?) server in their infrastructure. This is about as bad as IR can get: you realize you got hacked, you re-build your entire infrastructure and publicly say it's fixed, and then you get popped again...

snailmailman 7y ago
i think so. here is a wayback machine link to the previous blog post https://web.archive.org/web/20190412000400/https://matrix.org/blog/2019/04/11/security-incident/ https://web.archive.org/web/20190412000400/https://matrix.or... it stated "Having fully flushed out the attacker [...]" which i guess turned out to be false :-/ also im getting invalid HTTPS certs on the blog now. for some reason im getting a cert that looks like its for github.com ? edit: now im getting a lets encrypt cert on matrix.org, but a cloudflare SSL error page when i go to www.matrix.org ? the lets encrypt cert looks like it was just issued about an hour ago. edit2: i guess both with and without www. are lets encrypt, but the with www. cert was issued back in february (and gives a cloudflare SSL error page), while without www. was issued today. (and gives the current hacked message)
- deepwell 7y ago
  Here is the message of the hacker, after the matrix.org admins allegedly cleaned everything up and wrote the original blog post: https://web.archive.org/web/20190412055614/https://matrix.org/blog/2019/04/11/security-incident/ https://web.archive.org/web/20190412055614/https://matrix.or...

ge0rg 7y ago

The most favorable reading of the current defacement page is that the attackers still controls the DNS, but no other parts of the infrastructure. Otherwise, the page probably wouldn't run off github.

localhostdotdev 7y ago

content before it gets fixed: Time for actual transparency. [list of servers, uname -a for each] root@[name]:/var/lib/postgresql# df -h [list of partitions] $ cat users.txt | grep [name] | head -n1 @[name]:matrix.org|[hash] $ wc -l users.txt [~6M users] See you soon. (affects whole site, even https://matrix.org https://matrix.org, site is on jekyll BTW)

l2dy 7y ago
Here's the source of that page: https://github.com/matrixnotorg/matrixnotorg.github.io/blob/master/blog/2019/04/11/security-incident/index.html https://github.com/matrixnotorg/matrixnotorg.github.io/blob/....
- localhostdotdev 7y ago
  account got banned :) (it's on archive.org though)

mkj 7y ago

> As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history That seems like a fairly bad usability/security design?

zigara 7y ago

The attacker seems to have responded: https://github.com/matrix-org/matrix.org/issues/357 https://github.com/matrix-org/matrix.org/issues/357 edit: just saw the rest: https://github.com/matrix-org/matrix.org/issues?utf8=%E2%9C%93&q=is%3Aissue+SECURITY https://github.com/matrix-org/matrix.org/issues?utf8=%E2%9C%... "[SECURITY] SSH Agent Forwarding I noticed in your blog post that you were talking about doing a postmortem and steps you need to take. As someone who is intimately familiar with your entire infrastructure, I thought I could help you out. Complete compromise could have been avoided if developers were prohibited from using ForwardAgent yes or not using -A in their SSH commands. The flaws with agent forwarding are well documented."

deleted 7y ago
[deleted]

irgeek 7y ago

Assuming the GitHub issues are from the actual attacker -- and I see no reason to doubt they are -- this is very troubling: https://github.com/matrix-org/matrix.org/issues/363 https://github.com/matrix-org/matrix.org/issues/363 Compromise began well over a month ago Yikes. That's a long time for a compromise to go unnoticed.

ygjb 7y ago
Not really, the average in the industry seems to be floating between 70 and 400 days depending on the source of your stats on the topic (different vendors and reports use different stats for this)