6 ms·

Perfctl: Stealthy malware targeting Linux servers

168 points by johnfernow 2y ago

johnfernow 2y ago
From the article: > "Aqua Nautilus researchers aim to shed light on a Linux malware that, over the past 3-4 years, has actively sought more than 20,000 types of misconfigurations in order to target and exploit Linux servers. If you have a Linux server connected to the internet, you could be at risk. In fact, given the scale, we strongly believe the attackers targeted millions worldwide with a potential number of victims of thousands, it appears that with this malware any Linux server could be at risk. ... - It utilizes rootkits to hide its presence. - When a new user logs into the server, it immediately stops all “noisy” activities, lying dormant until the server is idle again. - It utilizes Unix socket for internal communication and TOR for external communication. - After execution, it deletes its binary and continues to run quietly in the background as a service. - It copies itself from memory to various locations on the disk, using deceptive names. - It opens a backdoor on the server and listens for TOR communications. - It attempts to exploit the Polkit vulnerability (CVE-2021-4043) to escalate privileges. In all the attacks observed, the malware was used to run a cryptominer, and in some cases, we also detected the execution of proxy-jacking software. During one of our sandbox tests, the threat actor utilized one of the malware’s backdoors to access the honeypot and started deploying some new utilities to better understand the nature of our server, trying to understand what exactly we are doing to its malware." The article goes into more depth of the attack flow, what the malware does, and how they detected it.
- CGamesPlay 2y ago
  The CVE has a typo; the actual is CVE-2021-4034. https://nvd.nist.gov/vuln/detail/CVE-2021-4034 https://nvd.nist.gov/vuln/detail/CVE-2021-4034
  - ykonstant 2y ago
    From the link: >The current version of pkexec doesn't handle the calling parameters count correctly and ends trying to execute environment variables as commands. Oh, for fucks sake.
  - stuff4ben 2y ago
    Thank you for that. I was wondering why a medium vuln was causing so much headache in a binary that wasn't even described by the article.
- guenthert 2y ago
  > - It opens a backdoor on the server and listens for TOR communications. So a `lsof -iTCP` should list it, right? Is it using TCP port 9050 or a custom port? EDIT: Ha, they are (not surprisingly) way ahead of me. From the article: "The malware continues to copy itself from memory to half a dozen other locations, with names that appear as conventional system files. It also drops a rootkit and a few popular Linux utilities that were modified to serve as user land rootkits (i.e. ldd, lsof)."
  - crackez 2y ago
    There's always - cat /proc/net/tcp* And remember: echo * can be your "ls" in a pinch.
    - Pesthuf 2y ago
      But let’s be honest, there’s no reason to use these unless you already know your server is compromised. In which case the server would be taken down rather than ssh‘d into. And even then the attacker could patch cat, bash, provide sneaky aliases or just compromise Libc altogether.
    - rurban 2y ago
      Not with a kernel module. Then this is also compromised
- loeg 2y ago
  I don't think we need a copy/paste of the very first page of the article in the comments.
pm2222 2y ago
Does uefi secureboot help or not at all?
- johnfernow 2y ago
  I'm not sure, outside of my expertise, but I think it might not help. The attack takes place after boot, so maybe at best UEFI secureboot could prevent persistence of the malware, but I don't think it'd even achieve that, as the malware adds popular Linux utilities that were modified to serve as user land rootkits, and runs them by modifying the ~/.profile script. That script is ran when the user logs in (it starts the malware first, and then everything that's supposed to run on the server after), and I don't believe UEFI secureboot has any protections against ~/.profile script modifications or rootkits ran after boot.
- vaylian 2y ago
  See the mitigation section in the article: https://www.aquasec.com/blog/perfctl-a-stealthy-malware-targeting-millions-of-linux-servers/#section-7 https://www.aquasec.com/blog/perfctl-a-stealthy-malware-targ... UEFI won't help here. But keeping your system up to date and limiting the system to the necessary functionality will help you.
  - opengears 2y ago
    In the mitigation section there is written 'Deploy Runtime Protection: Use advanced anti-malware and behavioral detection tools that can detect rootkits, cryptominers, and fileless malware like perfctl.' -- which tools can we currently use to detect perfctl?
    - ykonstant 2y ago
      I hear Crowdstrike is king (≖ ͜ ≖)
      - doubled112 2y ago
        To be fair, a system that rebooted and won't come back up IS pretty secure.
PhilipRoman 2y ago
A lot of focus on the malware itself, but not so much on the misconfigurations and vulnerabilities which enable it. Would love to see that list. Other than that, the evasion techniques look pretty traditional. And of course the privilege escalation is done by a polkit vulnerability...
- allkindsof 2y ago
  Also more interested in what the misconfigs are.
- fredsted 2y ago
  It seems to be a RocketMQ vuln; it's described further down the page.
- deleted 2y ago
  [deleted]
opengears 2y ago
are there any scripts or steps to 100% detect perfectl yet?
- Gys 2y ago
  > In all the attacks observed, the malware was used to run a cryptominer I assume it starts by detecting a continuous 100% utilization of the cpu’s.
  - jellykid 2y ago
    Supposedly it tones down it's activity while a user is logged in and waits for the machine to go idle. Another reason to have centralized performance monitoring.
    - Gys 2y ago
      Yes, but tools like htop show the average load over the last 15 min. So I assume that will show a high utilization.
- aflukasz 2y ago
  Article mentions couple of const paths that are used, like /root/.config/cron/perfcc. Also, it mentions that ~/.profile is modified (EDIT: and many others, actually), so IDS like AIDE, if operated correctly, should alert you on that. I don't see any mentions about attempts to circumvent locally run IDS. I wonder if/why malware author did not attempt any evasive actions here, given how much they try otherwise. Maybe cost/benefit ratio is too low?
- theamk 2y ago
  From the text, tons! This rootkit does not seem very stealthy at all. IMHO, a simplest one is to check $PATH. If there are suspicious entries, like /bin/.local/bin, it's a sign of infection. You can also check for presence of the specific files as mentioned close to the end of article.
sea-gold 2y ago
Here is Ars Technica's write up: https://arstechnica.com/security/2024/10/persistent-stealthy-linux-malware-has-infected-thousands-since-2021/ https://arstechnica.com/security/2024/10/persistent-stealthy...
- kvemkon 2y ago
  And the discussion https://news.ycombinator.com/item?id=41736572 https://news.ycombinator.com/item?id=41736572.
theamk 2y ago
Heh, my work has a firewall policy: any activity towards TOR servers flags an alert and makes security contact you. If you don't confirm it was by design, they'll start full scale "computer compromised" procedure. (And if you do confirm it it was by design, then they'll ask you to change that design if possible :) ) I thought it was overly paranoid, but it seems that would have really helped in this case.
- waihtis 2y ago
  Unless youre doing security research, there close to zero legitimate uses of Tor for the average citizen.
  - indrora 2y ago
    There are plenty of reasons to use Tor. Not only to obfuscate location but to defend against mass surveillance, including that of the state [1]. Remember that Tor was developed originally and funded by the US State Department. Tor makes communications of LBTQ/Anti-authoritarian/journalist individuals safer in a world where the State Department has to put out advisories [2] about traveling while queer, a pride flag sticker is illegal and punishable by death or imprisonment in over 24 countries and journalists are imprisoned for talking to those who speak out. Tor is an essential tool for all citizens -- Especially those in the US who would be targeted by those who seek infinite, unrestrained power. [1] https://i.kym-cdn.com/photos/images/original/002/128/768/403.jpg https://i.kym-cdn.com/photos/images/original/002/128/768/403... [2] https://travel.state.gov/content/travel/en/traveladvisories/traveladvisories/worldwide-caution.html https://travel.state.gov/content/travel/en/traveladvisories/...
    - mppm 2y ago
      In theory, yes. In practice, the safety that Tor provides is in numbers, and the numbers just don't look good any more. By using Tor, you are entering a fairly small pool that you share with legit criminals and will be blocked and targeted accordingly. A good VPN like Mullvad is the saner option for most people.
TZubiri 2y ago
"CVE-2023-33246 is a vulnerability found in RocketMQ, which is a software that manages messages" A more appropriate but less clickbaity title would be "Stealthy malware targetting servers running RocketMQ"
- nine_k 2y ago
  The description of the payloads and of the hiding methods was educational though. Other malwares likely use similar techniques after the initial penetration.
- deleted 2y ago
  [deleted]
- cyanydeez 2y ago
  The entry method is distinct from the exploitation or persistence method. Like, how one picks a lock versus how one lives rent free in the attic without discovery. Theyre not specifically dependent activities.
fungah 2y ago
I've been dealing with something similar - maybe actually this for 2 months. There were so.e great insights from this researcher but they're missing some very fucked up elements of this malware. 1. I'm pretty sure it has. "fuck with it" scale. It leaves you alone if you don't fuck with it. In fact, I'd bet money that this malware did all the cryptocurrency shit for a reason like a bait and switch. 2. It effects android too. Doesn't seem to matter what device or how updated it is. 3. And windows. 4. It isn't persistent through rootkits. I mean, it is. But it's also deeper. My current thinking is that is persistent on my machines using the RAM training alorithms to spin itself up. From.. 5. Your display. I have four displays that have had their firmware fucked with. Just discovered it's on a brand new mobo that I set up ensuring there was not a single peripheral I'd used before, no leds. Hadn't even installed an os before running a ram only mode Linux session from a hardware write protected usb THROUGH a write protected usb bridge. The only thing it was connected to that wasn't new was my monitor, and the UBS key (created from a secure pc and immediately write protected). I think that this thing is EVERYWHERE. I've seen references in bash files from the initramfs that allude to escalating is action based on variables I. Ould t pin down
- fasa99 2y ago
  Hi friend, I'm dealing with the same thing and agree with everything you said. ADDITIONALLY we found that it loves office printers, great attack vector to hit the whole office. Also did you figure out why it reprograms display firmware? It spreads through displays and webcams optically. It can send or receive from either (same way NSA uses speakers as microphones basically). That's not the crazy part though, the crazy part is that it does human and canine retinal embeddings from either. It basically writes code on the retina and then that person can spread it to new systems. That turned out to be the biggest problem on our side. Sunglasses don't work.
joyjudy 2y ago
[dead]

168 points by johnfernow 2y ago

From the article: > "Aqua Nautilus researchers aim to shed light on a Linux malware that, over the past 3-4 years, has actively sought more than 20,000 types of misconfigurations in order to target and exploit Linux servers. If you have a Linux server connected to the internet, you could be at risk. In fact, given the scale, we strongly believe the attackers targeted millions worldwide with a potential number of victims of thousands, it appears that with this malware any Linux server could be at risk. ... - It utilizes rootkits to hide its presence. - When a new user logs into the server, it immediately stops all “noisy” activities, lying dormant until the server is idle again. - It utilizes Unix socket for internal communication and TOR for external communication. - After execution, it deletes its binary and continues to run quietly in the background as a service. - It copies itself from memory to various locations on the disk, using deceptive names. - It opens a backdoor on the server and listens for TOR communications. - It attempts to exploit the Polkit vulnerability (CVE-2021-4043) to escalate privileges. In all the attacks observed, the malware was used to run a cryptominer, and in some cases, we also detected the execution of proxy-jacking software. During one of our sandbox tests, the threat actor utilized one of the malware’s backdoors to access the honeypot and started deploying some new utilities to better understand the nature of our server, trying to understand what exactly we are doing to its malware." The article goes into more depth of the attack flow, what the malware does, and how they detected it.

CGamesPlay 2y ago
The CVE has a typo; the actual is CVE-2021-4034. https://nvd.nist.gov/vuln/detail/CVE-2021-4034 https://nvd.nist.gov/vuln/detail/CVE-2021-4034
- ykonstant 2y ago
  From the link: >The current version of pkexec doesn't handle the calling parameters count correctly and ends trying to execute environment variables as commands. Oh, for fucks sake.
- stuff4ben 2y ago
  Thank you for that. I was wondering why a medium vuln was causing so much headache in a binary that wasn't even described by the article.
guenthert 2y ago
> - It opens a backdoor on the server and listens for TOR communications. So a `lsof -iTCP` should list it, right? Is it using TCP port 9050 or a custom port? EDIT: Ha, they are (not surprisingly) way ahead of me. From the article: "The malware continues to copy itself from memory to half a dozen other locations, with names that appear as conventional system files. It also drops a rootkit and a few popular Linux utilities that were modified to serve as user land rootkits (i.e. ldd, lsof)."
- crackez 2y ago
  There's always - cat /proc/net/tcp* And remember: echo * can be your "ls" in a pinch.
  - Pesthuf 2y ago
    But let’s be honest, there’s no reason to use these unless you already know your server is compromised. In which case the server would be taken down rather than ssh‘d into. And even then the attacker could patch cat, bash, provide sneaky aliases or just compromise Libc altogether.
  - rurban 2y ago
    Not with a kernel module. Then this is also compromised
loeg 2y ago
I don't think we need a copy/paste of the very first page of the article in the comments.

pm2222 2y ago

Does uefi secureboot help or not at all?

johnfernow 2y ago
I'm not sure, outside of my expertise, but I think it might not help. The attack takes place after boot, so maybe at best UEFI secureboot could prevent persistence of the malware, but I don't think it'd even achieve that, as the malware adds popular Linux utilities that were modified to serve as user land rootkits, and runs them by modifying the ~/.profile script. That script is ran when the user logs in (it starts the malware first, and then everything that's supposed to run on the server after), and I don't believe UEFI secureboot has any protections against ~/.profile script modifications or rootkits ran after boot.
vaylian 2y ago
See the mitigation section in the article: https://www.aquasec.com/blog/perfctl-a-stealthy-malware-targeting-millions-of-linux-servers/#section-7 https://www.aquasec.com/blog/perfctl-a-stealthy-malware-targ... UEFI won't help here. But keeping your system up to date and limiting the system to the necessary functionality will help you.
- opengears 2y ago
  In the mitigation section there is written 'Deploy Runtime Protection: Use advanced anti-malware and behavioral detection tools that can detect rootkits, cryptominers, and fileless malware like perfctl.' -- which tools can we currently use to detect perfctl?
  - ykonstant 2y ago
    I hear Crowdstrike is king (≖ ͜ ≖)
    - doubled112 2y ago
      To be fair, a system that rebooted and won't come back up IS pretty secure.

PhilipRoman 2y ago

A lot of focus on the malware itself, but not so much on the misconfigurations and vulnerabilities which enable it. Would love to see that list. Other than that, the evasion techniques look pretty traditional. And of course the privilege escalation is done by a polkit vulnerability...

allkindsof 2y ago
Also more interested in what the misconfigs are.
fredsted 2y ago
It seems to be a RocketMQ vuln; it's described further down the page.
deleted 2y ago
[deleted]

opengears 2y ago

are there any scripts or steps to 100% detect perfectl yet?

Gys 2y ago
> In all the attacks observed, the malware was used to run a cryptominer I assume it starts by detecting a continuous 100% utilization of the cpu’s.
- jellykid 2y ago
  Supposedly it tones down it's activity while a user is logged in and waits for the machine to go idle. Another reason to have centralized performance monitoring.
  - Gys 2y ago
    Yes, but tools like htop show the average load over the last 15 min. So I assume that will show a high utilization.
aflukasz 2y ago
Article mentions couple of const paths that are used, like /root/.config/cron/perfcc. Also, it mentions that ~/.profile is modified (EDIT: and many others, actually), so IDS like AIDE, if operated correctly, should alert you on that. I don't see any mentions about attempts to circumvent locally run IDS. I wonder if/why malware author did not attempt any evasive actions here, given how much they try otherwise. Maybe cost/benefit ratio is too low?
theamk 2y ago
From the text, tons! This rootkit does not seem very stealthy at all. IMHO, a simplest one is to check $PATH. If there are suspicious entries, like /bin/.local/bin, it's a sign of infection. You can also check for presence of the specific files as mentioned close to the end of article.

sea-gold 2y ago

Here is Ars Technica's write up: https://arstechnica.com/security/2024/10/persistent-stealthy-linux-malware-has-infected-thousands-since-2021/ https://arstechnica.com/security/2024/10/persistent-stealthy...

kvemkon 2y ago
And the discussion https://news.ycombinator.com/item?id=41736572 https://news.ycombinator.com/item?id=41736572.

theamk 2y ago

Heh, my work has a firewall policy: any activity towards TOR servers flags an alert and makes security contact you. If you don't confirm it was by design, they'll start full scale "computer compromised" procedure. (And if you do confirm it it was by design, then they'll ask you to change that design if possible :) ) I thought it was overly paranoid, but it seems that would have really helped in this case.

waihtis 2y ago
Unless youre doing security research, there close to zero legitimate uses of Tor for the average citizen.
- indrora 2y ago
  There are plenty of reasons to use Tor. Not only to obfuscate location but to defend against mass surveillance, including that of the state [1]. Remember that Tor was developed originally and funded by the US State Department. Tor makes communications of LBTQ/Anti-authoritarian/journalist individuals safer in a world where the State Department has to put out advisories [2] about traveling while queer, a pride flag sticker is illegal and punishable by death or imprisonment in over 24 countries and journalists are imprisoned for talking to those who speak out. Tor is an essential tool for all citizens -- Especially those in the US who would be targeted by those who seek infinite, unrestrained power. [1] https://i.kym-cdn.com/photos/images/original/002/128/768/403.jpg https://i.kym-cdn.com/photos/images/original/002/128/768/403... [2] https://travel.state.gov/content/travel/en/traveladvisories/traveladvisories/worldwide-caution.html https://travel.state.gov/content/travel/en/traveladvisories/...
  - mppm 2y ago
    In theory, yes. In practice, the safety that Tor provides is in numbers, and the numbers just don't look good any more. By using Tor, you are entering a fairly small pool that you share with legit criminals and will be blocked and targeted accordingly. A good VPN like Mullvad is the saner option for most people.

TZubiri 2y ago

"CVE-2023-33246 is a vulnerability found in RocketMQ, which is a software that manages messages" A more appropriate but less clickbaity title would be "Stealthy malware targetting servers running RocketMQ"

nine_k 2y ago
The description of the payloads and of the hiding methods was educational though. Other malwares likely use similar techniques after the initial penetration.
deleted 2y ago
[deleted]
cyanydeez 2y ago
The entry method is distinct from the exploitation or persistence method. Like, how one picks a lock versus how one lives rent free in the attic without discovery. Theyre not specifically dependent activities.

fungah 2y ago

I've been dealing with something similar - maybe actually this for 2 months. There were so.e great insights from this researcher but they're missing some very fucked up elements of this malware. 1. I'm pretty sure it has. "fuck with it" scale. It leaves you alone if you don't fuck with it. In fact, I'd bet money that this malware did all the cryptocurrency shit for a reason like a bait and switch. 2. It effects android too. Doesn't seem to matter what device or how updated it is. 3. And windows. 4. It isn't persistent through rootkits. I mean, it is. But it's also deeper. My current thinking is that is persistent on my machines using the RAM training alorithms to spin itself up. From.. 5. Your display. I have four displays that have had their firmware fucked with. Just discovered it's on a brand new mobo that I set up ensuring there was not a single peripheral I'd used before, no leds. Hadn't even installed an os before running a ram only mode Linux session from a hardware write protected usb THROUGH a write protected usb bridge. The only thing it was connected to that wasn't new was my monitor, and the UBS key (created from a secure pc and immediately write protected). I think that this thing is EVERYWHERE. I've seen references in bash files from the initramfs that allude to escalating is action based on variables I. Ould t pin down

fasa99 2y ago
Hi friend, I'm dealing with the same thing and agree with everything you said. ADDITIONALLY we found that it loves office printers, great attack vector to hit the whole office. Also did you figure out why it reprograms display firmware? It spreads through displays and webcams optically. It can send or receive from either (same way NSA uses speakers as microphones basically). That's not the crazy part though, the crazy part is that it does human and canine retinal embeddings from either. It basically writes code on the retina and then that person can spread it to new systems. That turned out to be the biggest problem on our side. Sunglasses don't work.

joyjudy 2y ago

[dead]