37 ms·
Backdoor in upstream xz/liblzma leading to SSH server compromise
- inevitable112 2y agoSurely the real target of this was Tor (which links liblzma) not random SSH servers.
- Tenobrus 2y agoIt looks like the person who added the backdoor is in fact the current co-maintainer of the project (and the more active of the two): https://tukaani.org/about.html https://tukaani.org/about.html
- kzrdude 2y agoMakes me wonder if he's an owner of the github organization, and what happens with it now?
- kzrdude 2y agoIn various places they say Lasse Collin is not online right now, but he did make commits a week ago https://git.tukaani.org/?p=xz.git;a=summary https://git.tukaani.org/?p=xz.git;a=summary
- shp0ngle 2y agowe should take this diagram and change "random person in nebraska" to "possibly a state-level attacker" https://xkcd.com/2347/ https://xkcd.com/2347/ nice
- deleted 2y ago[deleted]
- nolist_policy 2y agoDebian is considering that their infrastructure may be compromised[1]. [1] https://fulda.social/@Ganneff/112184975950858403 https://fulda.social/@Ganneff/112184975950858403
- yogorenapan 2y agoVery strange behavior from the upstream developers. Possible government involvement? I have a feeling LANG is checked to target servers from particular countries
- deleted 2y ago[deleted]
- _lvbh 2y agoOne thing to note is that the person that added the commits only started contributing around late 2022 and appears to have a Chinese name. Might be required by law to plant the backdoor. That would be quite scary considering they have contributed to a wide variety of projects including C++ https://learn.microsoft.com/en-us/cpp/overview/whats-new-cpp-docs?view=msvc-170 https://learn.microsoft.com/en-us/cpp/overview/whats-new-cpp...
- ComputerGuru 2y agoNo one is being “required by law” to add vulnerabilities, it’s more likely they are foreign agents to begin with.
- bombcar 2y agoDepends on the law and where you are. Publicly we have https://www.eff.org/issues/national-security-letters/faq https://www.eff.org/issues/national-security-letters/faq and it's likely that other requests have occurred from time to time, even in the USA.
- computerfriend 2y ago> No one is being “required by law” to add vulnerabilities This is absolutely not the case in many parts of the world.
- yorwba 2y agoI don't think you need to worry about the C++ contribution: https://github.com/MicrosoftDocs/cpp-docs/commit/9a96311122acd6c6de1e2624cd0f95655fac6500 https://github.com/MicrosoftDocs/cpp-docs/commit/9a96311122a...
- Rucadi 2y agoSaw this on nix, which was using a compromised version in the unstable channel, I hope not too many systems are affected.
- jeffbee 2y agoSafety through obscurity and weirdness! If you disable ifunc, like any sensible person, this backdoor disables itself.
- prydt 2y agoI'm curious now. What is ifunc? (Had difficulty finding it through a search)
- jeffbee 2y agoifunc is a GNU method of interposing function calls with platform-optimized versions of the function. It is used to detect CPU features at runtime and insert, for example, AVX2-optimized versions of memcmp. It is seen in crypto a lot, because CPUs have many crypto-specific instructions. However, I don't like it much and I think software should be compiled for the target machine in the first place. My 1 hardened system that is reachable from the public network is based on musl, built mostly with llvm, and with ifunc disabled.
- cesarb 2y ago> However, I don't like it much and I think software should be compiled for the target machine in the first place. That means you either have to compile software locally on each machine, or you have a combinatorial explosion of possible features. Compiling locally has several drawbacks. It needs the full compilation environment installed on every machine, which uses a lot of disk space, and some security people dislike it (because then attackers can also compile software locally on that machine); compiling needs a lot of memory and disk space, and uses a lot of processor time and electric power. It also means that signature schemes which only allow signed code cannot be used (or you need to have the signing key available on the target machine, making it somewhat pointless). The combinatorial explosion of features has been somewhat tamed lately, by bundling sets of feature into feature levels (x86_64-v1, etc), but that still quadruples the amount of compiled code to be distributed, and newer features still have to be selected at runtime.
- 2y ago
- _mlbt 2y ago> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?
- delphij 2y agoFreeBSD is not affected as the payloads in question were stripped out, however we are looking into improvements to our workflow to further improve the import process.
- deleted 2y ago[deleted]
- rasengan 2y ago> One portion of the backdoor is solely in the distributed tarballs. For easier reference, here's a link to debian's import of the tarball, but it is also present in the tarballs for 5.6.0 and 5.6.1: Ubuntu 22.04 version: dpkg -l |grep liblzma ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-format compression library Whew!
- cf100clunk 2y agoI am *not* a security researcher, nor a reverse engineer. There's lots of stuff I have not analyzed and most of what I observed is purely from observation rather than exhaustively analyzing the backdoor code. I love this sort of technical writing from contributors outside the mainstream debugging world who might be averse to sharing. What an excellently summarized report of his findings that should be seen as a template.
- anarazel 2y agoFWIW, it felt intimidating as hell. And I'm fairly established professionally. Not sure what I'd have done earlier in my career (although I'd probably not have found it in the first place).
- internetter 2y ago> Not sure what I'd have done earlier in my career To anybody in this sorta situation, you should absolutely share whatever you have. It doesn’t need to be perfect, good, or 100% accurate, but if there’s a risk you could help a lot of people
- RockRobotRock 2y agoI hope you've hired a PR person for all the interviews :)
- aerhardt 2y agoThis story is an incredible testament to how open-source software can self-regulate against threats, and more broadly, it reminds us that we all stand on the shoulders of contributors like you. Thank you!
- ddalex 2y agoThis is one threat that was discovered, only because the implementer was sloppy. Think about what various corps and state-level actors have been putting in there.
- 2y ago
- agwa 2y ago> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded into their address space (even if you don't use systemd as PID 1), increasing the risks of supply chain attacks. Thanks, systemd.
- capitainenemo 2y agoFWIW, I did a quick check on a Devuan system. The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages. However that stub does not link to lzma.
- cf100clunk 2y agoOn an MX Linux (non-systemd Debian-derived distro) box I ran ldd on /sbin/ssh and also ran: [EDIT: this string gives cleaner results:] lsof -w -P -T -p $(pgrep sshd)|grep mem and saw liblzma in the results of both, so there is some sort of similar trickery going on.
- capitainenemo 2y agoHuh. That's rather surprising. Do you know how MX Linux handles systemd? Devuan does that shimming of upstream. Do they perhaps just try to leave out certain packages? Anyway. I did not see lzma in the results on Devuan running a process check (just in case). I did see it on a Debian.
- cf100clunk 2y agoIt turns out MX uses a package called systemd-shim that seems to be the Debian one: $aptitude show systemd-shim Package: systemd-shim Version: 10-6 State: installed Automatically installed: no Priority: extra Section: admin Maintainer: Debian QA Group <packages@qa.debian.org> Architecture: amd64 Uncompressed Size: 82.9 k Depends: libc6 (>= 2.34), libglib2.0-0 (>= 2.39.4), cgmanager (>= 0.32) Suggests: pm-utils Conflicts: systemd-shim:i386 Breaks: systemd (< 209), systemd:i386 (< 209) Description: shim for systemd This package emulates the systemd function that are required to run the systemd helpers without using the init service
- 0x0 2y agoHomebrew is currently shipping 5.6.1 (and was shipping 5.6.0 as well). Hopefully not affected on mac?
- stephenr 2y agoThe issue is caused by patches to add integration with systemd, so no, this won't affect SSH on a Mac.
- 0x0 2y agoJust because macs don't use systemd, doesn't mean the backdoor won't work. The oss-sec post talks about liblzma having backdoors in crc32_resolve() and crc64_resolve() and that it has not been fully reversed. This could perhaps affect more than just sshd on x86-64 linux?
- anarazel 2y ago> Just because macs don't use systemd, doesn't mean the backdoor won't work. Practically speaking it can't - For one the script injected into the build process tests that you're running on x86-64 linux, for another, the injected code is elf code, which wouldn't link on a mac. It also needs to manipulate dynamic linker datastructures, which would also not work the same on a mac. > This could perhaps affect more than just sshd on x86-64 linux? This however is true - /usr/sbin/sshd was the only argv[0] value that I found to "work", but it's possible there are others. "/usr/sbin/sshd" isn't a string directly visible in the injected code, so it's hard to tell.
- stephenr 2y agoThe article explains numerous concurrent conditions that have to be met for the backdoor to even be activated (at build time, not runtime), which combined make it extremely unlikely this will affect SSH on macOS: - linux - x86-64 - building with gcc & the GNU linker - part of a .deb or .rpm build Add to that, as the article explains: openssh does not directly use liblzma, the only reason SSH is affected at all, is because some Linux Distros patch openssh to link it against systemd, which does depend on liblzma. Could it affect things other than SSH on a Mac? Unlikely. The compromise was introduced in 5.6.0, but macOS Sonoma has 5.4.4 (from August last year).
- asveikau 2y agoThat's completely crazy, the backdoor is introduced through a very cryptic addition to the configure script. Just looking at the diff, it doesn't look malicious at all, it looks like build script gibberish.
- youainti 2y agoSummary: "The upstream xz repository and the xz tarballs have been backdoored." It is known to be in version 5.6.0 and 5.6.1, and the obfuscated code is found in the test directory.
- buildbot 2y agoThis potentially could be a full automated rootkit type breach right? Great - is any system with 5.6.1 possibly vulnerable? Also super weird a contributor thought they could slip this in and not have it be noticed at some point. It may point to burning that person (aka, they go to jail) for whatever they achieved with this. (And whoever they are…)
- q3k 2y agoNixOS/Pkgs 23.11 unaffected, unstable contains backdoored implementations (5.6.0, 5.6.1) but their OpenSSH sshd does not seem to link against systemd/liblzma, and the backdoor doesn't get configured in (only happens on .deb/.rpm systems).
- o11c 2y agoNote that NixOS has a unique advantage in that `dlopen` is easier to analyze, but you do have to check for it. A lot of people are looking only at `ldd` and missing that they can be vulnerable at runtime.
- jchw 2y agoIt may not have really mattered much for NixOS: > b) argv[0] needs to be /usr/sbin/sshd For once, the lack of FHS interoperability is a benefit, if only on accident.
- q3k 2y agoRight, but in this case it's not even compiled it, which is arguably better than compiled in but assumed dormant :) (at least until someone actually does a full analysis of the payload).
- dandanua 2y agoThat's one of the advantages of NixOS - viruses and mass hacks have lesser chance to function due to how different this OS is. Until it gets more popular, of course.
- AdmiralAsshat 2y ago> Red Hat assigned this issue CVE-2024-3094. Does that mean this affects RHEL and Fedora?
- formerly_proven 2y agoRHEL no, Fedora 41 and Rawhide yes. https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users https://www.redhat.com/en/blog/urgent-security-alert-fedora-... https://lists.debian.org/debian-security-announce/2024/msg00057.html https://lists.debian.org/debian-security-announce/2024/msg00...
- dralley 2y agoNote that Fedora 40 isn't even released yet, it's in beta, Fedora 41 / rawhide is basically a development branch used only by a small number of people.
- dTP90pN 2y agoA small number of people with likely professional involvement in the Fedora project and possibly RHEL. A supply chain attack serve as the basis for another supply chain attack.
- jethro_tell 2y agoRHEL won't get this bug for 2 years =)
- fargle 2y agoi knew there was an advantage to being 8-10 years out of date at all times... and when they do port finally backport this bug in 2026, they will probably implement the systemd integration with openssl (pbthththt...) via 600 patch files in some nonstandard divergent manner that thwarts the payload anyhow. see? i knew they were super duper secure.
- richardwhiuk 2y ago
- lpapez 2y agoSo many security companies publishing daily generic blog posts about "serious supply chain compromises" in various distros on packages with 0 downloads, and yet it takes a developer debugging performance issues to find an actual compromise. I worked in the software supply chain field and cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.
- markus_zhang 2y agoThat's the entire point of certification, and any certification at all. Certification does not guarantee performance. Actually, I would always cast a suspect glance to anyone who is FOCUSED on getting certification after certification without any side project.
- r0ckarong 2y ago> cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong. That's the entire point. You did everything you could by getting someone else look at it and saying it's fine.
- numpad0 2y agoThis needs a Rust joke. You know, the problem with the whole certification charade is it slows down jobs and prevents __actual_problems getting evaluated. But is it safe?
- bawolff 2y ago> the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong. That is actually a major point of a lot of corporate security measures (shifting risk)
- landownersubgrp 2y agoAnd that's a good thing.
- fourfour3 2y agoLooks like Arch Linux shipped both compromised versions - and 5.6.1-2 is out to hopefully resolve it.
- tutfbhuf 2y agoI upgraded Arch Linux on my server a few hours ago. Arch Linux does not fetch one of the compromised tarballs but builds from source and sshd does not link against liblzma on Arch. [root@archlinux ~]# pacman -Qi xz | head -n2 Name : xz Version : 5.6.1-2 [root@archlinux ~]# pacman -Qi openssh | head -n2 Name : openssh Version : 9.7p1-1 [root@archlinux ~]# ldd $(which sshd) | grep liblzma [root@archlinux ~]# It seems that Arch Linux is not affected.
- gpm 2y ago5.6.1-1 was built from what I understand to be one of the affected tarballs. This was patched in 5.6.1-2: https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/commit/881385757abdc39d3cfea1c3e34ec09f637424ad https://gitlab.archlinux.org/archlinux/packaging/packages/xz... I agree on the sshd linking part.
- tutfbhuf 2y agoInteresting, they just switched from tarballs to source 19 hours ago. It seems to me that Frederik Schwan had prior knowledge of the security issue, or it is just a rare coincidence.
- ComputerGuru 2y agoDistributions were notified under embargo.
- gpm 2y agoOn arch, `ldd $(which sshd)` doesn't list lzma or xz, so I think it's unaffected? Obviously still not great to be shipping malicious code that just happens to not trigger.
- bawolff 2y agoThe terrifying part is that this was primarily found because the backdoor was poorly made and causing performance problems. Makes you wonder what more competent actors can do.
- rwmj 2y agoI've analysed the backdoor myself and it's very sophisticated, not poorly made at all. The performance problem is surprising in this context, but I think next time they won't make that mistake.
- Nextgrid 2y agoDo you have a writeup or any details as to what it does? The logical thing based on this post is that it hooks the SSH key verification mechanism to silently allow some attacker-controlled keys but I wonder if there's more to it?
- rwmj 2y agoI was starting one, but the openwall message linked here is far more detailed and gets much further than I did. It's fiendishly difficult to follow the exploit.
- dhx 2y agosshd starts with root privileges and then proceeds to, in summary:[1] 1. Parse command line arguments 2. Setup logging 3. Load configuration files 4. Load keys/certificates into memory (notably including private keys) 5. Listen on a socket/port for incoming connections 6. Spawn a child process with reduced permissions (on Linux, using seccomp filters [2]) to respond to each incoming connection request This backdoor executes at order 0 before sshd's main function is invoked, overwriting internal sshd functions with compromised ones. As some ideas of what the backdoor could achieve: 1. Leak server private keys during handshakes with users (including unauthenticated users) allowing the keys to be passively stolen 2. Accept backdoor keys as legitimate credentials 3. Compromise random number generation to disable perfect forward secrecy 4. Execute code on the host (supplied remotely by a malicious user) with the 'root' permissions available to sshd upon launch. On most Linux distributions, systemd-analyze security sshd.service will give a woeful score of 9.6/10 (10 being the worst).[3] There is essentially NO sandboxing used because an assumption is made that you'd want to login as root with sshd (or sudo/su to root) and thus would not want to be restricted in what filesystem paths and system calls your remote shell can then invoke. The same attacker has also added code to Linux kernel build scripts which causes xz to be executed (xz at this point has a backdoor compiled into it) during the build of the Linux kernel where xz compression is used for the resulting image. Using this approach, the attacker can selectively choose to modify certain (or all) Linux kernel builds to do some very nasty things: 1. Leak Wireguard keys allowing them to be passively intercepted. 2. Compromise random number generation, meaning keys may be generated with minimal entropy (see Debian certificate problem from a few years ago). 3. Write LUKS master keys (keys used by dm-crypt for actually decrypting disks) to disks in retrievable format. 4, Introduce remote root code execution vulnerabilities into basic networking features such as TCP/IP code paths. [1] 'main' function: https://anongit.mindrot.org/openssh.git/tree/sshd.c https://anongit.mindrot.org/openssh.git/tree/sshd.c [2] https://anongit.mindrot.org/openssh.git/tree/sandbox-seccomp-filter.c https://anongit.mindrot.org/openssh.git/tree/sandbox-seccomp... [3] https://github.com/gentoo/gentoo/blob/HEAD/net-misc/openssh/files/sshd.service.1 https://github.com/gentoo/gentoo/blob/HEAD/net-misc/openssh/...
- rwmj 2y agoVery annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it's "great new features". We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo. He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.
- coding123 2y ago[flagged]
- zb3 2y agoNot sure why are people downvoting you... it's pretty unlikely that various Chinese IoT companies would just decide it's cool to add a backdoor, which clearly implies that no matter how good their intentions are, they simply might have no other choice.
- noncoml 2y agoBecause it’s naive to think that the owner of the account used his real identity.
- 2y ago
- thesnide 2y agoThe discussion to upload it to Debian is interesting on its own https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
- 0x0 2y agoWow, that's a lot of anonymous accounts adding comments there urging for a fast merge! And this "Hans Jansen" guy is apparently running around salsa.debian.org pushing for more updates in other projects as well: https://salsa.debian.org/users/hjansen/activity https://salsa.debian.org/users/hjansen/activity
- matthews2 2y ago> running around salsa.debian.org pushing for more updates in other projects as well This is quite common in most (all?) distributions. People are going through lists of outdated packages, updating them, testing them, and pushing them.
- mardifoufs 2y agoThat account seems to be a contributor for xz though, you can see him interact a lot with the author of the backdoor on the GitHub repo. Some pull requests seem to be just the two of them discussing and merging stuff (which is normal but looks weird in this context)
- deleted 2y ago[deleted]
- 2OEH8eoCRo0 2y agoAnd now we see why I don't trust anons, aliases, or anime characters to make contributions. My GitHub says exactly who I am!
- LtWorf 2y agoIt has been on the agenda for years to identify FOSS contributors with an id… Wet dream for authoritarians like you. What would it solve when identity theft happens on a mass scale on a day to day basis? It'd just ruin the life of some random person whose identity got stolen to create the account…
- formerly_proven 2y agoQuite ironic: The most recent commit in the git repo is "Simplify SECURITY.md", committed by the same Github account which added the backdoor. https://github.com/tukaani-project/xz/commit/af071ef7702debef4f1d324616a0137a5001c14c https://github.com/tukaani-project/xz/commit/af071ef7702debe...
- rany_ 2y agoIt's not ironic, this change is really sinister IMO. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.
- arp242 2y agoI've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts). Would be interesting to see what's going on here; the person who did the releases has done previous releases too (are they affected?) And has commits going back to 2022 – relatively recent, but not that recent. Many are real commits with real changes, and they have commits on some related projects like libarchive. Seems like a lot of effort just to insert a backdoor. Edit: anyone with access can add files to existing releases and it won't show that someone else added it (I just tested). However, the timestamp of the file will be to when you uploaded it, not that of the release. On xz all the timestamps of the files match with the timestamp of the release (usually the .tar.gz is a few minutes earlier, which makes sense). So looks like they were done by the same person who did the release. I suspected someone else might have added/altered the files briefly after the release before anyone noticed, but that doesn't seem to be the case.
- sslayer 2y agoI would be curious if their commits could be analyzed for patterns that could then be used to detect commits from their other account
- carom 2y agoThere was a DARPA program on this topic called Social Cyber. [1] 1. https://www.darpa.mil/program/hybrid-ai-to-protect-integrity-of-open-source-code https://www.darpa.mil/program/hybrid-ai-to-protect-integrity...
- deleted 2y ago[deleted]
- bombcar 2y agoOne thing that is annoying is that many open source projects have been getting "garbage commits" apparently from people looking to "build cred" for resumes or such. Easier and easier to hide this junk in amongst them.
- 2y ago
- bonyt 2y agoFor those panicking, here are some key things to look for, based on the writeup: - A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the last month or so. If you're not on a rolling release distro, your version is probably older. - A debian or RPM based distro of Linux on x86_64. In an apparent attempt to make reverse engineering harder, it does not seem to apply when built outside of deb or rpm packaging. It is also specific to Linux. - Running OpenSSH sshd from systemd. OpenSSH as patched by some distros only pulls in libsystemd for logging functionality, which pulls in the compromised liblzma5. Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade. It is possible there are other flaws or backdoors in liblzma5, though.
- treffer 2y agoUbuntu still ships 5.4.5 on 24.03 (atm). I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though. So for 5.4.5 the tagged release and download on github differ. It does change format strings, e.g. +#: src/xz/args.c:735 +#, fuzzy +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard" There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs. [Edit] formatting
- fransje26 2y agoThanks for the heads up.
- chasil 2y agoRHEL9 is shipping 5.2.5; RHEL8 is on 5.2.4.
- mort96 2y agoFYI, your formatting is broken. Hacker News doesn't support backtick code blocks, you have to indent code. Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?
- colanderman 2y agoThe latest commit from the user who committed those patches is weirdly a simplification of the security reporting process, to not request as much detail: https://github.com/tukaani-project/xz/commit/af071ef7702debef4f1d324616a0137a5001c14c https://github.com/tukaani-project/xz/commit/af071ef7702debe... Not sure what to make of this.
- caelum19 2y agoPotentially the purpose is that if someone goes to the effort to get those details together, they are more likely to send the same report to other trusted individuals. Maybe it was originally there to add legitimacy, then they got a report sent in, and removed it to slow the spread of awareness
- londons_explore 2y ago> Affected versions of XZ Utils Most people, to find the affected versions, would either have to bisect or delve deep enough to find the offending commit. Either of which would reveal the attacker. By not asking for the version, there is a good chance you just report "It's acting oddly, plz investigate".
- rany_ 2y agoI think the reason is pretty obvious. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.
- colanderman 2y agoThat repository is now disabled. But here's a similar change to the .github repository of tukaani-project from @JiaT75 to the bug report template: + or create a private Security Advisory instead. Under a commit titled "Wrap text on Issue template .yaml files." [1] https://github.com/tukaani-project/.github/commit/44b766adc413b7b7ccfa2bfdd89d2540e88739ca https://github.com/tukaani-project/.github/commit/44b766adc4...
- deleted 2y ago[deleted]
- AdmiralAsshat 2y agoYikes! Do you have any info on the individual's background or possible motivations?
- b112 2y agoYikes indeed. This fix is being rolled out very fast, but what about the entire rest of the codebase? And scripts? I mean, years of access? I'd trust no aspect of this code until a full audit is done, at least of every patch this author contributed. (note: not referring to fedora here, a current fix is required. But just generally. As in, everyone is rolling out this fix, but... I mean, this codebase is poison in my eyes without a solid audit)
- b112 2y agoThis seems to be the account, correct me if wrong (linked from the security email commit link): https://github.com/JiaT75 https://github.com/JiaT75 I hope authors of all these projects have been alerted. STest - Unit testing framework for C/C++. Easy to use by simply dropping stest.c and stest.h into your project! libarchive/libarchive - Multi-format archive and compression library Seatest - Simple C based Unit Testing Everything this account has done should be investigated. Woha, is this legit or some sort of scam on Google in some way?: https://github.com/google/oss-fuzz/pull/11587 https://github.com/google/oss-fuzz/pull/11587 edit: I have to be missing something, or I'm confused. The above author seems to be primary contact for xz? Have they just taken over?? Or did the bad commit come from another source, and a legit person applied it? A bit confused here.
- tux3 2y agoThe concern about other projects is fine, but let's be careful with attacks directed at the person. Maybe their account is compromised, maybe the username borrows the identity of an innocent person with the same name. Focus on the code, not people. No point forming a mob. (e: post above was edited and is no longer directed at the person. thanks for the edit.)
- deleted 2y ago
- ParetoOptimal 2y agoIf you have a recently updated NixOS unstable it has the affected version: $ xz --version xz (XZ Utils) 5.6.1 liblzma 5.6.1 EDIT: I've been informed on the NixOS matrix that they are 99% sure NixOS isn't affected, based on conversations in #security:nixos.org
- deleted 2y ago[deleted]
- PedroBatista 2y agoGiven the recent ( not so recent ) attacks/"bugs" I feel there is a need to do more than the already hard task of investigating and detecting attacks but also to bring IRL consequences to these people. My understanding is that right now it's pretty much a name and shame of people who most of the time aren't even real "people" but hostile agents either working for governments or criminal groups ( or both ) Getting punched in the face is actually a necessary human condition for a healthy civilization.
- buildbot 2y agoIn the article it says CISA was notified - that sounds like it's going to be a federal investigation if nothing else. If I was this person, I wouldn't be in the USA (or any US friendly nation) ASAP.
- graemep 2y agoOne of Jia Tan's recent contributions is "Speed up CRC32 calculation on LoongArch" I would guess the odds are that this is not someone in the US.
- buildbot 2y agoYeah I saw that - I wouldn't bet on them being in the US but who knows. Maybe they just really love CRC32 ;) And introducing backdoors (if it that was them not an account takeover).
- kevin_b_er 2y agoThose tarballs are PGP signed, too..
- rrix2 2y agoThat was a review of someone else's work? https://github.com/tukaani-project/xz/pull/86 https://github.com/tukaani-project/xz/pull/86
- 20after4 2y ago> "Docs: Simplify SECURITY.md." https://github.com/tukaani-project/xz/commit/af071ef7702debef4f1d324616a0137a5001c14c https://github.com/tukaani-project/xz/commit/af071ef7702debe... Removes instructions about details relevant to security reports. Heh, nice one.
- bagels 2y agoIs this a crime? Has anyone been prosecuted for adding a backdoor like this?
- pvg 2y agoHas anyone been prosecuted for adding a backdoor Google up Randal Schwartz. Caution: clickhole.
- bagels 2y agoSeems a little different. Based on a quick read, he gained unauthorized access to systems. In this case, backdoor code was offered to and accepted by xz maintainers.
- pvg 2y agoSeems a little different. Based on a quick read It is a little different but a thing that you might have missed in the quick read is that one of the things he was accused of was installing and using a backdoor.
- bagels 2y agoOne involves making unauthorized access, the other does not.
- ptx 2y agoLots of things are crimes even though they're just offering something to a victim who willingly accepts it, e.g. phishing attacks, fraudulent investment schemes, contaminated food products.
- bagels 2y agoSure. I'm wondering if there is a specific law that was broken here. It seems to me that it might be beneficial if there were some legal protection against this sort of act.
- move-on-by 2y agoFascinating. Just yesterday the author added a `SECURITY.md` file to the `xz-java` project. > If you discover a security vulnerability in this project please report it privately. *Do not disclose it as a public issue.* This gives us time to work with you to fix the issue before public exposure, reducing the chance that the exploit will be used before a patch is released. Reading that in a different light, it says give me time to adjust my exploits and capitalize on any targets. Makes me wonder what other vulns might exist in the author's other projects.
- ncr100 2y agoSecurity Researchers: Is this request-for-private-disclosure + "90-days before public" reasonable? It's a SEVERE issue, to my mind, and 90 days seems too long to me.
- BartjeD 2y agoThe fraudulent author must have enjoyed the 'in joke' -- He's the one create vulnerabilities..
- bawolff 2y agoWhether its reasonable is debatable, but that type of time frame is pretty normal for things that aren't being actively exploited. This situation is perhaps a little different as its not an accidental bug waiting to be discovered but an intentionally placed exploit. We know that a malicious person already knows about it.
- londons_explore 2y agoIf you were following Google Project Zero's policy (which many researchers do), any in-the-wild exploits would trigger an immediate reveal.
- larschdk 2y agoDetecting a security issue is one thing. Detecting a malicious payload is something completely different. The latter has intent to exploit and must be addressed immediately. The former has at least some chance of noone knowing about it.
- returningfory2 2y agoA couple of years ago I wrote a Go library that wraps the xz C code and allows you to do xz compression in Go: https://github.com/jamespfennell/xz https://github.com/jamespfennell/xz About a week ago I received the first PR on that repo, to upgrade to 5.6.1. I thought it was odd to get such a random PR...it's not the same GitHub account as upstream though.
- pinko 2y ago> it's not the same GitHub account as upstream This is valuable information, and a sign that this may be the tip of an iceberg.
- Bromeo 2y agoI don't want to read too much into it, but the person (supposedly) submitting the PR seems to work at 1Password since December last year, as per his Linkedin. (And his Linkedin page has a link to the Github profile that made the PR).
- returningfory2 2y agoYeah the GitHub account looks really really legitimate. Maybe it was compromised though?
- computerfriend 2y agoThe PR's two commits are signed by a key that was also used to sign previous commits belonging to that author.
- dralley 2y agoHold up, are you saying that https://github.com/jaredallard https://github.com/jaredallard and the accounts affiliated with this XZ backdoor share a PGP key? Or something else?
- 2y ago
- notyoutube 2y agoIs the solution against such attacks in the future only to scrutinize more, or are there other reasonable options in terms of hardening?
- JanisErdmanis 2y agoThe lesson here seems to not depend on tools written in languages that have complex, obscure build systems and no one is either able or interested to read. Using tools rewritten in Rust, Go or any other languege which resolves dependencies within project seems the only way to do hardening here.
- stbenjam 2y agoI agree there's safer languages than C, but nobody reads the 50,000 lines changed when you update the vendoring in a random golang project. It would be easy to introduce something there that nobody notices too.
- JanisErdmanis 2y agoIt is generally harder to introduce vulnerabilities in readable language even more when it is memory safe. Sure life is not perfect and bad actors would have found a ways to inject vulnerabilities also in Rust, Go codebase. The benefit of modern languages is that there is one way to build things and the source code is the only thing that needs to be auditted.
- mstef 2y agothis backdoor had nothing at all to do with memory safety.
- Lichtso 2y agoOnce somebody actually does this people are gonna complain the same as always: "The sole purpose of your project is to rewrite perfectly fine stuff in Rust for the sake of it" or something along these lines.
- perihelions 2y agoImagine a more competent backdoor attempt on xz(1)—one that wouldn't have been noticed this quickly. xz is everywhere. They could pull off a "reflections on trusting trust": an xz which selectively modifies a tiny subset of the files it sees, like .tar.xz software tarballs underlying certain build processes. Not source code tarballs (someone might notice)—tarballs distributing pre-compiled binaries. edit to add: Arch Linux' entire package system used to run on .tar.xz binaries (they switched to Zstd a few years ago [0]). [0] https://news.ycombinator.com/item?id=19478171 https://news.ycombinator.com/item?id=19478171 ("Arch Linux propose changing compression method from xz to zstd (archlinux.org)")
- nolist_policy 2y agodeb packages are xz compressed...
- 1oooqooq 2y agomy freaking kernels/initrd are xz or zstd compressed!
- nolist_policy 2y ago... and Debian is very serious about it: https://fulda.social/@Ganneff/112184975950858403 https://fulda.social/@Ganneff/112184975950858403
- joeyh 2y agoA backdoored xz could also run payloads hidden inside other xz files, allowing targeted attacks.
- Phenylacetyl 2y agoThe same authors have also contributed to Zstd
- joeyh 2y agodetails please? I do not see any such contributions to https://github.com/facebook/zstd https://github.com/facebook/zstd
- alright2565 2y agohttps://github.com/tukaani-project/tukaani-project.github.io/commit/42fbb038b4f36c9d1830144bafa6c9d0d101aaa9#diff-0eb547304658805aad788d320f10bf1f292797b5e6d745a3bf617584da017051R83-R87 https://github.com/tukaani-project/tukaani-project.github.io... > Note: GitHub automatically includes two archives Source code (zip) and Source code (tar.gz) in the releases. These archives cannot be disabled and should be ignored. The author was thinking ahead! Latest commit hash for this repo: 8a3b5f28d00ebc2c1619c87a8c8975718f12e271
- o11c 2y agoFor a long time, there was one legitimately annoying disadvantage to the git-generated tarballs though - they lost tagging information. However, since git 2.32 (released June 2021; presumably available on GitHub by August 2021 when they blogged about it) you can use `$Format:%(describe)$` ... limited to once per repository for performance reasons.
- rany_ 2y agoExcept this change was made in 2023, it is just scary how good this threat actor was.
- legobmw99 2y agoI believe they also do not include sub modules, which is a big disadvantage for some projects
- xvilka 2y agoYes. Also, GitHub recently made some upgrades that forced checksum changes on the autogenerated archives: https://github.blog/changelog/2023-01-30-git-archive-checksums-may-change/ https://github.blog/changelog/2023-01-30-git-archive-checksu...
- rom1v 2y agoBtw, this is not the only project providing a source tarball different from the git repo, for example libusb also does this (and probably others): - https://github.com/libusb/libusb/issues/1468#issuecomment-1974787595 https://github.com/libusb/libusb/issues/1468#issuecomment-19... - https://github.com/orgs/community/discussions/6003 https://github.com/orgs/community/discussions/6003
- londons_explore 2y agoI think the lesson here for packagers is that binary testdata should not be present while doing the build. It is too easy to hide things in testdata.
- yencabulator 2y agoNice idea, but then you just hide the attack in logo.png that gets embedded in the binary. Less useful for libraries, works plenty good for web/desktop/mobile.
- consumer451 2y agoThis entire thread is above my pay grade, but isn’t minimizing the attack surface always a good thing?
- ReflectedImage 2y agoIt's all irrelevant. The attacker social engineered their way to being the lead maintainer for the project.
- StressedDev 2y agoThe problem with the parent's suggestion is you end up banning lots of useful techniques while not actually stopping hackers from installing back doors or adding security exploits. The basic problem is once an attacker can submit changes to a project, the attacker can do a lot of damage. The only real solution is to do very careful code reviews. Basically, having a malicious person get code into a project is always going to be a disaster. If they can get control of a project, it is going to be even worse.
- consumer451 2y ago> The only real solution is to do very careful code reviews. Are there any projects that are well resourced enough to do this consistently, including all dependencies?
- wood_spirit 2y agoA lot of eyes will be dissecting this specific exploit, and investigating this specific account, but how can we find the same kind of attack in a general way if it’s being used in other projects and using other contributor names?
- londons_explore 2y agoNote that the malicious binary is fairly long and complex. This attack can be stopped by disallowing any binary testdata or other non-source code to be on the build machines during a build. You could imagine a simple process which checks out the code, then runs some kind of entropy checker over the code to check it is all unminified and uncompressed source code, before finally kicking off the build process. autogenerated files would also not be allowed to be in the source repo - they're too long and could easily hide bad stuff. Instead the build process should generate the file during the build.
- trulyrandom 2y agoThis requires a more comprehensive redesign of the build process. Most Linux distributions also run the tests of the project they're building as part of the build process.
- uecker 2y agoThe code that runs during testing should not be allowed to affect the package though. If this is possible, this is misdesigned.
- saulrh 2y agoProfile guided optimization is, unfortunately, wildly powerful. And it has a hard requirement that a casual link exists from test data (or production data!) to the build process.
- treffer 2y ago1. Everything must be visible. A diff between the release tarball and tag should be unacceptable. It was hidden from the eyes to begin with. 2. Build systems should be simple and obvious. Potentially not even code. The inclusion was well hidden. 3. This was caught through runtime inspection. It should be possible to halt any Linux system at runtime, load debug symbols and map _everything_ back to the source code. If something can't map back then regard it as a potentially malicious blackbox. There has been a strong focus and joint effort to make distributions reproducible. What we haven't managed though is prove that the project compromises only of freshly compiled content. Sorta like a build time / runtime "libre" proof. This should exist for good debugging anyway. It wouldn't hinder source code based backdoors or malicious vulnerable code. But it would detect a backdoor like this one. Just an initial thought though, and probably hard to do, but not impossibly hard, especially for a default server environment.
- KingOfLechia 2y ago[dead]
- Scaevolus 2y agoIt's wild that this could have laid dormant for far longer if the exploit was better written-- if it didn't spike slow down logins or disturb valgrind.
- deleted 2y ago[deleted]
- frankjr 2y agoNow consider that your average Linux distribution pulls in tens of thousands of packages, each of which can be similarly compromised. Pretty scary to think about.
- RGamma 2y agoThe terrible desktop software security model of weak/essentially non-existent security boundaries at run and compile time makes this all the more spicy. Computer security for billions runs on the simultaneous goodwill of many thousand contributors. Optimistically said it's actually a giant compliment to the programming community. And this is not even talking about hardware backdoors that are a million times worse and basically undetectable when done well. The myriad ways to betray user trust at any level of computation make me dizzy...
- afh1 2y agoI have exactly 719 packages on my Gentoo box, just rebuilt everything as part of the profile 23 upgrade.
- Luker88 2y ago@people who write github scanners for updates and security issues (dependabot and the like) Can we start including a blacklist of emails and names of contributors (with reasons/links to discussions)? I can't track them and I don't want them in my projects. Might not be very helpful as it is easy to create new identities, but I see no reason to make it easier for them. Also, I might approach differently someone with lots of contributions to known projects than a new account, so it still helps.
- arp242 2y agoIt takes a minute to create a new email address. And you can change or fake an email address on a git commit trivially. You, too, can writing code as anyone you want by just doing "git commit --author='Joe Biden <icecream@whitehouse.gov>'". On the internet nobody knows you're Joe Biden.
- nine_k 2y agoYou can write a rather simple GitHub action that would do that: look at a PR and reject / close it if you don't like it for some reason. AFAIK open-source projects have a free quota of actions. OTOH sticking to the same email for more than one exploit might be not as wise for a malicious agent.
- the8472 2y agogithub already suspended the account
- pfortuny 2y agoUnfortunately, this is how good bad actors work: with a very long-term point of view. There is no “harmless” project any more.
- ametrau 2y agoProbably a state actor. You can look far into the future when you’re working for the party.
- DrewRWx 2y agoAnd that long term perspective could be used constructively instead!
- calvinmorrison 2y agoWhich like, also wouldn't be totally weird if I found out that the xz or whatever library maintainer worked for the DoE as a researcher? I kind of expect governments to be funding this stuff.
- CanaryLayout 2y agoFrom what I read on masto, the original maint had personal life breakdown, etc. Their interest in staying as primary maint is gone. This is a very strong argument for FOSS to pick up the good habit of ditching/un-mainlining projects where they are sitting around for state actors to volunteer injecting commits to, and dep-stripping active projects from this cruft. Who wants to maintain on a shitty compression format? Someone who is dephunting, it turns out. Okay so your pirate-torrent person needs liblzma.so Offer it in the scary/oldware section of the package library that you need to hunt down the instructions to turn on. Let the users see that it's marked as obsolete, enterprises will see that it should go on the banlist.
- soraminazuki 2y agoUm, what? This incident is turning into such a big deal because xz is deeply ingrained as a core dependency in the software ecosystem. It's not an obscure tool for "pirates."
- n3uman 2y agohttps://github.com/tukaani-project/tukaani-project.github.io/commit/743dc204551ecccf831108bb33ad1ee00116b58e https://github.com/tukaani-project/tukaani-project.github.io... Does this mean anything that it changed to a parameter??
- danielhlockard 2y agono. unlikely.
- Aissen 2y agoLooks like one of the backdoor authors even went and disabled the feature the exploit relied on directly on oss-fuzz to prevent accidental discovery: https://social.treehouse.systems/@Aissen/112180302735030319 https://social.treehouse.systems/@Aissen/112180302735030319 https://github.com/google/oss-fuzz/pull/10667 https://github.com/google/oss-fuzz/pull/10667 But luckily there was some serendipity: "I accidentally found a security issue while benchmarking postgres changes." https://mastodon.social/@AndresFreundTec/112180083704606941 https://mastodon.social/@AndresFreundTec/112180083704606941
- miduil 2y agoThis is getting addressed here: https://github.com/google/oss-fuzz/issues/11760 https://github.com/google/oss-fuzz/issues/11760
- nialv7 2y agoThis in of itself can be legitimate. ifunc has real uses and it indeed does not work when sanitizer is enabled. Similar change in llvm: https://github.com/llvm/llvm-project/commit/1ef3de6b09f6b21a383fc7cf1ce1283df738015a https://github.com/llvm/llvm-project/commit/1ef3de6b09f6b21a...
- deleted 2y ago[deleted]
- throwaway290 2y agoand that was in mid 2023. Very funny that Wikipedia on this issue says > It is unknown whether this backdoor was intentionally placed by a maintainer or whether a maintainer was compromised Yeah, if you've been compromised for a year your attacker is now your identity. Can't just wave hands, practice infosec hygiene
- mdip 2y agoAnyone keeping current with OpenSUSE Tumbleweed got a update...downgrade. Prior to `zypper dup --no-allow-vendor-change` I had 5.6.0, now I'm at 5.4.6.
- intel_brain 2y agoI see `5.6.1.revertto5.4-3.2`
- weinzierl 2y agoThe backdoor is not in the C source directly, but a build script uses data from files in the test dir to only create the backdoor in the release tars. Did I summarize that correctly?
- soneil 2y agoThat's how I understand it. A build script that's in the releases tarballs but not the git repo, checks to see if it's being run as part of the debian/build or rpm build processes, and then injects content from one of the "test" files.
- bombcar 2y agoI could imagine another similar attack done against an image processing library, include some "test data" of corrupted images that should "clean up" (and have it actually work!) but the corruption data itself is code to be run elsewhere.
- sylware 2y agoThis is why the less the better... even if it means less comfortable... to a certain point obviously. And that includes SDKs...
- hgs3 2y agoI don't understand why you were downvoted. Having fewer moving parts does make it easier to catch issues.
- sylware 2y agoEverything which is not engaging in licking Big Tech balls (open source or not) on HN is served with severe downvoting, that most of the time (probably real trash human beings or AI trolls with headless blink/geeko|webkit).
- crispyambulance 2y agoI am not embarrassed to say... is there anything in there that someone who runs a server with ssh needs to know? I literally can't make heads or tails of the risk here. All I see is the very alarming and scary words "backdoor" and "ssh server" in the same sentence. If I am keeping stuff up to date, is there anything at all to worry about?
- pxx 2y agoYou should probably not be running your own publicly-accessible ssh servers if this email is not sufficient to at least start figuring out what your next actions are. The email itself comes with an evaluation script to figure out if anything is currently vulnerable to specifically this discovery. For affected distributions, openssh servers may have been backdoored for at least the past month.
- crispyambulance 2y agoYet here I am, getting up every morning and getting dressed and tying my shoes all by myself, and then maintaining a small number of servers that have openssh on them! Thanks, though, for pointing out the little script at the very end of that technical gauntlet of an email intended for specialists. I had gotten through the first 3 or 4 paragraphs and had given up. What I should have done is just googled CVE-2024-3094, whatever, still glad I asked.
- ShamelessC 2y ago> You should probably not be running your own publicly-accessible ssh servers if this email is not sufficient to at least start figuring out what your next actions are. That seems like a fairly unreasonable stance.
- frenchman99 2y agoNot at all. For instance, I don't know what the next steps are, but I run SSH servers behind Wireguard, exactly to prevent them being accessible in the case of such events. Wireguard is simple to setup, even if I lack the expertise to understand exactly how to go forward.
- sschueller 2y agoSo much for a quiet Easter holiday. Fuck
- multimoon 2y agoIt seems like based on the (very well written) analysis that this is a way to bypass ssh auth, not something that phones out which would've been even scarier. My server runs arch w/ a LTS kernel (which sounds dumb on the surface, but was by far the easiest way to do ZFS on Linux that wasn't Ubuntu) and it seems that since I don't have SSH exposed to the outside internet for good reason, and my understanding is Arch never patched shhd to begin with that I and most people who would be in similar situations to me are unaffected. Still insane that this happened to begin with, and I feel bad for the Archlinux maintainers who are now going to feel more pressure to try to catch things like this.
- NekkoDroid 2y agoBeing included via libsystemd isn't the only way ssh can load liblzma, it can come as an indirect dependency of Selinux (and its PAM stack) IIUC. Which makes it even a bit more funny (?) since Arch also doesn't officially support any Selinux stuff. There might be other ways sshd might pull in lzma, but those are the 2 ways I saw commonly mentioned. On a different note, pacman/makepkg got the ability to checksum source repository checkouts in 6.1.
- LeoPanthera 2y agoxz is just a horribly designed format, and always has been. If you use it, please switch to Lzip. Same compression level, but designed by someone competent. https://www.nongnu.org/lzip/ https://www.nongnu.org/lzip/ https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
- someguydave 2y agoThanks for that link, lzip sounds useful
- account42 2y agoSomeone competent? More like a drama queen butthurt that his pet project did not win the popularity contest. Not the kind of person I want to rely on for important tools.
- psi-jack 2y agoBe an asshole elsewhere. This makes me LESS want to use lzip because of such aggressive non-useful slander and just plain nonsense.
- gmnon 2y agoFunny how Lasse Collin started to ccing himself and Jia Tan from 2024-03-20 (that was a day of tons of xz kernel patches), he never did that before. :) https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.collin@tukaani.org/ https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.co...
- ncr100 2y agoAlso interesting, to me, how the GMail account for the backdoor contributor ONLY appears in the context of "XZ" discussions. Google their email address. Suggests a kind of focus, to me, and a lack of reality / genuineness.
- fullstop 2y agoThis also means that Google might know who they are, unless they were careful to hide behind VPN or other such means.
- bombcar 2y agoThis is extremely suspicious. It looks like someone may have noticed a unmaintained or lightly maintained project related to various things, and moved to take control of it. Otherwhere in the discussion here someone mentions the domain details changed; if you have control of the domain you have control of all emails associated with it.
- deleted 2y ago[deleted]
- ui2RjUen875bfFA 2y agothose pipe usages are quite suspicious https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/scripts/Makefile.lib?h=next-20240328&id=d86bab34742d0a67aeb76a3070bc90f7433b78e3#n528 https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n... https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/scripts/Makefile.lib?h=next-20240328&id=d86bab34742d0a67aeb76a3070bc90f7433b78e3#n531 https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n... pipeing into this shell script which now uses "eval" https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/scripts/xz_wrap.sh?h=next-20240328#n36 https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n... i guess this will be revisited and removed soon
- xyst 2y agoTime for another OS wipe. Glad I keep bleeding edge versions VMd
- pushedx 2y agoMirror of the report, since the Openwall servers appear to be down. https://web.archive.org/web/20240329182300/https://www.openwall.com/lists/oss-security/2024/03/29/4 https://web.archive.org/web/20240329182300/https://www.openw...
- deleted 2y ago[deleted]
- probably_satan 2y ago[dead]
- markus_zhang 2y agoKeeps one wonder how many similar backdoors are there in the wild. What is the best way to execute such a move? This is sophisticated enough, but not good enough to stay unnoticed for a long while. If I were a state actor I'd think about at least 6-12 months.
- wannacboatmovie 2y agoReally disappointed in the number of posters here who are playing down rushing to judgement and suggesting perhaps a legitimate developer was compromised, when it's very clear this is sophisticated and not the work of a single person. I'm recalling bad memories of the Juniper backdoor years ago. Whoever did this, was playing the long game. As the top post pointed out, there was an effort to get this into Fedora.... which eventually makes its way into RHEL (read: high value targets). This was not for short term payoffs by some rogue developer trying to mine crypto or other such nonsense. What you are seeing here is the planting of seeds for something months or a year down the road.
- jcalvinowens 2y agoOof, this is on my Sid laptop: {0}[calvinow@mozart ~] dpkg-query -W liblzma5 liblzma5:amd64 5.6.0-0.2 {0}[calvinow@mozart ~] hexdump -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | grep -c f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410 1 Glad I stopped running sshd on my laptop a long time ago... still probably going to reinstall :/
- msm_ 2y agoNo obvious need to reinstall if you didn't use ssh and expose it publicly and are not a politically important person. All signs suggest that it was a nation state attack, and you are likely not a target.
- jcalvinowens 2y agoWe'll see... given that sshd is just one of many possible argv[0] it may have choosen to act on, I'm going to be a little paranoid until it's been fully analyzed. It just takes half an hour to reinstall, I have some shows to catch up on anyway :)
- frenchman99 2y agoI was thinking about reinstalling, because I'm on Manjaro Linux, which has the version in question. But it's unclear if earlier versions are also vulnerable. And if it did nasty things to your machine, how do you make sure that the backups you have do not include ways for the backdoor to reinstate itself?
- jcalvinowens 2y agoSure, the backdoor could have e.g. injected an libav exploit into a video file to re-backdoor my system when I watch it... that's too paranoid for me. I don't backup the whole system, just a specific list of things in /home.
- liveoneggs 2y agoThe best part is everyone disabling security tests that started failing
- c_rrodriguez 2y agoEverybody here In jumping into the pure malice bandwagon, I have a better hypothesis. Abandonment and inaction, the actual developers of these tools are elsewhere, oblivious to this drama, trying to make living because most of the time you are not compensated nor any corporation cares about making things sustainable at all. This is the default status of everything your fancy cloud depends on underneath. An attacker took over of the project slowly and stayed dormant until recently.
- johnklos 2y agoExcept that doesn't match reality. Someone has worked on xz for several years. Are you saying that this somewhat active contributor was likely actively contributing, then all of a sudden stopped, also stopped paying attention, and also allowed their account to be compromised or otherwise handed it over to a nefarious party? That fails the sniff test.
- c_rrodriguez 2y agoSee, people drop dead from OSS projects pretty frecuently, usually because they take on other life responsabilities and there is no cushion or guard against a bus factor. Then it is very easy to get credentials compromised or have your project took over by someone else.
- dkarras 2y agofunding model of OSS work is obviously a problem, but these problems are deeper than that. even a very well compensated OSS developer can get a knock on the door from a government agency (or anyone with a "$5 wrench")[1] and they might feel "compelled" to give up their maintainer creds. [1]: https://xkcd.com/538/ https://xkcd.com/538/
- ColonelPhantom 2y agoWell, yeah. The attacker, operating largely under the name Jia Tan, has successfully manipulated the original author (Lasse Collin) to become a maintainer. The attacker indeed laid dormant for two years, pretending to just be maintaining xz. I really don't see any way how this wasn't malice on Jia's part. But I do think your hypothesis applies to Lasse, who was just happy someone could help him maintain xz.
- elchief 2y ago"Amazon Linux customers are not affected by this issue, and no action is required. AWS infrastructure and services do not utilize the affected software and are not impacted. Users of Bottlerocket are not affected." https://aws.amazon.com/security/security-bulletins/AWS-2024-002/ https://aws.amazon.com/security/security-bulletins/AWS-2024-...
- ikekkdcjkfke 2y agoGithub should probably remove the dopamine hits of green checkmarks etc. like in serious stock broker apps
- Nathanba 2y agoThey should also remove the emojis, there is no need to have people feel good about upvotes. I've long felt uncomfortable with emojis on Slack as well. Responding to a coding or infrastructure issue should not be a social activity, I respond because it's my job and if the issue is worth it, not because a human being should feel appreciated (either them or me).
- Jonnax 2y agoMany people write code for fun and slack is a social communications platform. If you can't imagine people using these tools for other reasons than pure unemotional business value then you don't understand their market. Your suggestions would lose those platforms users and revenue.
- dpkirchner 2y agoThe emojis reduce (but not eliminate) the number of "me too!"s PRs will get, which IMO is a good thing.
- account42 2y agoThat only requires a vote button, not a whole range of cringe cartoon images.
- elintknower 2y agoCandidly how would someone protect against a vulnerability like this?
- anononaut 2y agoCompile all your packages from source would be a start.
- Hackbraten 2y agoYou’re not wrong. However, building from source wouldn’t have protected you against this specific backdoor. The upstream source tarball itself was compromised in a cleverly sneaky way.
- ui2RjUen875bfFA 2y agoYou might read https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4 "However, building from source wouldn’t have protected you against this specific backdoor." Depends on how exactly you build from source. A generic build was not the target. Andres Freund showed that the attack was targeted against a specific type of build system.
- zamalek 2y agoBuilding from git, or the github automatic tarball would have. The larger issue here is authenticating tarballs against the source.
- account42 2y agoThere is no reason to believe the exploit would have been spotted earlier had the attacker included the final part in git.
- deleted 2y ago[deleted]
- devttyeu 2y ago
- kosolam 2y agoJesus! Does anyone know if Debian stable is affected?
- ValdikSS 2y agoIt's not. Neither Ubuntu.
- anononaut 2y agoDo you have a source my friend? I thought Ubuntu was built off of Debian testing or unstable
- ValdikSS 2y agoThe latest version in 23.10 is 5.4.1-0.2 https://packages.ubuntu.com/mantic/liblzma5 https://packages.ubuntu.com/mantic/liblzma5 And in unreleased 24.04 is 5.4.5-0.3 https://packages.ubuntu.com/noble/liblzma5 https://packages.ubuntu.com/noble/liblzma5 There are no changelog entries indicating that the package was reverted.
- deleted 2y ago[deleted]
- djao 2y agoThe stable releases don't have this particular backdoor, but they're still using older versions of the library that were released by the same bad actor.
- bhaak 2y agoI looked at the differences between the GitHub repository and released packages. About 60 files are in a release package that are not in the repo (most are generated files for building) but also some of the .po files have changes. That's devastating. If you don't build your release packages from feeding "git ls-files" into tar, you are doing it wrong.
- icommentedtoday 2y agoWhy not `git archive`?
- bhaak 2y agoBecause I didn't know about it. Although if I look at its documentation, it's already a somewhat complicate invocation with unclear effects (lots of commandline options). Git seems to not be able to do KISS. git ls-files and tar is a simple thing everybody understands and can do without much issues.
- cryptonector 2y agohttps://news.ycombinator.com/item?id=39872062 https://news.ycombinator.com/item?id=39872062
- sesuximo 2y agoI think this is unfortunately very common practice
- oxymoron290 2y agoJai Tan's commit history on his github profile suggests he took off for Christmas, new years, and spring break. I smell an American.
- bloak 2y agoInteresting. Is there also a pattern in the times of day? (I don't so much mean the times in commits done by the developer because they can be fake. I'd be more interested in authentic times recorded by GitHub, if any such times are publicly accessible.) Another thing would be to examine everything ever written by the user for linguistic clues. This might point towards particular native languages or a particular variant of English or towards there being several different authors.
- bombcar 2y agoSomeone said commits lined up with Beijing time, but I've not verified that. But that wouldn't count for much, someone employed by anyone could work any hours.
- rany_ 2y agoAlso git actually stores the timezone information. You could see it is consistently China time (GMT+8). P.S. could be Taiwanese as China and Taiwan share the same timezone. Below are links to the git mailbox files where you could see the timezone. From 2022: - https://github.com/tukaani-project/xz/commit/c6977e7400088177556e8771bcb839eb7d90caa3.patch https://github.com/tukaani-project/xz/commit/c6977e740008817... - https://github.com/tukaani-project/xz/commit/7c16e312cb2f40b81154c0e5be13a3c6b8da485d.patch https://github.com/tukaani-project/xz/commit/7c16e312cb2f40b... From 2024: - https://github.com/tukaani-project/xz/commit/af071ef7702debef4f1d324616a0137a5001c14c.patch https://github.com/tukaani-project/xz/commit/af071ef7702debe... - https://github.com/tukaani-project/xz/commit/a4f2e20d8466369b1bb277c66f75c9e4ba9cc378.patch https://github.com/tukaani-project/xz/commit/a4f2e20d8466369...
- Perenti 2y ago
- returningfory2 2y agoAnother interesting data point: about 2 years ago there was a clear pressure campaign to name a new maintainer: https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h... At the time I thought it was just rude, but maybe this is when it all started.
- leosanchez 2y agoHow many people are involved in this ?
- masklinn 2y agoCould be just a single person with a bunch of identities.
- IshKebab 2y agoI would put money on government hackers. They're the sort of people that have the time to pull something like this off. Frankly I'm really surprised it isn't more common, though maybe it is and these guys were just super blatant. I would have expected more plausible deniability.
- JaDogg 2y agoGood cop bad cop play maybe.
- jamespo 2y ago"Jigar Kumar" seems to have disappeared
- Nathanba 2y agotrue, that is suspicious as well. A person that hasn't even created any bugs or issues suddenly has a big problem with the speed of development? Especially the way this was phrased: "You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?" "Why delay what your repo needs?" This sounds like scammer lingo
- MaximilianEmel 2y agoHas this affected OpenBSD at all?
- ikmckenz 2y agoSeems the backdoor relied on Debian and others patching their copies of openssh to support systemd notifications, and this would obviously not be the case on OpenBSD. To be sure the current ports version of xz is 5.4.5: https://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/ports/archivers/xz/Makefile?rev=1.50&content-type=text/plain https://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/ports/a... Although the maintainer was working on updating to 5.6.1, but this news broke before the diff was landed: https://marc.info/?l=openbsd-ports&m=171174441521894&w=2 https://marc.info/?l=openbsd-ports&m=171174441521894&w=2
- port443 2y agoI think its much more likely this was not a bad actor, given their long history of commits. It's a known fact that China will "recruit" people to operate them. A quote: > They talk to them, say my friend, I see you like our special menu. Are you from China? Are you here on a VISA? Do you have family back there? Would you like your family to stay alive? Is your loyalty to this temporary employer or is your loyalty to your motherland? You know, a whole bunch of stuff like that. That’s how Chinese intelligence operations acts... This just gives feelings of less "compromised account" and more "Your account is now our account"
- threeseed 2y agoIt's also a known fact that China will coerce people by threatening family and friends. Seen this happen to friends here in Australia who were attending pro-Taiwan protests.
- Johnny555 2y agoIsn't that still a "bad actor" even if they are coerced into it?
- foobiekr 2y agoYes.
- deleted 2y ago[deleted]
- Terr_ 2y agoFor the purposes of security discussions, I would say yes. You often don't know their real identity let alone their motivations and tribulations. However if we were critiquing characters in a book-- especially ones where narrative voice tells us exactly their true motivations--then maybe not, and they get framed as a "dupe" or "manipulated" etc.
- Almondsetat 2y ago"bad actor" doesn't mean "bad faith", it's not a value judgement
- k8svet 2y agoWait, I'm on mobile. Did this partially slip by because of the ABSURD PRACTICE of publishing release.tarballs that do not 1:1 correspond with source? Let me guess, autotools? I want to rage shit post but I guess I'll wait for confirmation first. EDIT: YUP, AT LEAST PARTIALLY. Fucking god damn autotools.
- deleted 2y ago[deleted]
- hypnagogic 2y agoBeen saying this the whole day now, GitHub really needs an automated diff / A/B check-up on tarballs against the actual repo, flag everything with at least a warning (+[insert additional scrutiny steps here]) when the tarball isn't matching the repo.
- 5p4n911 2y agoThe author (Jia Tan) also changed the xz.tukaani.org (actually the github.io, where the main contributor is, surprise, also them) release description to state all new releases are signed by their OpenPGP key. I'd guess that was one of the first steps to a complete project takeover. I hope Lasse Collin still has control of his accounts, though the CC on the kernel mailing list looks kind of suspicious to me.
- tree24005 2y ago[flagged]
- 0xthr0w4 2y agoOut of curiosity I looked at the list of followers of the account who committed the backdoor. Randomly picked https://github.com/Neustradamus https://github.com/Neustradamus and looked at all their contributions. Interestingly enough, they got Microsoft to upgrade ([0],[1]) `vcpkg` to liblzma 5.6.0 3 weeks ago. [0] https://github.com/microsoft/vcpkg/issues/37197 https://github.com/microsoft/vcpkg/issues/37197 [1] https://github.com/microsoft/vcpkg/pull/37199 https://github.com/microsoft/vcpkg/pull/37199
- sroussey 2y agoOMG: look at the other contributions. He is trying to take over projects and pushing some change to sha256 in a hundred projects. Example: https://github.com/duesee/imap-flow/issues/96 https://github.com/duesee/imap-flow/issues/96
- masklinn 2y agoThis guy's interactions seem weird but it might just be because of the non-native english or a strange attitude, or he's very good at covering his track e.g. found a cpython issue where he got reprimanded for serially opening issues: https://github.com/python/cpython/issues/115195#issuecomment-1935707908 https://github.com/python/cpython/issues/115195#issuecomment... But clicking around he seems to mostly be interacting with interest around these bits e.g. https://github.com/python/cpython/issues/95341#issuecomment-1802889364 https://github.com/python/cpython/issues/95341#issuecomment-... or pinging the entire python team to link to the PR... of a core python developer: https://github.com/python/cpython/issues/95341#issuecomment-1872618454 https://github.com/python/cpython/issues/95341#issuecomment-... If I saw that on a $dayjob project I'd pit him as an innocuous pain in the ass (overly excited, noisy, dickriding). Here's a PR from 2020 where he recommends / requests the addition of SCRAM to an SMTP client: https://github.com/marlam/msmtp/issues/36 https://github.com/marlam/msmtp/issues/36 which is basically the same thing as the PR you found. The linked documents seem genuine, and SCRAM is an actual challenge/response authentication method for a variety of protocols (in this case mostly SMTP, IMAP, and XMPP): https://en.wikipedia.org/wiki/Salted_Challenge_Response_Authentication_Mechanism https://en.wikipedia.org/wiki/Salted_Challenge_Response_Auth... Although, and that's a bit creepy, he shows up in the edition history for the SCRAM page, the edit mostly seem innocent though he does plug his "state of play" github repository.
- deleted 2y ago[deleted]
- gouggoug 2y agoList of pull request requesting the updating to liblzma 5.6.0 [0] I wonder what amount of scrutiny all the accounts that proposed the upgrade should be put under. [0] https://github.com/search?q=liblzma+5.6.0&type=pullrequests https://github.com/search?q=liblzma+5.6.0&type=pullrequests
- mikolajw 2y agoTukaani website states "jiatan" as the nickname of the malicious code committer on Libera Chat. WHOWAS jiatan provided me the following information: jiatan ~jiatan 185.128.24.163 * :Jia Tan jiatan 185.128.24.163 :actually using host jiatan jiatan :was logged in as jiatan tungsten.libera.chat :Fri Mar 14:47:40 2024 WHOIS yields nothing, the user is not present on the network at the moment. Given that 185.128.24.163 is covered with a range-block on the English Wikipedia, it appears this is a proxy.
- chrononaut 2y ago> it appears this is a proxy. Yes, that IP address appears associated with witopia[.]net, specifically vpn.singapore.witopia[.]net points to that IP address.
- circusfly 2y agoWaiting for the new YouTube videos on this. "Woah! Linux has a back door dudes!". My distribution, Ubuntu (now Kubuntu) 2022 isn't affected.
- fullstackchris 2y agonot sure why you're being downvoted. this is exactly what is going to happen.
- Lockal 2y agoStill better than TwoMinuteToiletPapers and other AI-bamboozled channels hyping over proprietary OpenAI crap (text/photo/video), what a time to be alive!
- autoexecbat 2y agoI'm really curious about if the act of injecting a backdoor into OSS software is legal/illegal ? Are they somehow in the clear unless we can show they actively exploited it?
- mnau 2y agoProbably depends on criminal code a country. Mine does (EU country): > Section 231 Obtaining and Possession of Access Device and Computer System Passwords and other such Data > (1) Whoever with the intent to commit a criminal offence of Breach of secrecy of correspondence [...] or a criminal offence of Unauthorised access to computer systems and information media [...] produces, puts into circulation, imports, exports, transits, offers, provides, sells, or otherwise makes available, obtains for him/herself or for another, or handles > a) a device or its component, process, instrument or any other means, including a computer programme designed or adapted for unauthorised access to electronic communications networks, computer system or a part thereof, or > b) a computer password, access code, data, process or any other similar means by which it is possible to gain access to a computer system or a part thereof, shall be sentenced .. (1 year as an individual, 3 years as a member of a organized group)
- Culonavirus 2y agoThe way I see it: People are being charged for their speech all the time. Especially outside the US, but even in the US. And code is speech. And that is even before all the hacking/cracking/espionage laws get involved. There's a reason all the (sane) people doing grey/black hat work take their security and anonymity extremely seriously.
- vhiremath4 2y agoMy favorite part was the analysis of "I'm not really a security researcher or reverse engineer but here's a complete breakdown of exactly how the behavior changes." You only get this kind of humility when you're working with absolute wizards on a consistent basis.
- rpigab 2y agoI'd love to be at Microsoft right now and have the power to review this user's connection history to Github, even though VPN exists, many things can be learned from connection habits, links to ISPs, maybe even guess if VPNs were used, roundtrip time on connections can give hints. I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.
- qecez 2y ago> I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets. Which is why there's probably nothing remotely interesting in them logs.
- mhh__ 2y agoIntelligence agencies get caught red handed all the time so I wouldn't be too sure. If it was an organised group I'm sure they were careful, of course, but it only takes one fuckup.
- megous 2y agoOh my, another reason not to use github. :D So many reasons poping up just in this comment section alone.
- alpb 2y agoThat’d be illegal for an employee to do.
- bananapub 2y agopeople are mis-reading the Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708 it wasn't the apparently newly-created identity "Hans Jansen" just asking for a new version to be uploaded, it was "Hans Jansen" providing a new version to be uploaded as a non-maintainer-upload - Debian-speak for "the maintainer is AWOL, someone else is uploading their package". if "Hans Jansen" is another attacker then they did this cleverly, providing the new - compromised - upstream tarballs in an innocent-looking way and avoiding anyone examining the upstream diff.
- vasili111 2y agoCould anyone please tell me if current stable version of Debian has that backdoor or not?
- yabones 2y agoDebian stable has 5.4.1, the backdoored versions are 5.6.0-5.6.1 https://packages.debian.org/bookworm/xz-utils https://packages.debian.org/bookworm/xz-utils https://packages.debian.org/bookworm/liblzma5 https://packages.debian.org/bookworm/liblzma5
- anononaut 2y agoDebian Stable seems to be in the clear. https://lists.debian.org/debian-security-announce/2024/msg00057.html https://lists.debian.org/debian-security-announce/2024/msg00...
- teddyh 2y agoIt does not contain the backdoor: <https://security-tracker.debian.org/tracker/CVE-2024-3094 https://security-tracker.debian.org/tracker/CVE-2024-3094<
- kazinator 2y agoDoesn't this call for criminal charges?
- mnau 2y agoGood luck finding him/her. GitHub probably already gave feds all logs and IPs, but I would bet 100:1 that it's all going to be a VPN or something like that.
- Unfrozen0688 2y ago[flagged]
- ozgune 2y agoI read through the entire report and it gradually got more interesting. Then, I got to the very end, saw Andres Freund's name, and it put a smile on my face. :) Who else would have run a PostgreSQL performance benchmark and discover a major security issue in the process?
- mik1998 2y agoPersonally, I use lzip ever since I read https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html Seems like the complexity of XZ has backfired severely, as expected.
- bananapub 2y ago> Seems like the complexity of XZ has backfired severely, as expected. this is a very bad reading of the current situation.
- ta8645 2y agoThis kind of shallow dismissal is really unhelpful to those of us trying to follow the argument. You take a tone of authoritative expert, without giving any illuminating information to help outsiders judge the merit of your assertion. Why is it a very bad reading of the current situation? What is a better reading?
- davispw 2y agoTo summarize the article, the back door is introduced through build scripts and binaries distributed as “test” data. Very little to do with the complexity or simplicity of xz; more that it was a dependency of critical system binaries (systemd) and ripe for hostile takeover of the maintainer role.
- supriyo-biswas 2y agoIntroducing a back door is not the same thing as a badly designed file format.
- bananapub 2y agoI am not sure I agree that every low quality post needs a detailed rebuttal? HN couldn't function under such rules. as to the specific comment: > Seems like the complexity of XZ has backfired severely, as expected. to summarise: someone found a project with a vulnerable maintenance situation, spent years getting involved in a project, then got commit rights, and then commited a backdoor in some binaries and the build system, then got sock puppets to agitate for OSes to adopt the backdoored code. the comment I replied to made a "shallow" claim of complexity without any details, so let's look at some possible interpretations: - code complexity - doesn't seem super relevant - the attacker hide a highly obfuscated backdoor in a binary test file and committed it - approximately no one is ever going to catch such things without a process step of requiring binaries be generatable in a reasonable-looking and hopefully-hard-to-backdoor kind of way. cryptographers are good at this: https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number - build complexity - sure, but it's auto*, that's very common. - organisational complexity - the opposite is the case. it had one guy maintaining it, who asked for help. - data/file format complexity - doesn't seem relevant unless it turns out the obfuscation method used was particularly easy for this format, but even in that case, you'd think others would be vulnerable to something equivalent perhaps OP had some other thing in mind, but then they could have said that, instead of making a crappy comment.
- lacoolj 2y agoWhat a disappointment. It's something always in the back of our minds as developers using public libraries, but when something like this happens, non-developers that hear about it start to associate it with the rest of the open-source community. It's essentially a terrorist attack on developer experience. Thankfully, management doesn't follow the same approach as the TSA.
- Zigurd 2y ago"Lasse Collin," as other posters here have found, does not seem to exist as an experienced coder. Oddly, there is a Swedish jazz musician named Lasse Collin, which would otherwise be one of those names, especially the last name, that would stick out. Instead it is buried under a lot of mentions of a musician.
- rany_ 2y agoSearching for my real name on Google doesn't return anything either, I don't think this means anything.
- Zigurd 2y agoLasse Collin the contributor is findable, especially if you add "tukaani" to the search. But not in any other context, unless that's what old jazz musicians do in their retirement.
- rany_ 2y agoI don't think that's what they meant. The idea is to find information about their personal life, not OSS contributions. Something that proves they're a real person.
- akyuu 2y agoLasse Collin has been working on xz for decades: https://sourceforge.net/p/sevenzip/discussion/45797/thread/09814bb2/?limit=25#d81a https://sourceforge.net/p/sevenzip/discussion/45797/thread/0... Now, whether his GitHub account is currently being controlled by him is another question. Also, for some more context: In 2022, Lasse said he was struggling to work on xz and was looking for maintainers, and mentioned Jia Tan: https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...
- nateskulic 2y agoFairly deep bugs for a Bazaar.
- alathers 2y agoThank the gods I didn't plan on having a life this weekend
- userbinator 2y agoLooking at how many requests to update to the backdoored version have been made, I wonder if the fact that many people (including developers) have been conditioned to essentially accept updates as "always-good" is a huge contributing factor in how easy it is to spread something like this. The known unknowns can be better than the unknown unknowns.
- frenchman99 2y agoTotally agree. With things like Dependabot encouraged by GitHub, people now get automated pull requests for dependency updates, increasing the speed of propagation of such vulnerabilities.
- kapouer 2y agoBoth https://github.com/tukaani-project https://github.com/tukaani-project members accounts have been suspended. (to see that, you can list the followers of each account).
- dmarto 2y agoKinda relevant, as I saw few comments about how safer languages are the solution. Here[0] is a very simple example, that shows how easy such supply chain attacks are in Rust; and lets not forget that there was a very large python attack just a few days ago[1]. [0] - https://github.com/c-skills/rust1 https://github.com/c-skills/rust1 [1] - https://checkmarx.com/blog/over-170k-users-affected-by-attack-using-fake-python-infrastructure/ https://checkmarx.com/blog/over-170k-users-affected-by-attac...
- mrcus 2y agoI am very concerned about Rust. Rust’s “decision” to have a very slim standard library has advantages, but it severely amplifies some other issues. In Go, I have to pull in zero dependencies to make an HTTP request. In Rust, pulling reqwest pulls in at least 30 distinct packages (https://lib.rs/crates/reqwest https://lib.rs/crates/reqwest). Date/time, “basic” base64, common hashing or checksums, etc, they all become supply chain vectors. The Rust ecosystem’s collective refusal to land stable major versions is one of the amplifying issues. “Upgrade fatigue” hits me, at least. “Sure, upgrade ring to 0.17” (which is effectively the 16th major version). And because v0.X versions are usually incompatible, it’s not really possible to opt not to upgrade, because it only takes a short while before some other transitive dependency breaks because you are slow to upgrade. I recently spent a while writing my code to support running multiple versions of the `http` library, for example (which, to be fair, did just land version 1.0). My NATS library (https://lib.rs/crates/async-nats https://lib.rs/crates/async-nats) is at version 34. My transitive base64 dependency is at version 22 (https://lib.rs/crates/base64 https://lib.rs/crates/base64). This makes it nearly impossible for me to review these libraries and pin them, because if I pin foo@0.41.7, and bar needs foo@0.42.1, I just get both. bar can’t do =>0.41, because the point of the 0.X series is that it is not backwards compatible. It makes this process so time consuming that I expect people will either just stop (as if they did) reviewing their dependencies, or accept that they might have to reinvent everything from URL parsing to constructing http headers or doing CRC checks. Combine this with a build- and compile-time system that allows completely arbitrary code execution, which is routinely just a wrapper for stuff like in the zx attack (look at a lot of the low-level libs you inevitably pull in). Sure, the build scripts and the macro system enables stuff like the amazing sqlx library, but said build and macro code is already so hard to read, it really takes proper wizardry to properly understand.
- mrcoffee4u 2y agocan someone ELI5 ?
- QuantumG 2y agoHouse of cards experiences strong wind.
- deleted 2y ago[deleted]
- jchoksi 2y agoThe two active maintainers seem to be: Lasse Collin <lasse.collin@tukaani.org> and Jia Tan <jiat0218@gmail.com> Searching DDG for "jiat0218" I came across a blog post which I found weird. Seems to be dated: 2006-05-03 Blog post: "Kuso拍賣．有靈氣的筷子 - 闕小豪" <https://char.tw/blog/post/24397301 https://char.tw/blog/post/24397301> Internet Archive link: <https://web.archive.org/web/20240329182713/https://char.tw/blog/post/24397301 https://web.archive.org/web/20240329182713/https://char.tw/b...> The contents of the page when translated seems to be about jiat0218 auctioning a pair of spiritual chopsticks as a prank. The blog entry is basically a QA between jiat0218 and various other people about these chopsticks. If Jia Tan does turn out to be a compromised maintainer working for a state actor then some of the content on the blog page can be viewed in a more sinister way (i.e. spycraft / hacks for sale etc.). Example question 38: Question 38 accounta066 (3): Are these chopsticks really that good? I kind of want to buy them! But I recently sent money for online shopping but didn’t receive anything. It’s very risky; currently jiat0218 you don’t have any reviews, you can interview me. Do you want to hand it over?! … A sincere buyer will keep it. Reply to jiat0218 (4): First of all, I would like to express my condolences to you for your unfortunate experience! What can I say about this kind of thing...My little sister has always been trustworthy. What’s more, this is a pair of spiritual chopsticks, so I hope to have a good one. It’s the beginning! As you can see, my little sister is very careful and takes her time when answering your questions. Except for the two messages that were accidentally deleted by her, she always answers your questions. If this still doesn’t reassure you, then I can only say that I still have room to work hard. You are still welcome to bid... ^_^ Note however, it could all just be what it purports to be which is a prank auction of spiritual chopsticks.
- fragmede 2y agoChopsticks could also be a codeword for something. Maybe some sort a backdoor into a system somewhere.
- alwayslikethis 2y agoThis is likely just a coincidence. 0218 looks like a birthday and jiat is probably the name + initial. 18 years is also too long of a time horizon for this.
- dboreham 2y agoSomething about this I found surprising is that Linux distros are pulling and packaging pre-built binaries from upstream projects. I'd have expected them to build from source.
- richardwhiuk 2y agoThey were pulling a tarball from upstream and building it - the tarball was compromised.
- Lockal 2y agoThe answer is not complete. There were 2 ways to pull sources: bad - https://github.com/tukaani-project/xz/releases/download/ https://github.com/tukaani-project/xz/releases/download/... or: good - https://github.com/tukaani-project/xz/archive/refs/tags/ https://github.com/tukaani-project/xz/archive/refs/tags/... Specifically in Gentoo, there is a note in https://github.com/gentoo/gentoo/blob/master/app-arch/xz-utils/xz-utils-9999.ebuild https://github.com/gentoo/gentoo/blob/master/app-arch/xz-uti... # Remember: we cannot leverage autotools in this ebuild in order # to avoid circular deps with autotools Namely, to unpack autoconf-2.72e.tar.xz from gnu.org you need xz-tools. And this is just the shortest circle. It is not very common, but xz-utils was one of few rare cases where regeneration of autohell files was considered as unnecessary complication (it backfired).
- dpkirchner 2y agoUnfortunately, those GitHub links are no longer valid, so we randos can't use them to learn what went wrong here. Hopefully GH will reverse this decision once the dust settles.
- NekkoDroid 2y agoThe gist of it is: The "good" one is the auto generated "Source code" releases made by github. The "bad" one is a manually generated and uploaded source code release, which can have whatever you want.
- haolez 2y agoI'm not trying to troll, but I'm wondering if a distro like Gentoo is less susceptible to such attacks, since the source code feels more transparent with their approach. But then again, it seems that upstream was infected in this case, so I'm not sure if a culture of compiling from source locally would help.
- StressedDev 2y agoIt is not going to make a difference. If you run malicious code, you will get hacked. Compiling the code yourself does not prevent the code from being malicious. The one it might help is it might make it easier to find the back door once you know there is one.
- stephc_int13 2y agoI guess that rewriting liblzma in Rust would not have prevented this backdoor. But would have likely increased the confidence in its safety. Using the build system (and potentially the compiler) to insert malicious backdoors is far from a new idea, and I don't see why this example would the only case.
- yencabulator 2y agoThe backdoor hinged on hiding things in large shell scripts, obscure C "optimizations", and sanitizer disabling. I'd expect all of those would be a much bigger red flag in the Rust world.
- anonymous-panda 2y agoDon’t know all the details and rust isn’t immune to a build attack, but stuff like that tends to stand out a lot more I think in a build.rs than it would in some m4 automake soup.
- minetest2048 2y agoThere was a drama back then where serde tried to ship its derive macro as a precompiled binary: https://news.ycombinator.com/item?id=37189462 https://news.ycombinator.com/item?id=37189462
- nullifidian 2y agoIt would have made it worse, because there would be 300 crates with 250 different maintainers, all pulled in by several trivial/baseline dependencies. More dependencies = higher the probability that a malicious maintainer has gotten maintainer's rights for one of them, especially because many original authors/maintainers of rust style microdepencency crates move on with their lives and eventually seek to exit their maintainer role. At least for classic C/C++ software, by the virtue of it being very inconvenient to casually pull 300 dependencies for something trivial, there are fewer dependencies, i.e. separate projects/repos, and these tend to be more self-contained. There are also "unserious" distributions like Fedora and something like stable/testing/unstable pipeline in Debian, which help with catching the most egregious attempts. Crates.io and npm are unserious by their very design, which is focused on maximizing growth by eliminating as many "hindrances" as possible.
- dang 2y agoRelated ongoing threads: Xz: Disable ifunc to fix Issue 60259 - https://news.ycombinator.com/item?id=39869718 https://news.ycombinator.com/item?id=39869718 FAQ on the xz-utils backdoor - https://news.ycombinator.com/item?id=39869068 https://news.ycombinator.com/item?id=39869068 Everything I Know About the XZ Backdoor - https://news.ycombinator.com/item?id=39868673 https://news.ycombinator.com/item?id=39868673
- notmysql_ 2y agoInterestingly on of the accounts that the GitHub account who introduced the backdoor follows was suspended very recently [1] who is also part of the org who runs XZ [1] https://github.com/JiaT75?tab=following https://github.com/JiaT75?tab=following
- rany_ 2y agoThat JiaT75 account is also suspended, if you check https://github.com/Larhzu?tab=following https://github.com/Larhzu?tab=following you'll see that they're suspended as well. It's pretty weird that it's that hard to find out whether a user is suspended.
- deleted 2y ago[deleted]
- fullstackchris 2y agopRoBaBlY a StaTe AcToR zero definition of what that means... egos of people who just like to say cool words they don't understand lol this comment will probably get deleted, but let the action of this comment being deleted stand that in 2024 we're all allowed to use big words with no definition of what they mean -> bad state actor? who? what motive? what country? all comments involving "state actor" are very broad and strange... i would like people to stop using words that have no meaning, as it really takes away from the overall conversation of what is going on. i mean you're seriously going to say "state actor playing the long game" to what end? the issue was resolved in 2 hours... this is stupid
- Hackbraten 2y agoFor starters, the backdoor was technically really sophisticated. For example, the malicious code circumvents a hardening technique (RELRO) in a clever way, which would otherwise have blocked it from manipulating the sshd code in the same process space at runtime. This is not something that script kiddies usually cook up in an afternoon to make a quick buck. You need experts and a lot of time to pull off feats like that. This points to an organization with excellent funding. I’m not surprised at all that people are attributing this to some unknown nation-level group.
- dlenski 2y agoA lot of software (including https://gitlab.com/openconnect/openconnect https://gitlab.com/openconnect/openconnect of which I'm a maintainer) uses libxml2, which in turn transitively links to libzma, using it to load and store compressed XML. I'm not *too* worried about OpenConnect given that we use `libxml2` only to read and parse uncompressed XML… But I am wondering if there has been any statement from libxml2 devs (they're under the GNOME umbrella) about potential risks to libxml2 and its users.
- bananapub 2y ago> only to read and parse uncompressed XML… how does libxml2 know to decompress something? does it require you, as the caller, to explicitly tell it to? or does it look at the magic bytes or filename or mimetype or something?
- dlenski 2y ago> how does libxml2 know to decompress something? > > does it require you, as the caller, to explicitly tell it to? In the entry point/function that we use, `xmlReadMemory` (https://gnome.pages.gitlab.gnome.org/libxml2/devhelp/libxml2-parser.html#xmlReadMemory https://gnome.pages.gitlab.gnome.org/libxml2/devhelp/libxml2...), it doesn't handle compressed XML at all. But there are indeed others where it attempts to auto-detect compression, although as I understand it from the docs only ZLib compression is autodetected… though I suspect these may be out-of-date and it may autodetect any/all compiled -in compression algorithms. Regardless, the fact that it links with liblzma is cause for concern, given the mechanism of operation of the liblzma/xz backdoor.
- enedil 2y agoThis doesn't matter, if libxml2 loads .so and the library is malicious, you are already potentially compromised, as it is possible to run code on library load.
- 0x0 2y agoInteresting commit in January where the actual OpenPGP key was changed: https://github.com/tukaani-project/tukaani-project.github.io/commit/76f58f1e14e44d69f712841090d331437c187b30 https://github.com/tukaani-project/tukaani-project.github.io...
- illusive4080 2y agoGitHub suspended this project
- gertvdijk 2y agoThey just signed each other's keys around that time, and one needs to redistribute the public keys for that; nothing suspicious about it I think. The key fingerprint 22D465F2B4C173803B20C6DE59FCF207FEA7F445 remained the same. before: pub rsa4096/0x59FCF207FEA7F445 2022-12-28 [SC] [expires: 2027-12-27] 22D465F2B4C173803B20C6DE59FCF207FEA7F445 uid Jia Tan <jiat0218@gmail.com> sig 0x59FCF207FEA7F445 2022-12-28 [selfsig] sub rsa4096/0x63CCE556C94DDA4F 2022-12-28 [E] [expires: 2027-12-27] sig 0x59FCF207FEA7F445 2022-12-28 [keybind] after: pub rsa4096/0x59FCF207FEA7F445 2022-12-28 [SC] [expires: 2027-12-27] 22D465F2B4C173803B20C6DE59FCF207FEA7F445 uid Jia Tan <jiat0218@gmail.com> sig 0x59FCF207FEA7F445 2022-12-28 [selfsig] sig 0x38EE757D69184620 2024-01-12 Lasse Collin <lasse.collin@tukaani.org> sub rsa4096/0x63CCE556C94DDA4F 2022-12-28 [E] [expires: 2027-12-27] sig 0x59FCF207FEA7F445 2022-12-28 [keybind] Lasse's key for reference: pub rsa4096/0x38EE757D69184620 2010-10-24 [SC] [expires: 2025-02-07] 3690C240CE51B4670D30AD1C38EE757D69184620 uid Lasse Collin <lasse.collin@tukaani.org> sig 0x38EE757D69184620 2024-01-08 [selfsig] sig 0x59FCF207FEA7F445 2024-01-12 Jia Tan <jiat0218@gmail.com> sub rsa4096/0x5923A9D358ADF744 2010-10-24 [E] [expires: 2025-02-07] sig 0x38EE757D69184620 2024-01-08 [keybind]
- MaximilianEmel 2y agoWe need to get these complex & bloated build-systems under control.
- 77pt77 2y agoWhat we need is to move away from 1970s build tools.
- 0x0 2y agoAll these older (4.x, 5.0.x etc) releases that were suddenly uploaded a few months ago should probably also be considered suspect: https://github.com/tukaani-project/tukaani-project.github.io/commit/071a8acf097e11b659f6d7eb0f3b2e5e8934de0e https://github.com/tukaani-project/tukaani-project.github.io...
- shortsunblack 2y agoPretty much proof that OSS != automatically more secure. And proof that OSS projects can get backdoored. See this for more ideas on this issue: https://seirdy.one/posts/2022/02/02/floss-security/ https://seirdy.one/posts/2022/02/02/floss-security/
- derkades 2y agoThe malware was hidden inside an opaque binary. If anything, this shows that we need more open source and more reproducibility.
- llmblockchain 2y agoWas Debian 12/stable unaffected? Only sid?
- joshhansen 2y agoMy understanding is that that's correct. I'm on fully upgraded stable (Debian 12) and my xz is 5.4.2 and liblzma as well.
- neoneye2 2y agoDamn. I'm on macOS and use homebrew. To my surprise I had "xz" version 6.5.1 installed on my computer! I ran "brew upgrade" and that downgraded to version 5.4.6.
- xvilka 2y agoMaybe it's finally time to start sunsetting LZMA and xz all together in favor of newer algorithms like Zstandard that also offer better performance but compression rates on par with LZMA.
- illusive4080 2y agoYes but don’t start thinking they’re immune to compromise
- xvilka 2y agoNobody is. But it's a great opportunity window.
- mrbluecoat 2y ago> I am *not* a security researcher, nor a reverse engineer. Could have fooled me - impressive write-up!
- afh1 2y agoPotentially malicious commit by same author on libarchive: https://github.com/libarchive/libarchive/pull/1609 https://github.com/libarchive/libarchive/pull/1609
- andix 2y agoIs there already a list of distributions that included the affected versions in non-prereelase channels?
- illusive4080 2y agoNone that I could find have included it. Not even NixOS 23.11.
- devttyeu 2y agoWouldn’t be surprised that the ssh auth being made slower was deliberate - that makes it fairly easy to index all open ssh servers on the internet, then to see which ones get slower to fail preauth as they install the backdoor
- A1kmm 2y agoLooks like GitHub has suspended access to the repository, which while it protects against people accidentally compiling and using the code, but certainly complicates forensic analysis for anyone who doesn't have a clone or access to history (which is what I think a lot of people will be doing now to understand their exposure).
- gpm 2y agoWell that's inconvenient, I was (probably, time permitting) going to propose to some of my friends that we attempt to reverse this for fun tomorrow. Anyone have a link to the git history? I guess we can use the ubuntu tarball for the evil version.
- A1kmm 2y agoIt looks like git clone https://git.tukaani.org/xz.git https://git.tukaani.org/xz.git still works for now (note: you will obviously be cloning malware if you do this) - that is, however, trusting the project infrastructure that compromised maintainers could have had access to, so I'm not sure if it is unmodified. HEAD (git rev-parse HEAD) on my result of doing that is currently 0b99783d63f27606936bb79a16c52d0d70c0b56f, and it does have commits people have referenced as being part of the backdoor in it.
- gowthamgts12 2y ago> https://git.tukaani.org/xz.git https://git.tukaani.org/xz.git it's throwing 403 now.
- amszmidt 2y agoWorks cloning though.
- gpm 2y agoApparently there's a wayback machine for git repos and it "just coincidentally" archived this repo the day before the news broke: https://archive.softwareheritage.org/browse/origin/visits/?origin_url=https://github.com/tukaani-project/xz https://archive.softwareheritage.org/browse/origin/visits/?o...
- sn 2y agoFor bad-3-corrupt_lzma2.xz, the claim was that "the original files were generated with random local to my machine. To better reproduce these files in the future, a constant seed was used to recreate these files." with no indication of what the seed was. I got curious and decided to run 'ent' https://www.fourmilab.ch/random/ https://www.fourmilab.ch/random/ to see how likely the data in the bad stream was to be random. I used some python to split the data into 3 streams, since it's supposed to be the middle one that's "bad": I used this regex to split in python, and wrote to "tmp": re.split(b'\xfd7zXZ', x) I manually used dd and truncate to strip out the remaining header and footer according to the specification, which left 48 bytes: $ ent tmp2 # bad file payload Entropy = 4.157806 bits per byte. Optimum compression would reduce the size of this 48 byte file by 48 percent. Chi square distribution for 48 samples is 1114.67, and randomly would exceed this value less than 0.01 percent of the times. Arithmetic mean value of data bytes is 51.4167 (127.5 = random). Monte Carlo value for Pi is 4.000000000 (error 27.32 percent). Serial correlation coefficient is 0.258711 (totally uncorrelated = 0.0). $ ent tmp3 # urandom Entropy = 5.376629 bits per byte. Optimum compression would reduce the size of this 48 byte file by 32 percent. Chi square distribution for 48 samples is 261.33, and randomly would exceed this value 37.92 percent of the times. Arithmetic mean value of data bytes is 127.8125 (127.5 = random). Monte Carlo value for Pi is 3.500000000 (error 11.41 percent). Serial correlation coefficient is -0.067038 (totally uncorrelated = 0.0). The data does not look random. From https://www.fourmilab.ch/random/ https://www.fourmilab.ch/random/ for the Chi-square Test, "We interpret the percentage as the degree to which the sequence tested is suspected of being non-random. If the percentage is greater than 99% or less than 1%, the sequence is almost certainly not random. If the percentage is between 99% and 95% or between 1% and 5%, the sequence is suspect. Percentages between 90% and 95% and 5% and 10% indicate the sequence is “almost suspect”."
- supriyo-biswas 2y agoNow to be fair, such an archive could have been created with a “store” level of compression that doesn’t actually perform any compression.
- 2y ago
- CGamesPlay 2y agoWhy has Github disabled the (apparently official) xz repository, but left the implicated account open to the world? It makes getting caught up on the issue pretty difficult, when GitHub has revoked everyone's access to see the affected source code. https://github.com/tukaani-project/xz https://github.com/tukaani-project/xz vs https://github.com/JiaT75 https://github.com/JiaT75
- dzaima 2y agoThe account has been suspended for a while, but for whatever reason that's not displayed on the profile itself (can be seen at https://github.com/Larhzu?tab=following https://github.com/Larhzu?tab=following). Repo being disabled is newer, and, while annoying and realistically likely pointless, it's not particularly unreasonable to take down a repository including a real backdoor.
- account42 2y agoTaking down the repo prevents more people inadvertendly pulling and building the backdoor so that makes sense. They should have immediately rehosted and archived the state at a different URL which makes it clear to not use it.
- betaby 2y agoHow that backdoor is triggered and what exactly it does?
- west0n 2y agoIt seems that to counter this type of supply chain attack, the best practices for managing software dependencies are to pin the version numbers of dependencies instead of using `latest`, and to use static linking instead of dynamic linking.
- xyzzy_plugh 2y agoThis gist summarizes the current situation very well: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27 https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b... Definitely looking like they were most likely some sort of state actor. This is very well done and all in plain sight. It's reassuring that it was discovered but given a simple audit of the release build artifacts would have raised alarms, how prevalent is this behavior in other projects? Terrifying stuff.
- Ansaben10 2y ago[flagged]
- eairy 2y ago[flagged]
- Brian_K_White 2y agoIt doesn't really relate to this issue other than that both issues share a common source, but I wish we'd never fallen for xz. I agree with the lzip guy https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
- bitwize 2y agoLooks like Jonathan Blow was right about open source.
- Jakesben10 2y ago[flagged]
- fwungy 2y agoBrain fart: would it be possible to attach passwords to a crypto based micro transaction such that every time you attempted a password entry your crypto account was charged a small fee for the login attempt? This would thwart brute force attacks, but not be a significant cost for users. If you could attach your login to the crypto account it would mean the account would have to be funded to allow the attempt. The token wouldn't store passwords it would just be a gatekeeper to the login attempt. The fees would be paid to the service providers as mining fees. E.g. foo@bar.com needs a password and a token provided from a designated crypto address to gain access to the service.
- byearthithatius 2y agoI hope mainstream news cover this so the general population can understand the issue with our software ecoysystems reliance on unpaid open-source maintainers
- account42 2y agoI worry the mainstream news take would just be "open source bad, microsoft closed source and google cloud good"
- korginator 2y agoxz is so pervasive, I just discovered on my Mac that the (affected?) version 5.6.1 made it into homebrew. The post in the linked article says that only Linux x86-64 systems are affected, but now I'm left scratching my head whether my Mac is also in trouble, just that we don't know it yet.
- BobbyTables2 2y agoWhy doesn’t GitHub force “releases” to be a simple repo tarball for sources and with binaries from GitHub actions or such… I find it incredibly ironic that a “version control” site gives no assurance of reproducible builds (nor reproducible source!!) The real villain is not the perpetrator, it is Microsoft, and it is all of us.
- cryptonector 2y agoBecause then for autoconf codebases you have to commit `./configure` or you have to require that users have autoconf installed and run `autoreconf -fi` first. Maybe autoconf-using projects should really just require that users have autoconf installed. Not that that would prevent backdoors, mind you.
- dpkirchner 2y agoIf committing configure is objectionable, perhaps there could be "service" repositories that are not directly writable and are guaranteed to be nothing more than the base repo + autoconf cruft used to generate the releases.
- cryptonector 2y agoWell, for example in jq we do commit bison/flex outputs because for users ensuring that they have the right version of those can be tricky. We could do the same w.r.t. autoconf and its outputs, though again, that won't preclude backdoors.
- dpkirchner 2y agoYeah, it's less about detecting backdoors specifically and more about having a way to compare releases to build jobs.
- cryptonector 2y agoCommitting built artifacts presents similar problems: how do you know that the committed artifacts are in fact derived from their sources? Or from non-backdoored versions of build tools for that matter? Hello Ken Thompson attacks. I don't believe there's a nice easy answer to these questions. What we do in jq is rely on GitHub Actions to run the build and `make dist`. In fact, we could now stop committing the bison/flex outputs, too, since we can make sure that the tarball includes them. We do also publish the git repo snapshots that GitHub auto-generates for releases, though we do that because GitHub doesn't give one a choice.
- 65a 2y agoIs there a proper reverse engineering of the payload yet?
- 17e55aab 2y agoa user offered 5.6.0 and 5.4.5 in an issue to microsoft/vcpkg 5.4.5 can be compromised https://github.com/microsoft/vcpkg/issues/37197 https://github.com/microsoft/vcpkg/issues/37197
- secondary_op 2y agoGithub making suspect repository private and hiding recent account activity is wrong move and is interfering with citizens investigation efforts.
- frenchman99 2y agoGoing forward this will require more than a citizens investigation. Law enforcement will surely be granted access. Also, tarballs are still available in package managers if you really want to dig into the code.
- zamalek 2y agoIt's a crime scene. It effectively has the "police" yellow tape around it.
- BarbaryCoast 2y agoThere's a bug in the detection script. The line: if [ "$path" == "" ] should be if [ "$path" = "" ]
- dualbus 2y agoBash accepts both variants of the equality operator. So it is not a bug.
- kn100 2y agoHere's a handy bash script I threw together to audit any docker containers you might be running on your machine. It's hacky, but will quickly let you know what version, if any, of xz, is running in your docker containers. ``` #!/bin/bash # Get list of all running Docker containers containers=$(docker ps --format "{{.Names}}") # Loop through each container for container in $containers; do # Get container image image=$(docker inspect --format='{{.Config.Image}}' "$container") # Execute xz --version inside the container version=$(docker exec "$container" xz --version) # Write container name, image, and command output to a text file echo "Container: $container" >> docker_container_versions.txt echo "Image: $image" >> docker_container_versions.txt echo "xz Version:" >> docker_container_versions.txt echo "$version" >> docker_container_versions.txt echo "" >> docker_container_versions.txt done echo "Output written to docker_container_versions.txt" ```
- Rhea_Karty 2y agoNotes on time stamps and time zones. A few interesting bits that I haven't fully fleshed out. TLDR: Some people have been throwing around that Jia is from “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone. The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=2021-12-31 https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps. Observation 1: Time zone basic analysis Here is the data on Jia’s time zone and the number of times he was recorded in that time zone: 3: + 0200 (in winter: February and November) 6: +0300 (in summer: in Jun, Jul, early October) 440: +0800 1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert. 2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March). 1. Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen… Observation 2: Time zone inconsistencies Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe. Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but a flight from China to anywhere in Eastern Europe is at a min 10 hours Jia Tan Thu, 5 May 2022 20:53:42 +0800 Jia Tan Sat, 19 Nov 2022 23:18:04 +0800 Jia Tan Mon, 7 Nov 2022 16:24:14 +0200 Jia Tan Sun, 23 Oct 2022 21:01:08 +0800 Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1) Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1) Jia Tan Wed, 5 Oct 2022 23:54:12 +0800 Jia Tan Wed, 5 Oct 2022 20:57:16 +0800 —> again, given the flight time, this is even more impossible Jia Tan Fri, 2 Sep 2022 20:18:55 +0800 Jia Tan Thu, 8 Sep 2022 15:07:00 +0300 Jia Tan Mon, 25 Jul 2022 18:30:05 +0300 Jia Tan Mon, 25 Jul 2022 18:20:01 +0300 Jia Tan Fri, 1 Jul 2022 21:19:26 +0800 Jia Tan Thu, 16 Jun 2022 17:32:19 +0300 Jia Tan Mon, 13 Jun 2022 20:27:03 +0800 —> the ordering of these time stamps and the switching back and forth between time zones looks strange. Jia Tan Thu, 15 Feb 2024 22:26:43 +0800 Jia Tan Thu, 15 Feb 2024 01:53:40 +0800 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Tue, 13 Feb 2024 22:38:58 +0800 —> this travel time is possible, but the duration of stay is unlikely Observation 3: Strange record of time stamps It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering. Observation 4: Bank holiday inconsistencies We notice that Jia’s work schedule and holidays seems to align much better with an Eastern European than a Chinese person. Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bank holidays:(https://www.bankofchina.co.id/en-id/service/information/latest-news/2022/public-holidays-in-china-hk-and-the-us-in-2023.html https://www.bankofchina.co.id/en-id/service/information/late...) Chinese bank holidays (just looking at 2023): - Working on 2023, 29 September: Mid Autumn Festival - Working on 2023, 05 April: Tomb Sweeping Day - Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year Eastern European holidays: - Never working on Dec 25: Christmas (for many EET countries) - Never working Dec 31 or Jan 1: New Years Observation 5: Little weekend work —> salary job? The most common working days for Jia were Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night. These times also line up well with Hans Jansen and Lasse Collin. I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.
- Rhea_Karty 2y agoTLDR: Some people have been throwing around “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone. These are my notes on time stamps/zones. There are a few interesting bits that I haven't fully fleshed out. The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=2021-12-31 https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps. Observation 1: Time zone basic analysis Here is the data on Jia’s time zone and the number of times he was recorded in that time zone: 3: + 0200 (in winter: February and November) 6: +0300 (in summer: in Jun, Jul, early October) 440: +0800 1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert. 2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March). Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen… Observation 2: Time zone inconsistencies Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe. Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but flight from China to anywhere in Eastern Europe is at a min 10 hours Jia Tan Thu, 5 May 2022 20:53:42 +0800 Jia Tan Sat, 19 Nov 2022 23:18:04 +0800 Jia Tan Mon, 7 Nov 2022 16:24:14 +0200 Jia Tan Sun, 23 Oct 2022 21:01:08 +0800 Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1) Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1) Jia Tan Wed, 5 Oct 2022 23:54:12 +0800 Jia Tan Wed, 5 Oct 2022 20:57:16 +0800 —> again, given the flight time, this is even more impossible Jia Tan Fri, 2 Sep 2022 20:18:55 +0800 Jia Tan Thu, 8 Sep 2022 15:07:00 +0300 Jia Tan Mon, 25 Jul 2022 18:30:05 +0300 Jia Tan Mon, 25 Jul 2022 18:20:01 +0300 Jia Tan Fri, 1 Jul 2022 21:19:26 +0800 Jia Tan Thu, 16 Jun 2022 17:32:19 +0300 Jia Tan Mon, 13 Jun 2022 20:27:03 +0800 —> the ordering of these time stamps, and the switching back and forth looks strange. Jia Tan Thu, 15 Feb 2024 22:26:43 +0800 Jia Tan Thu, 15 Feb 2024 01:53:40 +0800 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Tue, 13 Feb 2024 22:38:58 +0800 —> this travel time is possible, but the duration of stay is unlikely Observation 3: Strange record of time stamps It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering. Observation 4: Bank holiday inconsistencies We notice that Jia’s work schedule and holidays seem to align much better with an Eastern European than a Chinese person. Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bak holidays:(https://www.bankofchina.co.id/en-id/service/information/latest-news/2022/public-holidays-in-china-hk-and-the-us-in-2023.html https://www.bankofchina.co.id/en-id/service/information/late...) Chinese bank holidays (just looking at 2023): - Working on 2023, 29 September: Mid Autumn Festival - Working on 2023, 05 April: Tomb Sweeping Day - Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year Eastern European holidays: - Never working on Dec 25: Christmas (for many EET countries) - Never working Dec 31 or Jan 1: New Years Observation 5: No weekend work —> salary job? The most common working days for Jia was Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to be EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night. These times also line up well with Hans Jansen and Lasse Collin. I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.
- deleted 2y ago[deleted]
- CanaryLayout 2y agoWell isn't this an interesting commit. He finished his inject macro to compose the payload at build, so now he can start clearing up the repo so none of that shit gets seen when cruising through it. https://git.tukaani.org/?p=xz.git;a=commitdiff;h=4323bc3e0c1e1d2037d5e670a3bf6633e8a3031e;hp=5394a1665b7a108a54cb8b4ef3ebe59d3dbcca3a https://git.tukaani.org/?p=xz.git;a=commitdiff;h=4323bc3e0c1...
- astrange 2y agoThat's not what gitignore does. I can't think of a way it would let you hide this exploit.
- zeehio 2y agoOn Ubuntu there is a bug report asking to sync the 5.6 version from Debian experimental https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2055422 https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2055...
- Retr0id 2y agoThe `pack`[0] compression utility that reached the HN front page the other day[1] is setting off my alarm bells right now. (It was at the time too, but now doubly so) It's written in Pascal, and the only (semi-)documented way to build it yourself is to use a graphical IDE, and pull in pre-compiled library binaries (stored in the git repo of a dependency which afaict Pack is the only dependent of - appears to be maintained by the same pseudonymous author but from a different account). I've opened an issue[2] outlining my concerns. I'm certainly not accusing them of having backdoored binaries, but if I was setting up a project to be deliberately backdoorable, it'd look a lot like this. [0] https://pack.ac/ https://pack.ac/ [1] https://news.ycombinator.com/item?id=39793805 https://news.ycombinator.com/item?id=39793805 [2] https://github.com/PackOrganization/Pack/issues/10 https://github.com/PackOrganization/Pack/issues/10
- jum4 2y agoMaybe @JiaT75 got forced to do it. Maybe someone has more personal contact with him and can check how he is doing.
- rossant 2y agoIncredible. It's like discovering your colleague for 2 years at the secret nuclear weapon facility is a spy for another country, covering his tracks until the very last minute. Feels like a Hollywood movie is coming up. Should we start doing background checks on all committers to such critical IT infrastructure?
- throwaway290 2y agoNot even background check but a foreground check would already help. Like literally, who dis? any identity at all? Too often maintainers who have no time just blanket approve PRs and see if stuff breaks.
- arter4 2y agoBut how? Let's say you're one of 10 maintainers of an open source project. A new user wants to contribute. What do you do? Do you ask them to send you some form of ID? Assuming this is legal and assuming you could ensure the new user is the actual owner of an actual, non counterfeit ID, what do you do? Do you vet people based on their nationality? If so, what nationality should be blackballed? Maybe 3 maintainers are American, 5 are European and 2 are Chinese. Who gets to decide? Or do you decide based on the company they work for? Open source is, by definition, open. The PR/merge request process is generally meant to accept or refuse commits based on the content (which is why you have a diff), not on the owner. Building consensus on which commits are actually valid, even in the face of malicious actors, is a notoriously difficult problem. Byzantine fault tolerance can be achieved with a 2/3 + 1 majority, but if anyone can create new identities and have them join the system (Sybil attack) you're going to have to do things differently.
- pdimitar 2y agoThis was only a matter of time. Open source projects are under-staffed, maintainers are overworked and burned out, and everyone relies on the goodwill of all actors. Obviously a bad actor will make use of these conditions and the assumption of good will. We need automated tooling to vet for stuff like this. And maybe migrate away from C/C++ while we are at it because they don't make such scanning easy at all.
- Epa095 2y agoI hope Lasse Collin is doing OK! Here is a older message from him [1] "I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see. It's also good to keep in mind that this is an unpaid hobby project. " Github (Microsoft) are in a unique position to figure out if his account is hacked or not, and find a way to reach him. I hope they reach out and offer him some proper support! Economic support (if that's needed), or just help clearing his name. This is another tale of how we are building multi trillion dollar industries on the back of unpaid volunteers. It's not github 'job', and many other organisations have benefited even more from Lasses work, but they are in a unique position, and would be literally pocket change for them. 1:https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...
- zh3 2y agoComment from Andres Freund on how and why he found it [0] and more information on the LWN story about the backdoor. Recommend people read this to see how close we came (and think about what this is going to mean for the future). [0] https://lwn.net/Articles/967194/ https://lwn.net/Articles/967194/
- eBombzor 2y agoThat man deserves a Nobel Prize
- itsTyrion 2y agothat's... creative. and patient. 11/10 concerning - now I'm wondering how many other projects could have shit like this in them or added right as I'm writing this shudder
- kzrdude 2y agoJia Tan "cleaned up" in all their ZSTD branches some hours ago, probably hiding something https://github.com/JiaT75/zstd/branches/all https://github.com/JiaT75/zstd/branches/all
- zamalek 2y agoBad move. Destroying evidence is a felony.
- cypress66 2y agoIf you are this deep into it, it doesn't matter.
- delfinom 2y agoIf only you could prosecute people in adversarial countries for a felony, lol.
- maerF0x0 2y agoYou can if you can get them extradited (from any country, not just their home country).
- JackSlateur 2y agoNot everywhere, and only if you can prove that were evidences :)
- sroussey 2y agoGitHub/Microsoft likely has a backup. I’d be getting those out about now.
- imanhodjaev 2y agoI wonder which browsers link liblzma and can this lead to https eavesdropping?
- dfgdfg34545456 2y agochmod u+x running detect_sh script just runs with no output on my arch linux box? https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4
- Hackbraten 2y agoYes, Arch Linux’s OpenSSH binary doesn’t even link to liblzma, which means your installation is not affected by this particular backdoor. The authors of the `detect_sh` script didn’t have that scenario in mind, so the `ldd` invocation never finds a link and the script bails early without a message.
- dfgdfg34545456 2y agoThanks!
- 77pt77 2y agoremove the -e option on the script and run it. Anyway, arch is not affected because they don't modify openssh to link against any of this nonesense.
- imanhodjaev 2y agonow I wonder which browsers link liblzma?
- hcks 2y agoIt was caught out of luck due to performance degradation. So nobody reads the code - not even once- prior to merging into upstream supply chain?
- hcks 2y agohttps://x.com/bl4sty/status/1773780531143925959?s=20 https://x.com/bl4sty/status/1773780531143925959?s=20 So nobody reads releases notes either. But I’m sure this was a one off and were safe now
- dhx 2y agoA mirror of the offending repository created by someone else is available at [1]. GitHub should be keeping the evidence in the open (even if just renamed or archived in a safer format) instead of deleting it/hiding it away. The offending tarball for v5.6.1 is easier to find, an example being.[2] m4/.gitignore was updated 2 weeks ago to hide build-to-host.m4 that is only present in the release tarball and is used to inject the backdoor at build time.[3] [1] https://git.phial.org/d6/xz-analysis-mirror https://git.phial.org/d6/xz-analysis-mirror [2] https://mirrors.xtom.ee/gentoo/distfiles/9f/xz-5.6.1.tar.gz https://mirrors.xtom.ee/gentoo/distfiles/9f/xz-5.6.1.tar.gz [3] https://git.phial.org/d6/xz-analysis-mirror/commit/4323bc3e0c1e1d2037d5e670a3bf6633e8a3031e https://git.phial.org/d6/xz-analysis-mirror/commit/4323bc3e0...
- 8organicbits 2y agoThere's good discussion of the timeline here: https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo...
- user20180120 2y agoWhy is the Long Range Zip lrzip compression format not used? It gives better compression than xz when using the correct switches.
- squarefoot 2y agoState actor or not, let's not ignore that the backdoor has been discovered thanks to the open nature of the projects involved that allowed digging into the code. Just another example like the infamous Borland InterBase backdoor in the early 2K that remained dormant for years and was discovered months after the source code has been released. If the xz malware authors worked for any corp that produced closed source drivers or blobs that can't be properly audited, we would be fucked; I just hope this is not already happening, because the attack surface in all those devices and appliances out there running closed code is huge.
- Roark66 2y agoSadly this is exactly one of the cases where open source is much more vulnerable to a state actor sponsored attack than proprietary software. (it is also easier to find such backdoors in OS software but that's BTW) Why? Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he. Also they have to be hiring in the right team etc. Your operative has to be in a different country, needs a CV that checks out, passports/ids are checked etc. But to contribute to an OS project? You just need an email address. Your operative sends good contributions until they build trust, then they start introducing backdoors in the part of the code "no one, but them understands". The cost of such attack is a lot lower for a state actor so we have to assume every single OS project that has a potential to get back doored had many attempts of doing so. (proprietary software too, but as mentioned, this is much more expensive) So what is the solution? IDK, but enforcing certain "understandability" requirements can be a part of it.
- alufers 2y agoIs that true? Large companies producing software usually have bespoke infra, which barely anyone monitors. See: the Solarwinds hack. Similarly to the xz compromise they added the a Trojan to the binary artifacts by hijacking the build infrastructure. According to Wikipedia "around 18,000 government and private users downloaded compromised versions", it took almost a year for somebody to detect the trojan. Thanks to the tiered updates of Linux distros, the backdoor was caught in testing releases, and not in stable versions. So only a very low percentage of people were impacted. Also the whole situation happened because distros used the tarball with a "closed source" generated script, instead of generating it themselves from the git repo. Again proving that it's easier to hide stuff in closed source software that nobody inspects. Same with getting hired. Don't companies hire cheap contractors from Asia? There it would be easy to sneak in some crooked or even fake person to do some dirty work. Personally I was even emailed by a guy from China who asked me if I was willing to "borrow" him my identity so he could work in western companies, and he would share the money with me. Of course I didn't agree, but I'm not sure if everybody whose email he found on Github did. https://en.wikipedia.org/wiki/2020_United_States_federal_government_data_breach https://en.wikipedia.org/wiki/2020_United_States_federal_gov...
- 2y ago
- sirsinsalot 2y agoI think we have to assume that all community software is a target. The payoff for bad actors is too great. For every one of these we spot, assume there are two we have not.
- Dribble4633 2y agoHello, Github just disabled the repo : https://github.com/tukaani-project/xz https://github.com/tukaani-project/xz Do someone have an up to date fork to see the project history ?
- _zephyrus_ 2y agoIs there any news concerning the payload analysis? Just curious to see if it can be correlated with something I have in my sshd logs (e.g. login attempt with specific RSA keys).
- Randalthorro 2y agoSince GitHub disabled the repos.. I uploaded all GitHub Events from the two suspected users and from their shared project repo as easy to consume CSV files: https://github.com/emirkmo/xz-backdoor-github https://github.com/emirkmo/xz-backdoor-github For those who want to see the GitHub events (commits, comments, pull_requets, diffs, etc.)
- lostmsu 2y agoBetter make a torrent out of them.
- KOLANICH 2y agoPlease note: the changes have been made after GitHub has enforced 2FA (certainly not for "better security", but for promotion of FIDO2 and Windows Hello biometric impl of FIDO2, see https://codeberg.org/KOLANICH/Fuck-GuanTEEnomo https://codeberg.org/KOLANICH/Fuck-GuanTEEnomo for more info. Until recent times (for now access via git protocol is blocked for my acc, I guess based on lack of 2FA set up) it was even possible to push into all repos one has access by just using single-factor SSH key even without enabling 2FA in the account). As I have warned, nothing will protect when a backdoor is introduced by a malicious maintainer, or a "smart entrepreneur" who sold his project to a ad-company, or a loyal "patriot" living and earning money within reach of some state, or just a powerless man who got an offer he can't refuse. In general supply chain attacks by "legitimate" maintainers cannot be prevented. "Jia Tan" is just a sockpuppet to mitigate consequences to maintainers to make it look like they are not involved into it. They surely are. At least according to the current info it were they who have given the malicious account the permission to publish releases on behalf of the project and access to the repo. IMHO all maintainers of the backdooored projects anyhow related to accepting the malicious changes should be considered as accomplices and boycotted. We don't need evidence of their liability, it is they who need to maintain their reputation. We are just free to take our decisions based on their reputation. Even if they were hacked themselves, it is not our problem, it is their problem. Our problem is to keep ourselves safe. It may feel "unjust" to ruin reputation of a person based on the fact he may be cheated or hacked… But if a person can be cheated or hacked, why should he/she have such a good reputation as everyone else?! So, it makes a lot of sense to just exclude and replace everyone, for whome there exists evidence of comprometation, no matter due to unconcern or malice. But FOSS is a doocracy serving products at dumpling prices ($0, free of charge), and for majority backdoored software is completely acceptable given that they get them free of charge. And powerful actors who can afford to pay for software will just hire devs to develop their private versions, while allowing the public to pay $0 for their free versions and use the backdoors placed into them themselves. In other words a complete market failure. I think that 1. xz project must be shut down completely. I mean projects should stop using it as a dependency, exclude from distros, boycott it. LZMA algo was developed by Igor Pavlov in 7z project, but somehow it has happenned that liblzma was developed and maintained by unrelated folks. liblzma should be developed as a part of 7z project taking no code other than the trivial one for API compatibility adapter from xz. 2. Projects created by compromised authkrs should be boycotted. 3. Other projects touched by the compromised devs/maintainers should be audited. 4. All the projects using autotools should be audited and must replace autotools with cmake/meson. Autotools is a piece of shit, completely uncomprehensible. There is no surprise it was used to hude a backdoor - according to my experience in FOSS noone likes to touch its scripts anyhow. 5. No project should be built from releases. Project should be built from git directly. Implementing full support of SHA256 in git and git forges (GitHub, GitLab, Codeberg, sr.ht) should be accelerated to mitigate attacks using collisions to replace approved commits (I guess the randomness can be concealed from reviewer's eye in binary resource files, like pictures).
- throwaway67743 2y agoIt's always Debian, like last time when they removed RNG randomness from ssh because of a warning.
- zingelshuher 2y agoWhy isn't he identified personally? Very likely he is 'contributing' to other projects under different accounts.
- Decabytes 2y agoSo when are we going to stop pretending that OSS maintainers/projects are reaping what they sow when they "work for free" and give away their source code away using OSS licensed software, while large companies profit off of them? If they were paid more (or in some cases even actually paid), then they could afford to quit their day jobs, reducing burn out, they could actually hire a team of trusted vetted devs instead of relying on the goodwill of strangers who step up "just to help them out" and they could pay security researchers to vet their code. Turns out burned out maintainers are a great attack vector and if you are willing to play the long game you can ingratiate yourself with the community with your seemingly innocuous contributions.
- kortilla 2y agoPaid people get burnt out as well and they are just as likely to accept free help as an unpaid person.
- Decabytes 2y agoThat's true, but many of these maintainers work a day job on top of doing the open source work precisely because the open source work doesn't pay the bills. If they could get back 40 hours of their time I think many would appreciate it
- qwery 2y ago> So when are we going to stop pretending ... I'm not sure that we are. Doesn't everybody know that developing/maintaining free software is largely thankless work, with little to no direct recompense? I don't think moving towards unfree software is a good way to make free software more secure. It shouldn't be a surprise that proprietary software is less likely to be exploited in this way simply because they don't accept any patches from outside of the team. What you want is more people that understand and care about free software and low barriers to getting involved.
- Decabytes 2y ago> Doesn't everybody know that developing/maintaining free software is largely thankless work, with little to no direct recompense? No I don't think that is a universally acknowledged feeling. Numerous maintainers have detailed recieving entitled demands from users, as if they were paying customers to the open source software projects. Georges Stavracas' interview on the Tech over Tea podcast^1 describes many such experiences. Similarly, when Aseprite transitioned its license^2 to secure financial stability, it faced backlash from users accusing the developer of betraying and oppressing the community. On the flipside, if everyone truly does know this is the case, then it's a shame that so many people know, and yet are unwilling to financially support developers to change that. See all of the developers for large open source projects who have day jobs, or take huge pay cuts to work on open source projects. I get that not everyone can support a project financially, but I've personally tried to break that habit of expecting everything I use to be free, and go out of my way to look for donation buttons for project maintainers, and raise awareness during fundraisers. Now if only I could donate directly to Emacs development... I'd encourage other people to do the same. > What you want is more people that understand and care about free software and low barriers to getting involved. This is tough. For example, the intention behind initiatives like DigitalOcean's Hacktoberfest, are designed to do just this. It is a good idea in theory, submit 4 pull requests and win a tshirt, but not in practice. The event has been criticized for inadvertently encouraging superficial contributions, such as minor text edits or trivial commits, which burden maintainers^3, causing many maintainers to just archive their repos for the month of October. So, while there's a recognition of the need for more people who understand and value free software, along with lower barriers to entry, the current state of affairs often falls short. The path forward should involve not just increasing awareness and participation but also providing meaningful support and compensation to maintainers. By doing so, we can foster a more sustainable, secure, and vibrant open source community. Or at least that is how I feel... 1. https://www.youtube.com/watch?v=kO0V7BE1bEo https://www.youtube.com/watch?v=kO0V7BE1bEo 2. https://github.com/aseprite/aseprite/issues/1242 https://github.com/aseprite/aseprite/issues/1242 3. https://twitter.com/shitoberfest?lang=en https://twitter.com/shitoberfest?lang=en
- pinley 2y agohttps://imgur.com/WGaK3Tn https://imgur.com/WGaK3Tn
- 7ero 2y agois this sev0?
- 7ero 2y agoIs this sev0?
- jaromilrojo 2y agoThis is another proof that systemd is an anti-pattern for security: with its crawling and ever growing web of dependencies, it extends the surface of vulnerability to orders of magnitude, and once embraced not even large distro communities can defend you from that. A malware code injection in upstream xz-tools is a vector for remote exploitation of the ssh daemon due to a dependency on systemd for notifications and due to systemd's call to dlopen() liblzma library (CVE-2024-3094). The resulting build interferes with authentication in sshd via systemd.
- saagarjha 2y agoThis isn't Twitter you don't have to use hashtags
- heptazoid 2y agoThis isn't Xitter, you don't have to tell people how to write.
- acdha 2y agoPlease take the systemd trolling to Reddit. They likely targeted xz specifically because it’s so widely used but there are dozens of other libraries which are potential candidates for an attack on sshd, much less everything else which has a direct dependency unrelated to systemd (e.g. dpkg). Rather than distracting, think about how the open source projects you use would handle an attack like this where someone volunteers to help a beleaguered maintainer and spends time helpfully taking on more responsibilities before trying to weaken something.
- jaromilrojo 2y agoYou are distracting from facts with speculations and trolling FUD. I refer to what is known and has happened, you are speculating on what is not known.
- acdha 2y agoYour claim is an appeal to emotion trying to build support for a position the Linux community has largely rejected. Starting with the goal rather than looking unemotionally at the facts means that you’re confusing your goal with the attackers’ – they don’t care about a quixotic attempt to remove systemd, they care about compromising systems. Given control of a package which is on most Linux systems and a direct dependency of many things which are not systemd - run apt-cache rdepends liblzma5! – they can choose whatever they want to accomplish that goal. That could be things like a malformed archive which many things directly open or using something similar to this same hooking strategy to compromise a different system component. For example, that includes things like kmod and dpkg so they could target sshd through either of those or, if their attack vector wasn’t critically dependent on SSH, any other process running on the target. Attacking systemd for this is like saying Toyotas get stolen a lot without recognizing that you’re just describing a popularity contest.
- the_errorist 2y agoLooks like Lasse Collin has commented on LKML: https://lkml.org/lkml/2024/3/30/188 https://lkml.org/lkml/2024/3/30/188 Also, some info here: https://tukaani.org/xz-backdoor/ https://tukaani.org/xz-backdoor/
- jwilk 2y agoOr if you can't stand the lkml.org UI: https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/ https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/
- hypnagogic 2y agoIn the future: automated `diff` or any other A/B check to see whether or not the tarball matches the source repo (if not, auto-flag with a mismatch warning attribute), is that feasible to implement?
- qxfys 2y agoSo, it's been almost 24 hours since I read this yesterday. Is it confirmed that Jia Tan is the perpetrator? do we know who he/she really is? Or are we going to live for the rest of our lives only knowing the pseudo name? just like Satoshi Nakamoto did to us. ;)
- mise_en_place 2y agoThis is why we never upgrade software versions. I’ve been asked by our customers why we use such an old AMI version. This is why.
- gkoberger 2y agoThis feels like the exact opposite of the takeaway you should have. Old software isn't inherently more secure; you're missing thousands of security and bug fixes. Yes, this was bad, but look how quickly the community came together to catch it and fix it. It only took 6 days for it to be found and fixed.
- hypnagogic 2y ago- * _ring ring_ * - "Hello?" - "It's Lasse Collin." - "Why are you collin me? Why not just use the backdoor?"
- evilmonkey19 2y agoWhich OS are affected by this compromise?? Is Ubuntu affected?
- costco 2y agoAnyone have any idea what the code in the malicious liblzma_la-crc64-fast.o is actually doing? It's difficult to follow statically.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- snickerer 2y agoWhen I search for "digital masquerade" on Google, the first result is a book with this title from the author Jia Tan. I assume that is how the attackers got their fake name. Or they think using this author's name is a joke.
- wowserszzzzz 2y agoWow https://ubuntu.com/security/notices/USN-5378-2 https://ubuntu.com/security/notices/USN-5378-2
- wowserszzzzzz 2y ago[flagged]
- ptx 2y agoPython for Windows bundles liblzma from this project, but it appears to be version 5.2.5 [0] vendored into the Python project's repo on 2022-04-18 [1], so that should be fine, right? [0] https://github.com/python/cpython/blob/main/PCbuild/get_externals.bat https://github.com/python/cpython/blob/main/PCbuild/get_exte... [1] https://github.com/python/cpython-source-deps/tree/xz https://github.com/python/cpython-source-deps/tree/xz
- croemer 2y agoWhich nation state (if any) is most likely behind this? China based on name, or is this a red herring? The perpetrator did most GitHub actions between 10 and 18 UTC, which sort of rules out US based, unless the messages were scheduled. Consistent with Europe to Asia. See clickhouse for data: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGFjdG9yX2xvZ2luPSdKaWFUNzUnIE9SREVSIEJZIGZpbGVfdGltZSBERVND https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...
- rieter 2y agoCould easily be someone in the EST time zone. There aren't that many events that would be earlier than 8am EST.
- perryizgr8 2y agoWhy are projects like xz and sshd still active? Just freeze it, it works fine. Only changes should be fixes for vulnerabilities. None of this complicated new functionality. If you want something like that make a new project. If it is truly better people will use it.
- joshhansen 2y agoIs it time to deprecate the ability for code to implement linker symbols in other libraries? Shouldn't there be a strict namespace separation between binaries/libraries? liblzma being to implement openssh symbols seems like a symptom of a much larger problem.
- krascovict 2y agoHello everybody. I am taking the initiative to gather more information regarding the possible precursors and perpetrators of the backdoor. The purpose of this commentary is focused on open source information (OSINT). I am not a judge of anyone or any action that may occur, the objective of this comment is to help through accurate and quick information to help the core developers of the affected packages and consequently the Linux kernel (which may have been indirectly or directly affected) take action necessary in relation to the fact that occurred. NOTE: This comment will always have "edit" so always review it for information. Information I have so far. Summary: 1. GitHub Account Suspension: - The accounts of @JiaT75 and @Larhzu were suspended by GitHub. - All Tukaani repositories, including downloads, were disabled. - Investigate the cause of the account suspensions and whether there is any correlation with suspicious activities. 2. Possible Backdoor in xz/liblzma: - There are concerns about the presence of a backdoor in xz/liblzma. - Investigate whether there is evidence of compromise in the source code and recent updates. - Examine potential impacts, especially if the software is used in critical systems. 3. Updates and Patches in Packages: - Note recent updates in packages such as MinGW w64, pacman-static, Alpine, and OpenSUSE. - Review changelogs to understand if these updates are related to security fixes. 4. Jia's Activities on Platforms and Projects: - Investigate Jia's contributions to different projects and platforms, such as Arch Linux, Alpine Linux, and OpenSUSE. - Check for correlations between Jia's activities and reported security issues. 5. Libera Registration Information: - Analyze Jia's registration details on Libera to determine the timeline of their online activities. - Consider correlating this information with other online activities of Jia. 6. VPN Usage: - Confirm Jia's use of VPN and assess its impact on security investigations. - Explore possible reasons for using a VPN and how it may affect the identification and tracking of online activities. Links related to user JiaT75 [xz] Remove JiaT75 as a contact, determine correct contacts #11760 - Google/oss-fuzz https://github.com/google/oss-fuzz/issues/11760 https://github.com/google/oss-fuzz/issues/11760 Tuktest index hash #7 - tukaani-project/xz/pull/7 https://web.archive.org/web/20240329230522/https://github.com/tukaani-project/xz/pull/7 https://web.archive.org/web/20240329230522/https://github.co...
- bheadmaster 2y agoThis is exactly why I fight the windmills so hard when it comes automatic updates in Linux software. So much damage is caused just by adding a single maintainer to a project - imagine how much power you would have to wield the remote execution systems put in place by naive developers for "automatic updates". All it takes is a single malicious maintainer given access to the new version update of some popular user software, and they have a new botnet of thousands of devices at their disposal. Better yet, after the backdoor installation, they can just release the real update and cover their tracks forever. Automatic updates are like running web applications, but without any sandboxing or protection usually implemented by the browser.
- bicepjai 2y agoFor someone who does not understand the packages used, could you please summarize in layman non technical terms. Thanks I did read the main post.
- bicepjai 2y agoThis link helped a little. https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27 https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...
- ui2RjUen875bfFA 2y agoi added there something about a possible planned kernel attack which not mentioned so much yet. https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27?permalink_comment_id=5008999#gistcomment-5008999 https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...
- bicepjai 2y agoThis is perfect summary https://research.swtch.com/xz-timeline https://research.swtch.com/xz-timeline
- simplylashadow 2y ago[dead]
- 3v1n0 2y agoAlso the attacker included in the 5.6.0 release the support for the long-awaited multi-threading decompression (and - broken - sandbox) making it very attractive to upgrade to... It was probably a tactic to give a reason to upgrade. It's not always a fault for those who did or tried to do.