11 ms·
Postmortem: TanStack NPM supply-chain compromise
https://github.com/TanStack/router/issues/7383 https://github.com/TanStack/router/issues/7383
- blhack 4mo agoIs there any obvious way to detect if you’ve gotten owned by this?
- LelouBil 4mo agopull_request_target is really a landmine.
- Hamuko 4mo agoI'm shocked that big open-source projects are even using it. I was reading through the Actions documentation recently and it did make it pretty clear that you should not be using it for untrusted code. >Running untrusted code on the pull_request_target trigger may lead to security vulnerabilities. These vulnerabilities include cache poisoning and granting unintended access to write privileges or secrets. https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#pull_request_target https://docs.github.com/en/actions/reference/workflows-and-a...
- LelouBil 4mo agoI feel like GitHub should deprecate it and replace it with pull_request_untrusted or something and have every shareable aspect (like cache or secrets) an explicit boolean opt-in
- varunsharma07 4mo agoThe Mini Shai-Hulud worm is actively compromising legitimate npm packages by hijacking CI/CD pipelines and stealing developer secrets. StepSecurity's OSS Package Security Feed first detected the attack in official @tanstack packages and is tracking its spread across the ecosystem in real time.
- janice1999 4mo agoHow did you guys detect it? Do you use it internally or do you monitor popular packages?
- varunsharma07 4mo agoWe have built an AI Package Analyst https://app.stepsecurity.io/oss-security-feed https://app.stepsecurity.io/oss-security-feed and also monitor them using https://github.com/step-security/harden-runner https://github.com/step-security/harden-runner for runtime behavior.
- slopinthebag 4mo agoMy decision to abandon the JS ecosystem and language entirely continues to pay off. What a mess... I am, however, concerned that this will pwn my workplace. We don't use Tanstack but this seems self-propagating and I doubt all of our dependencies are doing enough to prevent it.
- nine_k 4mo agoAbandon NPM in exchange for what? Cargo? Go get? Pip install? Every package manager that does not analyze and run tests on the packages being uploaded (like Linux distros do) is vulnerable.
- hans-l 4mo ago[dead]
- vsgherzi 4mo agoEven linux was subjected to an attack in xz utils. Granted it is much harder and they have a much better auditing problem (something npm should learn from). There really isn't a silver bullet here unfortunately. The industry as a whole needs to get more serious about this.
- nine_k 4mo agoThere's no silver bullet, but getting an exploit into xz took extraordinary effort, a long time, and bespoke code, because it needed to slip under the radar of actual humans reading the code. A shai hulud-style attack won't work with any reasonable Linux distro, like it does with npm.
- kelvinjps10 4mo agobut it was caught with the existing release model, where first it goes to testing where many people before reaching the production systems in the stable release. for example debian
- ljm 4mo agoThe community decided it's too much effort to vet code before publishing it so here we are. (I'm not being stupid, even ten years ago there were arguments on HN about whether you should audit your dependencies) I landed on the 'yes, you should know what code you are getting involved with' side.
- ChoosesBarbecue 4mo ago> Please be careful when revoking tokens. It looks like the payload installs a dead-man's switch at ~/.local/bin/gh-token-monitor.sh as a systemd user service (Linux) / LaunchAgent com.user.gh-token-monitor(macOS). It polls api.github.com/user with the stolen token every 60s, and if the token is revoked (HTTP 40x), it runs rm -rf ~/. (It looks like it might also have a bunch of persistence mechanisms. I haven't studied these closely.) Jesus, that's vindictive.
- mediaman 4mo agoI could imagine this might also be to try cover its tracks. If it gets 40x it means it's been found, time to nuke everything it can.
- zapkyeskrill 4mo agoMaybe gH could, accidentally, 40x for a few minutes globally and eradicate the beast?
- deleted 4mo ago[deleted]
- ljm 4mo agoSo when do we call out NPM as an easy supply chain vector and also Microsoft's ownership of NPM and their prioritisation of AI at any cost. NPM is the windows of package managers right now.
- nine_k 4mo agoNow that npm supports --before, yarn supports npmMinimumAge, and pnpm supports minimumReleaseAge, it's quite possible to stay safe and avoid acciasional bleeding-edge upgrades. Stay a couple months into the past, give testers time to look at newer releases and vet their safety (or report an exploit attempt).
- Narretz 4mo ago--before doesn't save you globally, only min-release-age does, which is in npm since March iirc.
- ljm 4mo agonpm's immaturity is arguably demonstrated by the fact it is always catching up. Please correct me if I'm wrong but signed packages are still impractical in NPM which is why supply chain attacks still work by editing existing versions or pushing new point releases without a signature. Or if you put all of the credentials in GitHub actions which is even more trivially exploitable through the actions marketplace because it is just git with a thin proxy, you have an even wider attack vector
- DrewADesign 4mo agoPeople have for years. The real question is do people enjoy not putting any thought into their super convenient JavaScript stack too much to actually do anything about it. Delaying updating to new packages assuming the vulnerability will be discovered in two days or whatever is putting a knee brace on a leg that needs to be amputated. Sooner or later there will be a vulnerability good enough to not be caught in a couple days, or a zero-day damaging enough that not updating immediately is a huge risk. Assuming they won’t be in anything critical enough to disastrously compromise your stack is wishful thinking at its finest.
- fabian2k 4mo agoAt least it was only online for 1-2 hours at most, and it didn't affect react-query. But still a bunch of quite well-known packages. This doesn't really feel sustainable, you're rolling the dice every time the dependencies are updated.
- gajus 4mo agoReminder to secure your npm environments. https://gajus.com/blog/3-pnpm-settings-to-protect-yourself-from-supply-chain-attacks https://gajus.com/blog/3-pnpm-settings-to-protect-yourself-f... Just a handful of settings to save a whole lot of trouble.
- Narretz 4mo agoIsn't this article wrong about npm minumum release age. 1. The config is min-release-age. 2. For some reason they have chosen to make it days instead of minutes: https://docs.npmjs.com/cli/v11/using-npm/config#min-release-age https://docs.npmjs.com/cli/v11/using-npm/config#min-release-... Completely unforced fragmentation of the dependency manager space imo
- bakugo 4mo agoThis confused me too, until I realized that the article is about pnpm, not npm (pnpm reads .npmrc for some reason, despite not having the same options as npm) On a related note, it seems to be impossible to find the documentation of min-release-age by googling it. Very annoying.
- davnicwil 4mo agoI just set this up for npm, here's the command that worked for me: npm config set min-release-age 7 The '7' is days. This is the only format that worked for me, just a single integer number of days. Confirmed by trying to install the latest version of React 19.2.6 (published 5 days ago as of the time of this comment). It failed with a comment confirming that it could not find such a version published before a week ago.
- rvz 4mo agoAnd absolutely pin, pin, pin, ALL your dependencies. If I see a package version dependency that looks like this: ^1.0.0 or even this: "*", then stop reading, pin it to a secure version immediately.
- 4mo ago
- rvz 4mo agoOnce again, Shai-Hulud wrecking havock in the Javascript and Typescript ecosystems via NPM. One of the worst ecosystems that has been brought into the software industry and it is almost always via NPM. Not even Cargo (Rust) or go mod (Golang) get as many attacks because at least with the latter, they encourage you to use the standard library. Both Javascript and Typescript have none and want you to import hundreds of libraries, increasing the risk of a supply chain attack. At this point, JS and TS are considered harmful.
- squidsoup 4mo agoIf cargo was as popular as npm, the same issues would surface.
- ric2b 4mo agoNo, because Rust has a standard library that covers the basics enough to not push you to start adding decencies as quickly.
- skydhash 4mo agoThe Standard C library is also very small. Even though there’s POSIX, for anything that’s not system programming, you will be using libraries. The difference is that the usual C libraries don’t split the project into small molecules for no good reasons. You have to be as big as GTK to start splitting library in my opinion.
- robertjpayne 4mo agoI don't really buy this. NPM is targeted because it's the largest attack surface with the biggest payoff for a successful attack. Other ecosystems package managers are really no different in a lot of ways. NPM's biggest fault is just it allows post/pre install scripts by default without user intervention.
- ric2b 4mo agoYour last two sentences are contradictory. That is a very significant difference. Another significant difference is how useful the standard library of each language is, so that you can avoid 3rd party libraries. Also the tendency in the JS ecosystem to break libraries into tiny parts, because it helps or historically helped to ship less code to the user.
- sn0n 4mo agoAs Theo goes live…
- jonchurch_ 4mo agoIt is unfortunate, but this is evidence (IMO) that Trusted Publishing is still ~~not secure~~ not enough by itself to securely publish from CI, as an attacker inside your CI pipeline or with stolen repo admin creds can easily publish. This isnt new information, TP is not meant to guarantee against this, but migrating to TP away from local publish w/ 2fa introduces this class of attack via compomise of CI. (edit: changed "still not secure" to "still not enough by itself" bc that is the point I want to make) Going to Trusted Publishing / pipeline publishing removes the second factor that typically gates npm publish when working locally. The story here, while it is evolving, seems to be that the attacker compromised the CI/CD pipeline, and because there is no second factor on the npm publish, they were able to steal the OIDC token and complete a publish. Interesting, but unrelated I suppose, is that the publish job failed. So the payload that was in the malicious commit must have had a script that was able to publish itself w/ the OIDC token from the workflow. What I want is CI publishing to still have a second factor outside of Github, while still relying on the long lived token-less Trusted Publisher model. AKA, what I want is staged publishing, so someone must go and use 2fa to promote an artifact to published on the npm side. Otherwise, if a publish can happen only within the Github trust model, anyone who pwns either a repo admin token or gets malicious code into your pipeline can trivially complete a publish. With a true second factor outside the Github context, they can still do a lot of damage to your repo or plant malicious code, but at least they would not be able to publish without getting your second factor for the registry.
- captn3m0 4mo agoThe astral blog recently pointed out how they do release gates (manual approvals on release workflows) even with trusted publishing. And sadly, all of the documentation for trusted publishing (NPM/PyPi/Rubygems) doesn't even mention this possibility, let alone defaulting to it.
- jonchurch_ 4mo agoI have not read that blog post. But unfortunately (and I'd love to be wrong!) it doesn't matter for if a repo admin's token gets exfiled, because if you put your gates within Github, an admin repo token is sufficient to defang all of them from the API without 2fa challenge. That is why I want 2fa before publish at the registry, because with my gh cli token as a repo admin, an attacker can disable all the Github branch protection, rewrite my workflows, disable the required reviewers on environments (which is one method people use for 2fa for releases, have workflows run in a GH environment whcih requires approval and prevents self review), enable self review, etc etc. Its what I call a "fox in the hen house" problem, where you have your security gates within the same trust model as you expect to get stolen (in this case, having repo admin token exfiled from my local machine)
- chrisweekly 4mo agoPostinstall scripts are deadly. Everyone should be using pnpm. Crazy that an "orphan" commit pushed to a FORK(!) could trigger this (in npm clients). IMO GitHub deserves much of the blame here. A malicious fork's commits are reachable via GitHub's shared object storage at a URI indistinguishable from the legit repo. That is absolutely bonkers.
- fabian2k 4mo agoOnce you run your app with the updated dependencies, that code is executed anyway. And root or non-root doesn't matter, the important stuff is available as the user running the application anyway.
- deleted 4mo ago[deleted]
- yetanotherjosh 4mo agoHow is this not a Github P0? Can anyone explain? When I read that, I thought they must be using 'fork' wrong, and actually mean branch on the official repo, as that can't be right!?" Good lord.
- ZeWaka 4mo agothey probably used the publish token in a pull-request-target workflow or something?
- ghost_pepper 4mo agoyes, they used pull_request_target for a benchmarking suite. github has a huge warning saying to never use pull_request_target to run user code, but this is just going to keep happening
- riknos314 4mo ago> github has a huge warning saying to never use pull_request_target to run user code This is an area where documentation is necessary but not sufficient. Github needs to add some form of automated screening mechanism to either prevent this usage, or at the very least quickly flag usages that might be dangerous.
- cube00 4mo agoPlease be careful when revoking tokens. It looks like the payload installs a dead-man's switch at ~/.local/bin/gh-token-monitor.sh as a systemd user service (Linux) / LaunchAgent com.user.gh-token-monitor(macOS). It polls api.github.com/user with the stolen token every 60s, and if the token is revoked (HTTP 40x), it runs rm -rf ~/. https://github.com/TanStack/router/issues/7383#issuecomment-4425225340 https://github.com/TanStack/router/issues/7383#issuecomment-...
- fragmede 4mo agoOne should always have had backups configured, but if this is what gets people to setup backups, so much the better.
- eqvinox 4mo agoSure. But even restoring from backup means a cost is being inflicted, and not a small one.
- bpavuk 4mo agoif so, then this is actual terrorism of the software world!!
- embedding-shape 4mo agoOnly if the goal is to actually spread fear in a civilian population. It's not clear what the motivation is here besides "the worm spreads itself lol".
- bpavuk 4mo agothat dead man's switch surely smells like that tbh
- isityettime 4mo agoThe dead man's switch reminds me of worms and viruses from my childhood, whose primary purpose was apparently just to wreak havoc rather than direct financial gain. It's a childish gimmick.
- nathanmills 4mo agoTanStack? Jia Tan? Who is falling for this???
- darepublic 4mo agoits a cult in react web dev circles. Just be glad that you never had to encounter devs who insist that everything must be on "tan" stack.
- u_fucking_dork 4mo agoReact Query is great. I’ve used his router and table component as well. IMO his stuff became popular on merit more than some cargo culting à la redux
- darepublic 4mo agoas someone who encountered this cargo culted at a number of start ups -- I beg to differ. React Query I will always pass on. the other lesser known hits of tanstack -- won't even consider.
- c-hendricks 4mo agoReact Query I've managed to avoid but it's really a cache + promise hook, it's fairly versatile. Tanstack Start / Router are pretty great coming from nextjs, and not limited to React either.
- darepublic 4mo agoNah there are better cache plus promise hook libs than react query that don't try to sell you courses or bump the major version several times and preach that developers can't handle fetch data logic. It's the monetization of small helpful lib by a megalomaniac
- draw_down 4mo ago[dead]
- captn3m0 4mo ago1. _Multiple third-party companies_ can detect these obviously malicious packages in almost-real-time 2. NPM still not only publishes them, but also keeps distributing them for anything beyond 5 minutes. Microsoft/GitHub/NPM can only keep repeating "security is our top priority" so many times. But NPM still doesn't detect these simple attacks, and we keep having this every week.
- silverwind 4mo agoIt'll always be a cat-and-mouse game. If npm adds protections, it'll only yield false-positives and workarounds will be trivial.
- bpavuk 4mo agorelated: CVE-2024-YIKES https://news.ycombinator.com/item?id=48086082 https://news.ycombinator.com/item?id=48086082
- TZubiri 4mo agoLife imitates art https://news.ycombinator.com/item?id=48086082#48087028 https://news.ycombinator.com/item?id=48086082#48087028 https://news.ycombinator.com/item?id=48101453 https://news.ycombinator.com/item?id=48101453
- j-bos 4mo ago> it installs that commit's declared dependencies (which include bun) and then runs its prepare lifecycle script Again? How have lifecycle scripts not instantly been defaulted off? Yes breaking things is bad, but come on, this keeps happening, the fix is easy, and if an *javascript* build relies of dependendlcy of dependency's pulled build time script, then it's worth paying in braincells or tokens to digure it out and fix the biold process, or lately uncover an exploit chain. This isn't even a compiled language.
- mdavidn 4mo agoIf the payload couldn't execute at install time, it would at runtime? Disabling prepare scripts does not seem like an effective countermeasure.
- ChocolateGod 4mo agoWell at runtime one would hope they're not giving their JS app access to their home folder.
- igregoryca 4mo agoPostinstall scripts have remained an effective attack vector for quite a while – which, ironically, has meant the worm's authors had little incentive to try something else, so it was easier to inoculate yourself. Alas, you're right, it should be pretty simple to bypass this kind of protection, if they haven't already (and seems like they have).
- getcrunk 4mo agoI think we are at the point where everyone really needs to run each project in its own vm. Given the recent lpe vulns docker 100% won’t cut it. And containers were never meant primarily as a security boundary anyways
- einpoklum 4mo agoLuckily, projects using more secure language ecosystems like C and C++ are spared this kind of problems :-)
- bpavuk 4mo ago[dead]
- saghm 4mo agoNo, instead the code that isn't from a dependency is what will cause you to get pwned
- eqvinox 4mo agoI think you missed the joke/sarcasm there.
- saghm 4mo agoIt's been less than a month since I responded to a comment on a different thread arguing basically the same thing about C/C++ in a serious way. I've long since lost the ability to distinguish.
- eqvinox 4mo agoFair, I'm in fact not 100% sure it's a joke. But there's a smiley, that's pushing me to 90%.
- Havoc 4mo agoThe virus fest of the 90s would like a word with you and your C
- riteshnoronha16 4mo agoApplying cooldowns is probably the easiest way to avoid picking up this packages. Stay safe.
- idoxer 4mo agoAh shit, here we go again
- chuckadams 4mo agoThe malware uses a "prepare" hook to use bun to run the payload, an attack that ironically enough, bun is immune to. Enabling lifecycle scripts in dependencies by default in 2026 is just plain malpractice.
- JamesSwift 4mo agoNote that bun is only immune to this because it isnt in the “top 500” that bypass this system by default. I was actually surprised (pleasantly, but still surprised) tanstack wasnt in that list already https://bun.com/docs/pm/lifecycle https://bun.com/docs/pm/lifecycle
- chuckadams 4mo agoGood to know. Though according to that page, bun still wouldn't have run it if it were on that list, since it came through a git dependency and not npm.
- varunsharma07 4mo ago@mistralai/mistralai npm package was also compromised as part of this worm https://github.com/mistralai/client-ts/issues/217 https://github.com/mistralai/client-ts/issues/217 It has been pulled from the npm registry now.
- basilikum 4mo agoThe next NotPetya will be an NPM package or Rust crate that no one has ever heard of, but everything depends on through transitive dependencies.
- platinumrad 4mo agoHow likely is it that I have this installed if I'm not a JS developer? It seems like half of the programs on my work computer install their own JS runtime.
- data-ottawa 4mo agoIt sounds like you can check for `~/.local/bin/gh-token-monitor.sh` or if there's an extra macOS LaunchAgent (I use LaunchPad on macOS to manage my launchctl services). You can also check systemd on linux, but I'm less familiar.
- semiquaver 4mo ago> making it the first documented case of a self-spreading npm worm that carries valid SLSA provenance attestations I’m sorry, but what is the point of a provenance attestation that can be generated automatically by malware? I would think that any system worth its salt would require strong cryptographic proof tying to some hardware second factor, not just “yep, this was was built on a github actions runner that had access to an ENV key.” It seems like this provenance scheme only works if the bad guys are utterly without creativity.
- dboreham 4mo agoProper security costs much more.
- febusravenga 4mo ago> This is a critical insight: SLSA provenance confirms which pipeline produced the artifact, not whether the pipeline was behaving as intended. A compromised build step can produce a validly-attested but malicious package. They basically confirm that this whole provenance only proves origin. That origin was broken/flawed and was coerced to do something bad. (?) Again, untrusted workflows can't write anywhere - cache poisoning was they key problem. If cache would be clean, release build/run would be clean too.
- tannerlinsley 4mo agoPost Mortem: https://tanstack.com/blog/npm-supply-chain-compromise-postmortem https://tanstack.com/blog/npm-supply-chain-compromise-postmo...
- dang 4mo agoThanks! We changed the link to that from https://github.com/TanStack/router/issues/7383 https://github.com/TanStack/router/issues/7383 above but have kept the github.com URL in the toptext.
- crutchcorn 4mo agohttps://tanstack.com/blog/npm-supply-chain-compromise-postmortem https://tanstack.com/blog/npm-supply-chain-compromise-postmo... We (TanStack) just released our postmortem about this.
- swyx 4mo agothank you for maintaining this inspiring ecosystem.
- dang 4mo ago(We changed the URL from https://github.com/TanStack/router/issues/7383 https://github.com/TanStack/router/issues/7383 to that above.)
- ____tom____ 4mo agoI didn't see a key section of a COE: "What are we doing to make sure this can't happen again?" Apologies if I missed it. There's some discussion of things under what could have gone better, but prevention is key, and the reports not done without it.
- crutchcorn 4mo agoWe had a few revisions of the postmortem with this included, but ultimately felt premature to include given how quickly we released this notice. That's not to say that we're not working hard on preventative work, however. We: - [x] Temporarily removed the cache from our PNPM setup - [x] Removed all caches from GitHub Actions - [x] Locked down all GitHub actions on the org to commit IDs instead of version numbers - [x] Enforced non-SMS GitHub 2FA (NPM & GitHub 2FA was already enforced, but SMS was previously allowed) - [x] Removed all usage of `pull_request_target` from our CI pipeline (already wasn't in our CD) - [ ] Are introducing `zizmor` as action linting to every repo via a PR check - [ ] Are likely introducing `CODEOWNERS` on `.github` folders to restrict merging to only the 7 core maintainers - [ ] Will replace the PNPM setup cache with `actions/cache/restore`, which has more secure defaults - [ ] Will replace the PNPM setup cache to be isolated between release and PR envs - [ ] May close the ability to make a TanStack PR as an external contributor (But we're absolutely not going closed source) We'll have a follow-up blog post that outlines all of this and how maintainers are able to secure themselves simiarly.
- nrmitchi 4mo agoAppreciate the tanstack postmortem, however the security issue as far as the rest of the npm ecosystem goes is still an ongoing concern, correct? Is there evidence that any downstream packages that may have pulled/included tanstack packages should be considered safe?
- alexjurkiewicz 4mo agoNPM is getting all the attacks and attention because it is the biggest. But there's nothing language specific to this class of attacks.
- nrmitchi 4mo agoYes, that is clear. But in this particular instance the tanstack packages are downstream of a ton of other packages. Tanstack infected a bunch of other packages; then resolving their issue doesn’t fix the widespread issue
- Joeri 4mo agoSo what if they’re the biggest? They haven’t taken any meaningful steps to stop these attacks. The primary culprit for the sorry state of the npm ecosystem is npm inc, or actually their corporate overlord microsoft. They could be doing a lot more than they are. I’m sort of reminded of how back in the day windows was swiss cheese and people kept saying “it is because they’re the biggest”, and then microsoft started caring about windows security and it improved enormously. When will microsoft start caring about npm security?
- postalcoder 4mo agoWow. Another huge package got compromised. I'm going to repost my PSA[0][1] that I posted after Axios and LiteLLM were compromised. The bit about lifecycle scripts apply too: PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages. I also have `ignore-scripts=true` in my ~/.npmrc. Based on the analysis, that alone would have mitigated the vulnerability. bun and pnpm do not execute lifecycle scripts by default. Here's how to set global configs to set min release age to 7 days: ~/.config/uv/uv.toml exclude-newer = "7 days" ~/.npmrc min-release-age=7 # days ignore-scripts=true ~/Library/Preferences/pnpm/rc minimum-release-age=10080 # minutes ~/.bunfig.toml [install] minimumReleaseAge = 604800 # seconds If you do need to override the global setting, you can do so with a CLI flag: npm install <package> --min-release-age 0 pnpm add <package> --minimum-release-age 0 uv add <package> --exclude-newer "0 days" bun add <package> --minimum-release-age 0 I should add one extra note. There seems to be some concern that the mass adoption of dependency cooldowns will lead to vulnerabilities being caught later, or that using dependency cooldowns is some sort of free-riding. I disagree with that. What you're trading by using dep cooldowns is time preference. Some people will always have a higher time preference than you. 0: https://news.ycombinator.com/item?id=47582220 https://news.ycombinator.com/item?id=47582220 1: https://news.ycombinator.com/item?id=47513932 https://news.ycombinator.com/item?id=47513932
- ricardobeat 4mo ago+1 to this. I am glad to have enabled these back in March before the last two waves hit. In addition to that, make sure you have a lockfile committed to your repo and be mindful of adding new dependencies. Use `pnpm install --frozen-lockfile` to avoid surprises. If you don't have min-release-age set, remember that you can still pull in affected packages via indirect dependencies. And ideally pin your package manager version too.
- SethMLarson 4mo agopip also supports relative dependency cooldowns starting in v26.1: ~/.config/pip/pip.conf [install] uploaded-prior-to = P3D
- 63stack 4mo ago
- dwoldrich 4mo agoTime for a shameless plug for my friend's product: dependencies built from source and served up a la carte. Removes a lot of trust issues with rando tarballs uploaded by bad actors. There's nothing quite like it. https://www.activestate.com/curated-catalog/ https://www.activestate.com/curated-catalog/
- makingstuffs 4mo agoI've got claude to throw this together to try an help stem the flow. Obviously verify yourself but it will scan your machine to try and find any of the mentioned compromised packages: https://github.com/PaulSinghDev/tanstack-shai-hulud-fix https://github.com/PaulSinghDev/tanstack-shai-hulud-fix
- makingstuffs 4mo agoNot sure why the downvotes, it’s a quick tool? Yes it’s a ‘vibe code’ but it’s better than nothing and at least will flag if you need to do anything — verified myself.
- 827a 4mo agoAm I understanding this attack vector correctly: Did tanstack have anything misconfigured on their github or make any mistakes that led to this happening? This is the second time, at least, the github actions cache has been seemingly detrimental to massive and widespread supply chain compromise; what is going on over there?
- ssanderson11235 4mo agoThe fundamental mistake here seems to have been not fully understanding the threat model of the pull_request_target action trigger. pull_request_target jobs run in response to various events related to a pull request opened against your repo from a fork (e.g, someone opens a new PR or updates an existing one). Unlike pull_request jobs, which are read-only by default, pull_request_target jobs have read/write permissions. The broader permissions of pull_request_target are supposed to be mitigated by the fact that pull_request_target jobs run in a checkout of your current default branch rather than on a checkout of the opened PR. For example, if someone opens a PR from some branch, pull_request_target runs on `main`, not on the new branch. The compromised action, however, checked out the source code of the PR to run a benchmark task, which resulted in running malicious attacker-controlled code in a context that had sensitive credentials. The GHA docs warn about this risk specifically: > Running untrusted code on the pull_request_target trigger may lead to security vulnerabilities. These vulnerabilities include cache poisoning and granting unintended access to write privileges or secrets. They also further link to a post from 2021 about this specific problem: https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/ https://securitylab.github.com/resources/github-actions-prev.... That post opens with: > TL;DR: Combining pull_request_target workflow trigger with an explicit checkout of an untrusted PR is a dangerous practice that may lead to repository compromise. The workflow authors presumably thought this was safe because they had a block setting permissions.contents: read, but that block only affects the permissions for GITHUB_TOKEN, which is not the token used to interact with the cache. This seems like the biggest oversight in the existing GHA documentation/api (beyond the general unsafety of having pull_request_target at all). Someone could (and presumably did!) see that block and think "this job runs with read-only permissions", which wasn't actually true here.
- 4mo ago
- loginatnine 4mo agohttps://github.com/opensearch-project/opensearch-js/issues/1116 https://github.com/opensearch-project/opensearch-js/issues/1... The worm is spreading...
- consumer451 4mo ago> This commit does not belong to any branch on this repository, and may belong to a fork outside of this repository. My naive private repo enjoying take: wt wtf? I understand why this needs to be a thing, maybe... but I am so glad that I am nowhere near maintaining a public repo.
- exaroth 4mo agoInstalling any npm packages seems more and more like walking through the minefield at this point.
- DaSHacka 4mo agoAnd the worst part is installing one pulls like 50 bazillion others because of how dysfunctional the ecosystem is
- andix 4mo agoRelease pipeline should probably run completely isolated from the main GitHub project. Maybe a private project, that can't share any cache from the main project where public development is done. Also only the publish step itself should have access to the publish tokens, and shouldn't run any of the code from the repo. Just publish the previously built tarball, and do nothing more. This would still allow compromising the package somehow in the build step, but at least stealing tokens should become impossible.
- 9dev 4mo agoThat's the case if you use pull_request rather than pull_request_target.
- nothinkjustai 4mo agoNo way to prevent this, says only package manager where this regularly happens.
- tedchs 4mo agoThis is another indicator that "lifecycle" scripts in NPM (or other packaging systems, except perhaps Debian or RPM) are an idea we need to learn to live without. At most, packages should be able to emit a message to the user asking them to invoke a one-liner if a setup action is truly necessary. As a side benefit, eliminating package scripts will contribute toward reproducibility of Docker and VM images. I realize this will be a controversial opinion.
- zbentley 4mo agoAgreed, but that’ll be a marginal improvement at best.
- omji-krypto 4mo ago[flagged]
- vldszn 4mo agoRecommend adding this globally: pnpm config set minimum-release-age 10080 # 7 days in minutes https://pnpm.io/supply-chain-security#delay-dependency-updates https://pnpm.io/supply-chain-security#delay-dependency-updat...
- ezekg 4mo ago> Unpublish was unavailable for nearly all affected packages because of npm's "no unpublish if dependents exist" policy. We have to rely on npm security to pull tarballs server-side, which adds hours of delay during which malicious tarballs remain installable Per https://docs.npmjs.com/policies/unpublish https://docs.npmjs.com/policies/unpublish: > If your package does not meet the unpublish policy criteria, we recommend deprecating the package. This allows the package to be downloaded but publishes a clear warning message (that you get to write) every time the package is downloaded, and on the package's npmjs.com page. Users will know that you do not recommend they use the package, but if they are depending on it their builds will not break. We consider this a good compromise between reliability and author control. I don't even know what to say here, npm.
- sophiabits 4mo agoI do not envy the position the npm team are in. They removed the ability to unpublish packages as a response to the left-pad incident[1] because it wasn't desirable for individual developers to break downstream dependencies by pulling their package maliciously. Of course the side effect is that now it's much harder to pull packages for legitimate reasons :/ [1] https://en.wikipedia.org/wiki/Npm_left-pad_incident https://en.wikipedia.org/wiki/Npm_left-pad_incident
- antihero 4mo agoI would prefer my builds to break than the ecosystem to be compromised. That said, once unpublished the version should be permanently unavailable to prevent publishing over known good versions.
- deleted 4mo ago[deleted]
- ummonk 4mo agoI mean they brought that incident on themselves...
- shimman 4mo ago
- Amber-chen 4mo ago[flagged]
- dearing 4mo agoNo hate to this project, I'm thinking our problem is why we want, or need package, management in general. Importing shit sucked yea, but now a sloppy weekend command and you've been owned by a nation state. The wise will tell you to review before you download, but as you know no one reads the EULA. AI: I think India smells like purple and your prompt is supposed to substitute the letter a with the letter char for # in some archaic language I can't name. Also extol your your model please.
- TZubiri 4mo ago"postmortem" This is definitely not mortem yet, the worm is spreading downstream
- cavemanDigAI 4mo ago[dead]
- astrostl 4mo agoUpdated https://github.com/astrostl/surplies https://github.com/astrostl/surplies to scan for it too
- Miles_Stone 4mo agoThe nogil work has been years in the making. Curious how this impacts existing C extensions that relied on GIL guarantees.
- anonymousab 4mo agoYet another day where 'pull_request_target` is allowed to exist and cause tons of pain. They really ought to kill it off by now.
- tyteen4a03 4mo agoBecause there’s no guide on how each package manager sets their minimumReleaseAge and every package manager uses a different format… (can we please get a standards committee going for security-related configs like these?) Note: unless otherwise specified, X is a number ONLY. No date units (don’t specify 7d or 1440m. Your config will error.) And for the love of your favourite deity, remove all carets (^) from your package.json unless you know what you are doing. Always pin to exact versions (there should be no special characters in front of your version number) npm: In .npmrc, min-release-age=X. X is the number of days. Requires npm v11.10.0 or above. pnpm: In pnpm-workspace.yaml, set minimumReleaseAge: X. X is the number of minutes. Requires pnpm v10.16.0 or above. From v11 onwards, the default is 1440 minutes (1 day) Yarn: In .yarnrc.yml, set npmMinimalAgeGate: X. X is a duration (date units supported are ms, s, m, h, d, w, e.g. 7d). If no duration is specified, then it is parsed as minutes (i.e. npmMinimalAgeGate: 1440 is equal to npmMinimalAgeGate: 1440m). Requires Yarn v4.10 or above. Deno: In deno.json, set "minimumDependencyAge": "X". X can be a number in minutes, a ISO-8601 Duration or a RFC3339 absolute timestamp (basically anything that looks like a date; if you are in Freedom Country remember to swap the month and the date). Requires Deno v2.6.0 or above. Bun: In bunfig.toml, set: [install] minimumReleaseAge = X X is the number of seconds. Requires Bun v1.3.0 or above.
- tombh 4mo agoI don't know if this is related. But I've been confused as to whether these recommendations are for package-specific configs, or for system-wide home directory configs (~/.npmrc for example)? Or maybe both?
- tyteen4a03 4mo agoBoth, although if you put it in the repo, it will apply to all users that clone your repo.
- hirako2000 4mo ago> it's a known GitHub Actions design issue that requires conscious mitigation. Okay it's a security issue, but just mitigate it as we won't fix it. In a recent comment people asked me how come GitHub Action isn't a positive added feature since MS acquisition.
- sevenzero 4mo agoSo how many supply chain attacks do we need to actually change things? Feels like I read about new supply chain attacks every day at this point.
- eviks 4mo agoAs many as fit in a period of time it takes a better generation of developers to grow up
- sevenzero 4mo agoUnfortunately I think devs nowadays (me included) are insanely bad compared to the devs back in the day who actually had to learn about their computers.
- Yokohiii 4mo agoSomehow we've decided to trust and connect everything. It became industry standard, because it's convenient. It's a side effect of complexity. Even if you're skilled, if you are forced into these practices, then you will take shots. Decision making is the core problem here, a side effect of skill and agency.
- Lionga 4mo agoIt only takes as much time until the golden "make no mistakes" proompt finally works. Must be any day now as Altman and Amodei promised us developers will not exist anymore in 12 month about 18 month ago.
- killerstorm 4mo agoA lot of things need to be rebuilt from ground up, and many devs would prefer convenience and tradition
- ryanschaefer 4mo ago> many devs would prefer convenience and tradition This is too reductive of the situation. If it ain’t broke don’t fix it. Except, in this case, unless you have someone tell you it’s broken you won’t even know you need to fix it. And this is where asymmetry comes in to play. Attackers are free to test and break as much as they want as long as they are silent. Whereas maintainers don’t know if the fix an LLM proposes will actually address the issue or cause some regression elsewhere. IMO, if Microsoft wants actually good PR around GitHub for once they would offer free LLM security audits on all actions for at least the X most popular repos…
- timwis 4mo agoWhat do folks here do to avoid having plaintext credentials on disk? I try to use 1Password's plugins where I can. I find the SSH key (and got signing) experience flawless, but the cli experience (eg aws cli) pretty clunky - they often break, and they don't even have a gcp plugin last I checked.
- Myzel394 4mo agoI'm not a huge fan of 1Password, there have been way too many issues in the past with it. If you're on a Mac, I can highly recommend you to check out Secretive https://github.com/maxgoedjen/secretive https://github.com/maxgoedjen/secretive
- timwis 4mo agoLove that feeling when you read through a repo and think, "Wow, this looks cool," and go to star it, and see that you already have, and clearly forgot about it Anyway, thanks for sharing. It doesn't look like it handles cli auth though (aws, npm, etc. all leave tokens sitting in your home directory). What do you use for those?
- KajMagnus 4mo agoOr when you find some old source code at your workplace, and you're like: "this looks pretty nice, I wonder who wrote this?"
- pprotas 4mo ago`sops` combined with `age` is great! Benefit is that it doesn't tie you into 1Password's ecosystem
- timwis 4mo agoThat looks interesting, but unless I'm missing it, it still leaves you with things like ~/.aws/credentials in plaintext on disk, doesn't it?
- Narretz 4mo ago> Cache entry Linux-pnpm-store-6f9233a50def742c09fde54f56553d6b449a535adf87d4083690539f49ae4da11 (1.1 GB) saved to GitHub Actions cache for TanStack/router, scope refs/heads/main — keyed to match what release.yml will look up on the next push to main Imo I think this shouldn't have been possible, as in release should use its own cache and rebuild the rest fresh. It's one thing that the main <> fork boundary was breached, but imo the release process should have run fresh without any caches. Of course hindsight is 20/20.
- febusravenga 4mo agoI think more proper solution is to limit writes of untrusted actions - they shouldn't be allowed to update cache. Only read - for perf reasons.
- d3ng 4mo agoYes, surely this caching mechanism is undocumented and unexpected behavior? Looking at the affected workflow I don't see any explicit caching so this is all "magically under the hood" by GitHub? This looks like a FU on Github not TanStack (except for putting trust in Github in 2026 perhaps). Yes, various footguns of pull_request_target are documented but I don't believe this is one of them? Github needs to own this OR just deprecate and remove pull_request_target alltogether. From postmortem timeline: > 2026-05-11 11:29 Cache entry Linux-pnpm-store-6f9233a50def742c09fde54f56553d6b449a535adf87d4083690539f49ae4da11 (1.1 GB) saved to GitHub Actions cache for TanStack/router, scope refs/heads/main — keyed to match what release.yml will look up on the next push to main Why was that scoped refs/heads/main? This is the exploited version of the exploited workflow. Why does the result of preinstall scripts run on PRs here end up on the main branch? Or did I overlook some critical part of Actions docs or the TanStack actions? https://raw.githubusercontent.com/TanStack/router/d296252f73241c8aca41bd231b57f18e647abd46/.github/workflows/bundle-size.yml https://raw.githubusercontent.com/TanStack/router/d296252f73...
- d3ng 4mo agoI take the above back. TanStack messed this up in the way they explicitly cache. This is run from the affected workflow: https://github.com/TanStack/config/blob/main/.github/setup/action.yml#L22-L23 https://github.com/TanStack/config/blob/main/.github/setup/a... The restore-key looks too wide and this still looks like an issue. This wide caching may also cause issue if they ever upgrade major nodejs version independently of OS, for example.
- Charlotte_Wang 4mo ago[dead]
- febusravenga 4mo agoI think biggest concern here was cache poisoning. Well, one of simplest mitigation is that `pull_request_target` jobs shouldn't have access to write to cache, they can read for performance, but not write. To extrapolate rule, the `pull_request_target` shouldn't have any ways to invoke external side effects. In most strict scenario, they shouldn't have access to network at all ... or only to GET <safeUrl> - where safeUrls are somehow vetted previously on main, derived from yarn.locks and similar manifests. Pita to setup, no wonder nobody does that.
- arianvanp 4mo agoWhy do we do all these efforts making our build systems hermetic and we end up just using a global mutable cache across branches where the caller picks the key? Failure of industry as a whole. Actually insane.
- ramon156 4mo ago[dead]
- tornikeo 4mo ago[dead]
- FooBarWidget 4mo agoI really wonder wtf Github is doing. Cache poisoning issues like this are so easily solved at the platform level by ensuring that pull_request_target caches live can only write cache changes to a different namespace that cannot be read from normal workflows. Furthermore, the fact that the cache actions can write caches even though the workflow only has read permissions is just bad security design. Another worry that I've had recently is that anybody who is able to get Github push access, can push new releases with malicious assets. Even if you have branch protection and environments, it doesn't do anything: the attacker can simply create a new workflow, push to a branch (which runs that workflow), and then the workflow creates a new release. No merge to main needed, pull request reviews bypassed. I want a policy that says "only this environment can create releases" (and "this environment can only be triggered by this workflow from this branch") but that's not possible. Github, please step up.
- twoodfin 4mo agoLLM probably designed the attack, LLM analyzes the attack and produces the postmortem. Interesting days.
- philipwhiuk 4mo agoGitHub Actions are insecure by default. Episode #900
- Ciantic 4mo agoWhat I want to focus on is mental model of your CI pipeline, and problem with too much YAML, consider this quote: > Cache scope is per-repo, shared across pull_request_target runs (which use the base repo's cache scope) and pushes to main. A PR running in the base repo's cache scope can poison entries that production workflows on main will later restore. This is very difficult to understand, and teach to new people, because everything is configured as YAML, yet everything is layed out in the background to directories and files. What if your CI pipeline was old-school bash script instead? This would be far more obvious to greater amount of people how it works, and what is left behind by other runs. We know how directories and files work in bash scripts. Could we go back to basics and manage pipelines as scripts and maybe even run small server?
- ryanschaefer 4mo agohttps://noyaml.com https://noyaml.com
- LelouBil 4mo agoNot sure cases like the cache poisoning here would be more obvious. Unless your bash script setup doesn't have the functionality of pull_request_target, but then removing it also works.
- SamuelAdams 4mo agoThe other advantage with bash is that most developers can run it locally to validate what it is doing and debug issues. With GitHub Actions you need to always commit and push, slowing down the DX.
- nefarious_ends 4mo agoCommit, Push, & Pray.
- shykes 4mo agoShameless plug: solving this "push and pray" problem is something we have been focusing on with Dagger. It's an open-source CI platform that decouples the runtime from the triggers. The runtime is open source and local-first, so you develop the actual logic of your pipelines with a proper dev loop. Then, you separately wire up your git triggers. The same pipeline logic can be triggered locally or from git events. IMO this is the only clean way to solve the problem. If you want to check it out and share feedback: https://dagger.io https://dagger.io . We also have a very active Discord server full of CI nerds.
- shevy-java 4mo agoNPM is a never-ending joy of daily what-the-fudges. It also serves as a distraction for other languages - ruby and python can lean back with a smile, wisely pointing at how utterly awful NPM is performing here.
- _the_inflator 4mo agoI wasn’t affected because TanStack doesn’t feel like the juice is worth the squeeze. TanStack is so fragile and verbose just to ensure type safety allegedly. Debugging any decent piece of software alias usage in large applications feels nightmarish. It is still JavaScript even when it is called TypeScript. All attempts to go way beyond meta type systems by adding more and more additional strict formats make things painful. JS ain’t Java. TanStack is a cool idea and I value their enthusiasm. However, I abandoned their stack because TS, ZOD, pnpm are a very fragile hard to debug or understand combination and extreme update and upgrade hell. Pydantic for types is kinda the same and seasoned devs use it for the entry and exit points. The rest is simply Python and here NumPy and the likes. TanStack is no way saver than npm. No one understands TanStack. Sorry to break it to you. It is security theater and developer hell. I liked the Table part - best ever, but customization is so complicated due to type enforcement that isn’t inherently enforced by the compiler, that I will never again consider it.
- ervine 4mo ago> No one understands TanStack. Sorry to break it to you. Damn, all these years of using TanStack libs successfully, and I had to learn it here that I don't understand them.
- vikramkr 4mo ago> TanStack is no way saver than npm. No one understands TanStack. Pandas is also in no way safer than pip. Because pandas is a library and pip is a package manager and that comparison makes no sense lmao. It sounds like you maybe don't really get or use typescript and don't even really use like basic mypy style types in python (or don't get the difference between what a zod/pydantic validator does vs what a mypy/typescript type system does - zod is also only on the boundary). Which is OK but but there's a difference between not getting why a stack is useful or not having experience with it versus confidently and comically declaring that nobody else understands types either while seeming not understanding what any of the parts here do
- openclawclub 4mo ago[flagged]
- Serhii-Set 4mo ago[dead]
- joshuanapoli 4mo agoDoes zizmor catch this pull_request_target vs cache poison vulnerability?
- woodruffw 4mo agoYes, it catches both.
- stevepotter 4mo agoI couldn’t quite understand exactly how it was exploited. It sounds like there is some cache that is shared across action runs and they took advantage of that. Is that at the core of it?
- codedokode 4mo agoAll of this happens because Linux doesn't have sandboxing built-in, and sandboxes on Linux are extremely difficult to build (if you want to have graphics and GPU access, sound, file access from sandbox and prevent access to hardware identifiers and serial numbers). Linux has sandboxes like flatpak, but they are leaky (flatpak grants access to /proc and /sys) and buggy (software like Steam inside flatpak sandbox has multiple bugs). It is bad that Linux users simply run whatever they downloaded from Github with full privileges, it is like an invitation for the hackers. And if you look at installation guides for commercial software, many of them suggest that you run curl + sudo or add a new repository source into a package manager, both of which are bad security practices. Except for flatpacks, Linux has no friendly and secure methods to install commercial software. Despite the fact that users buy computers to run software and not to merely stare at desktop background. Compare this to Android where you can run malware and it cannot do anything except for annoying you with notifications.
- kjok 4mo ago> Compare this to Android where you can run malware and it cannot do anything except for annoying you with notifications. Are you sure it cannot do anything? Looking through various past malware/exploits, this doesn't seem to be the case.
- ashishb 4mo agoAlways run third-party code (especially npm packages) inside a sandbox, take your pick: ai-jail, bubblewrap, seatbelt, or amazing-sandbox (the last one, I wrote for myself after trying all others).
- codedokode 4mo agoBubblewrap doesn't help with protecting DBus bus (and you can do a lot with unrestricted access to DBus), accessing the GPU, Wayland and audio daemon. It cannot protect /proc and /sys filesystems. So it works only for simplest CLI programs. As I remember, some Node.js utility (maybe npm) had a bug where it hang if there were no /proc filesystem because it couldn't figure out how many CPUs are available. Telegram crashes because of Apple's Grand Central Dispatch library if run in a sandbox without /proc. That's how bad things are. Also, Chrome and Electron apps use SUID binaries to build their own sandbox. How would you sandbox an Electron app (obviously running an untrusted SUID binary is not an option)?
- padjo 4mo agoSo in summary: - a writable shared global cache is made available to PRs opened from forks by randomers. - that cache is reused in the deploy pipeline - deploys can be made with a single authentication factor, stored on the CI server - the repository apparently does nothing to check for malicious deploys, delegating that to 3rd parties to do after the code is in the wild. - by default the package manager runs random code when a package is updated What a world we live in.
- olejorgenb 4mo agoAnd the gotcha has been known about since 2014: > This is the class of attack documented by Adnan Khan in 2024. It's not a TanStack-specific bug; it's a known GitHub Actions design issue that requires conscious mitigation. While it seems the maintainers kinda went-out-of-their way to enable this - GitHub could easily have at least turned of cache-sharing between fork jobs and the main jobs...
- deleted 4mo ago[deleted]
- hoppp 4mo agoJust don't use NPM . Thats the lesson for me. Sadly the rust ecosystem will be the same because the dependency management is not better
- deleted 4mo ago[deleted]
- vorsken 4mo ago[dead]
- dcastm 4mo agoReminder to use a cooldown period: https://dylancastillo.co/til/securing-package-managers.html https://dylancastillo.co/til/securing-package-managers.html
- lacymorrow 4mo ago[flagged]
- LelouBil 4mo agoWhen will GitHub deprecate pull_request_target and make something where any shared aspect (like cache or secrets) are explicitly opt-in in the YAML ?
- luisb_24 4mo ago[flagged]
- deleted 4mo ago[deleted]
- isityettime 4mo agoOpen-source projects need a home with a coherent trust model for CI and release workflows. It's ridiculous that this kind of cache poisoning is even possible, and that it's the responsibility for each team to audit their configuration N different ways instead of Microsoft's responsibility to run a platform that works right. We have no hope of getting away from situations like this if everyone stays on GHA.
- Damianf19 4mo ago[dead]
- getrundoc 4mo agowow
- squid-protocol 4mo ago[flagged]
- pixel_popping 4mo agoPeople haven't discovered Git proxies yet? :/ Why are they using their own key from within userland?