7 ms·
co-founder of PostHog here. We were a victim of this attack. We had a bunch of packages published a couple of hours ago. The main packages/versions affected wer
by timgl 10mo ago
co-founder of PostHog here. We were a victim of this attack. We had a bunch of packages published a couple of hours ago. The main packages/versions affected were:
- posthog-node 4.18.1, 5.13.3 and 5.11.3
- posthog-js 1.297.3
- posthog-react-native 4.11.1
- posthog-docusaurus 2.0.6
We've rotated keys and passwords, unpublished all affected packages and have pushed new versions, so make sure you're on the latest version of our SDKs.
We're still figuring out how this key got compromised, and we'll follow up with a post-mortem. We'll update status.posthog.com with more updates as well.
- spiderfarmer 10mo agoIf we don't know how it got compromised, chances are this attack is still spreading?
- brabel 10mo agoIf anything people should use an older version of the packages. Your newest versions had just been compromised, why should anyone believe this time and next time it will be different?!
- timgl 10mo agoThe packages were published using a compromised key directly, not through our ci/cd. We rolled the key, and published a new clean version from our repo through our CI/CD: https://github.com/PostHog/posthog-js/actions/runs/19630358112/job/56210138138 https://github.com/PostHog/posthog-js/actions/runs/196303581...
- progbits 10mo agoWhy do you keep using token auth? This is unacceptable negligence these days. NPM supports GitHub workflow OIDC and you can make that required, disabling all token access.
- timgl 10mo agoYep, we are moving to workflow OIDC as the next step in recovery.
- huflungdung 10mo ago[dead]
- junon 10mo agoOIDC is not a silver bullet either and has its own set of vectors to consider too. If it works for your org model then great, but it doesn't solve every common scenario.
- woodruffw 10mo agoTrusted Publishing addresses the vector here, which is arbitrary persistence and delayed use of credentials by attackers. You're right that it's not a silver bullet (anything claiming to be one is almost certainly a financially induced lie), but it eliminates/foreshortens the attack staging window significantly.
- junon 10mo agoIt's not a silver bullet in that it can open up other vectors of much more probable attack. If I have a FOSS project where there are other contributors with write access but not publish access (very common) OIDC now allows them to publish via workflows, opening you up to Jia Tan type attacks. Like I said, if it works for your org, great. Otherwise it can cause bigger problems.
- woodruffw 10mo agoSorry, what's the threat model in which you're giving people write access to your repository but assuming that they can't publish? To establish a baseline here: the assumption with Trusted Publishing is that people are already using CI/CD to publish to places like PyPI. All Trusted Publishing does is replace the user-managed credential with a machine-intermediated one. In other words: if someone had write access to your repository before, Trusted Publishing does not change their privilege, since they could always push a new CI/CD workflow that uses your manual publishing secret directly. All Trusted Publishing does is formalize the treatment of the source repository as a source of truth, and introduce some misuse-resistance properties (short-lived credentials, minimal self-scoping). If this isn't your user model, that's perfectly fine. But in that case you shouldn't be using CI/CD to publish at all (which is also fine); it's not an issue with Trusted Publishing per se.
- _alternator_ 10mo agoGlad you updated on this front-page post. Your Twitter post is buried on p3 for me right now. Good luck on the recovery and hopefully this helps someone.
- Y_Y 10mo ago> so make sure you're on the latest version of our SDKs. Probably even safer to not have been on the latest version in the first place. Or safer again not to use software this vulnerable.
- tclancy 10mo agoPopularity and vulnerability go hand in hand though. You could be pretty safe by only using packages with zero stars on GitHub, but would you be happy or productive?
- BowBun 10mo agoAs a user of Posthog, this statement is absurd: > Or safer again not to use software this vulnerable. Nearly all software you use is susceptible to vulnerabilities, whether it's malicious or enterprise taking away your rights. It's in bad taste to make a comment about "not using software this vulnerable" when the issue was widespread in the ecosystem and the vendor is already being transparent about it. The alternative is you shame them into not sharing this information, and we're all worse for it.
- bilalq 10mo agoYou're probably already planning this, but please setup an alarm to fire off if a new package release is published that is not correlated with a CI/CD run.
- euph0ria 10mo agoVery nice way of putting it, kudos!
- twistedpair 10mo agoThis is built in NPM. You can get an email on every pkg publishing. Sure, it might be a little bit of noise, but if you get a notice @ 3am of an unexpected publishing, you can jump on unpublishing it.
- mbreese 10mo agoOr require manual intervention to publish a new package. I'm not sure why we need to have a fully automated pipeline here to go from CI/CD to public package release. It seems like having some kind of manual user interaction to push a new version of a library would be a good thing.
- magicalhippo 10mo agoI get that it can be useful sometimes. But requiring physical MFA to make a package available to the general public seems like a no-brainer to me. Users who really want to could opt in to the bleeding edge.
- woodruffw 10mo agoThe basic issue with manual interaction is a question of authority: a pretty common problem for companies (and open source groups) is when $EARLY_EMPLOYEE/$EARLY_CONTRIBUTOR creates and owns the entire publishing process for a key package, and then leaves without performing a proper transfer of responsibility. This essentially locks the company/group out of its own work, and increases support load on community maintained indices to essentially adjudicate rightful ownership of the package name. (There are a variety of ways to solve this, but the one I like best is automated publishing a la Trusted Publishing with environment mediated manual signoffs. GitHub and other CI/CD providers enable this.)
- silverlight 10mo agoDid the client side JS being infected produce any issues which would have affected end users? As in if a web owner were on an affected version and deployed during the window would the end user of their site have had any negative impact?
- timgl 10mo agoNo, just the host that was running the package (the exploit was pretty generic and not targeted at PostHog specifically). In fact, so far we think there were 0 production deployments of PostHog because the package was only live for a little bit.
- silverlight 10mo agoGlad to hear the impact was so muted. Thank you for the response!