5 ms·

It was the PYPI_PUBLISH token which was in our github project as an env var, that got sent to trivvy. We have deleted all our pypi publishing tokens. Our acco

It was the PYPI_PUBLISH token which was in our github project as an env var, that got sent to trivvy. We have deleted all our pypi publishing tokens. Our accounts had 2fa, so it's a bad token here. We're reviewing our accounts, to see how we can make it more secure (trusted publishing via jwt tokens, move to a different pypi account, etc.).

redrove 6mo ago
How did PYPI_PUBLISH lead to a full GH account takeover?
- franktankbank 6mo ago
  Don't hold your breath for an answer.
- ezekg 6mo ago
  I'd imagine the attacker published a new compromised version of their package, which the author eventually downloaded, which pwned everything else.
- chunky1994 6mo ago
  Their Personal Access Token must’ve been pwned too, not sure through what mechanism though
  - Imustaskforhelp 6mo ago
    They have written about it on github to my question: Trivvy hacked (https://www.aquasec.com/blog/trivy-supply-chain-attack-what-you-need-to-know/ https://www.aquasec.com/blog/trivy-supply-chain-attack-what-...) -> all circleci credentials leaked -> included pypi publish token + github pat -> | WE DISCOVER ISSUE | -> pypi token deleted, github pat deleted + account removed from org access, trivvy pinned to last known safe version (v0.69.3) What we're doing now: Block all releases, until we have completed our scans Working with Google's mandiant.security team to understand scope of impact Reviewing / rotating any leaked credentials https://github.com/BerriAI/litellm/issues/24518#issuecomment-4119972374 https://github.com/BerriAI/litellm/issues/24518#issuecomment...
    - franktankbank 6mo ago
      Does that explain how circleci was publishing commits and closing issues?
    - celticninja 6mo ago
      69.3 isnt safe. The safe thing to do is remove all trivy access. or failing that version. 0.35 is the last and AFAIK only safe version. https://socket.dev/blog/trivy-under-attack-again-github-actions-compromise https://socket.dev/blog/trivy-under-attack-again-github-acti...
      - Imustaskforhelp 6mo ago
        I have sent your message to the developer on github and they have changed the version to 0.35.0 ,so thanks. https://github.com/BerriAI/litellm/issues/24518#issuecomment-4120401246 https://github.com/BerriAI/litellm/issues/24518#issuecomment...
mike_hearn 6mo ago
Perhaps it's too obvious but ... just running the publish process locally, instead of from CI, would help. Especially if you publish from a dedicated user on a Mac where the system keychain is pretty secure.
- staticassertion 6mo ago
  I'm not sure how. Their local system seems just as likely to get compromised through a `pip install` or whatever else. In CI they could easily have moved `trivy` to its own dedicated worker that had no access to the PYPI secret, which should be isolated to the publish command and only the publish command.
  - mike_hearn 6mo ago
    User isolation works, the keychain isolation works. On macOS tokens stored in the keychain can be made readable only by specific apps, not anything else. It does require a bit of infrastructure - ideally a Mac app that does the release - but nothing you can't vibe code quickly.
    - staticassertion 6mo ago
      That's true, but it seems far more complex than just moving trivy to a separate workerflow with no permissions and likely physical isolation between it and a credential. I'm pretty wary of the idea that malware couldn't just privesc - it's pretty trivial to obtain root on a user's laptop. Running as a separate, unprivileged user helps a ton, but again, I'm skeptical of this vs just using a github workflow.
      - mike_hearn 6mo ago
        I'm looking for more general solutions. "Properly configure Trivy" is too specific, it's obvious in hindsight but not before. Privilege escalation on macOS is very hard indeed. Apple have been improving security for a long time, it is far, far ahead of Linux or Windows in this regard. The default experience in Xcode is that a release-mode app you make will be sandboxed, undebuggable, have protected keychain entries other apps can't read, have a protected file space other apps can't read, and its own code will also be read-only to other apps. So apps can't interfere with each other or escalate to each other's privileges even when running as the same UNIX user. And that's the default, you don't have to do anything to get that level of protection.
NewJazz 6mo ago
Are you spelling it with two vs on purpose?

redrove 6mo ago

How did PYPI_PUBLISH lead to a full GH account takeover?

franktankbank 6mo ago
Don't hold your breath for an answer.
ezekg 6mo ago
I'd imagine the attacker published a new compromised version of their package, which the author eventually downloaded, which pwned everything else.
chunky1994 6mo ago
Their Personal Access Token must’ve been pwned too, not sure through what mechanism though
- Imustaskforhelp 6mo ago
  They have written about it on github to my question: Trivvy hacked (https://www.aquasec.com/blog/trivy-supply-chain-attack-what-you-need-to-know/ https://www.aquasec.com/blog/trivy-supply-chain-attack-what-...) -> all circleci credentials leaked -> included pypi publish token + github pat -> | WE DISCOVER ISSUE | -> pypi token deleted, github pat deleted + account removed from org access, trivvy pinned to last known safe version (v0.69.3) What we're doing now: Block all releases, until we have completed our scans Working with Google's mandiant.security team to understand scope of impact Reviewing / rotating any leaked credentials https://github.com/BerriAI/litellm/issues/24518#issuecomment-4119972374 https://github.com/BerriAI/litellm/issues/24518#issuecomment...
  - franktankbank 6mo ago
    Does that explain how circleci was publishing commits and closing issues?
  - celticninja 6mo ago
    69.3 isnt safe. The safe thing to do is remove all trivy access. or failing that version. 0.35 is the last and AFAIK only safe version. https://socket.dev/blog/trivy-under-attack-again-github-actions-compromise https://socket.dev/blog/trivy-under-attack-again-github-acti...
    - Imustaskforhelp 6mo ago
      I have sent your message to the developer on github and they have changed the version to 0.35.0 ,so thanks. https://github.com/BerriAI/litellm/issues/24518#issuecomment-4120401246 https://github.com/BerriAI/litellm/issues/24518#issuecomment...

mike_hearn 6mo ago

Perhaps it's too obvious but ... just running the publish process locally, instead of from CI, would help. Especially if you publish from a dedicated user on a Mac where the system keychain is pretty secure.

staticassertion 6mo ago
I'm not sure how. Their local system seems just as likely to get compromised through a `pip install` or whatever else. In CI they could easily have moved `trivy` to its own dedicated worker that had no access to the PYPI secret, which should be isolated to the publish command and only the publish command.
- mike_hearn 6mo ago
  User isolation works, the keychain isolation works. On macOS tokens stored in the keychain can be made readable only by specific apps, not anything else. It does require a bit of infrastructure - ideally a Mac app that does the release - but nothing you can't vibe code quickly.
  - staticassertion 6mo ago
    That's true, but it seems far more complex than just moving trivy to a separate workerflow with no permissions and likely physical isolation between it and a credential. I'm pretty wary of the idea that malware couldn't just privesc - it's pretty trivial to obtain root on a user's laptop. Running as a separate, unprivileged user helps a ton, but again, I'm skeptical of this vs just using a github workflow.
    - mike_hearn 6mo ago
      I'm looking for more general solutions. "Properly configure Trivy" is too specific, it's obvious in hindsight but not before. Privilege escalation on macOS is very hard indeed. Apple have been improving security for a long time, it is far, far ahead of Linux or Windows in this regard. The default experience in Xcode is that a release-mode app you make will be sandboxed, undebuggable, have protected keychain entries other apps can't read, have a protected file space other apps can't read, and its own code will also be read-only to other apps. So apps can't interfere with each other or escalate to each other's privileges even when running as the same UNIX user. And that's the default, you don't have to do anything to get that level of protection.

NewJazz 6mo ago

Are you spelling it with two vs on purpose?