6 ms·

Public secrets exposure leads to supply chain attack on GitHub CodeQL

297 points by cyberbender 1y ago

helsinki 1y ago
As someone with the last name Prater—derived from Praetorian—I really wish I owned praetorian.com.
- ratg13 1y ago
  You would have had to have this thought prior to the release of the movie “The Net” in 1995
- smoyer 1y ago
  Their gokart project was awesome!
ashishb 1y ago
I am getting more and more convinced that CI and CD should be completely separate environments. Compromise of CI should not lead to token leaks related to CD.
- mdaniel 1y ago
  This area is near and dear to my heart, and I would offer that the solution isn't to decouple CD over into its own special little thing but rather to make the CD "multi factor" in that it must be "sub":"repo:octo-org/octo-repo:environment:prod"[1] and feel free to sprinkle in any other [fun claims][] you'd like to harden that system 1: https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services#:~:text=you%20can%20configure%20deployment%20rules%20on%20an%20environment%20to%20restrict%20which%20branches%20and%20tags%20can%20deploy%20to%20the%20environment%20or%20access%20environment%20secrets https://docs.github.com/en/actions/security-for-github-actio... fun claims: https://github.com/github/actions-oidc-debugger#readme https://github.com/github/actions-oidc-debugger#readme
  - ashishb 1y ago
    Doable but I would prefer a complete isolation for simplicity.
    - thund 1y ago
      there are ways to isolate code from CI from CD, it's just not as easy as setting up the classic repo. One can use multiple repos for example, or run CI and CD with different products.
- nrvn 1y ago
  This is essentially how separation of duties(and concerns) looks like. And this is how some of the good examples of projects work. Specific techniques and tooling and specific boundaries of CI and CD vary depending on the nature of the end product but conceptually you are absolutely right.
udev4096 1y ago
Using public github actions is just asking for trouble and more so without analyzing the workflow's procedure. Instead, just host one yourself using woodpecker or countless other great CI builders (circle, travis, gitlab, etc)
Sytten 1y ago
An again this would not be so bad an impact if github finally pushed their immutable actions [1]. I sound like a broken record since I keep repeating that this would solve like 70%+ of the scope of attacks on gha today. You would think that the weekly disaster they have would finally make them launch it. [1] https://github.com/features/preview/immutable-actions https://github.com/features/preview/immutable-actions
- thund 1y ago
  They probably have good reasons if it's still in preview, that could be serious bugs, security gaps, potential breaking changes that would cause more harm than good if rushed, etc
  - intelVISA 1y ago
    Too much stakeholder alignment?
    - tanepiper 1y ago
      More like last year they laid off a whole bunch of people. We've been waiting for several open tickets on GitHub to be picked up, some were but seem to be abandoned and others just ignored.
  - 1oooqooq 1y ago
    the only reason any company does or don't anything: not required for sales. in 2019 i saw a fortune500 tech company put in place their own vulnerability scanner internal application which included this feature for our enterprise github repos. the tool was built and deployed to an old Linux docker image that was never updated to not be the target of the attack they were preventing... they never vetted to random version they started with either. i guess one can still use zip bomb or even the xz backdoor for extra irony points when attacking that system. anyway, the people signing github checks also get promoted by pretending to implement that feature internally.
junto 1y ago
They weren’t kidding on the response time. Very impressive from GitHub.
- belter 1y ago
  Not very impressive to have an exposed public token with full write credentials...
  - 1a527dd5 1y ago
    Trying my best not to break the no snark rule [1], but I'm sure your code is 100% bullet proof against all current and future-yet-invented-attacks. [1] _and failing_.
    - belter 1y ago
      [flagged]
    - atoav 1y ago
      Nobody is immune against mistakes, but a certain class of mistakes¹ should never ever happen to anyone who should know better. And that in my book is anybody who has their code used by more people than themselves. I am not saying devs aren't allowed to make stupid mistakes, but if we let civil engineers have their bridges collapse with an "shit happens" -attitude trust in civil engineering would be questionable at best. So yeah shit happens to us devs, but we should be shamed if it was preventable by simply knowing the basics. So my opinion is anybody who writes code that is used by others should feel a certain danger-tingle whenever a secret or real user data is put literally anywhere. To all beginners that just means that when handling secrets, instead of pressing on, you should pause and make an exhaustive list of who would have read/write access to the secret under which conditions and whether that is intended. And with things that are world-readable like a public repo, this is especially crucial. Another one may or may not be your shells history, the context of your environment variables, whatever you copy-paste into the browser-searchbar/application/LLM/chat/comment section of your choice etc. If you absolutely have to store secrets/private user data in files within a repo it is a good idea to add the following to your .gitignore: *.private *.private.* And then every such file has to have ".private." within the filename (e.g. credentials.private.json), this not only marks it to yourself, it also prevents you to mix up critical with mundane configuration. But better is to spend a day to think about where secrets/user data really should be stored and how to manage them properly. ¹: a non-exhaustive list of other such mistakes: mistaking XOR for encryption, storing passwords in plaintext, using hardcoded credentials, relying on obscurity for security, sending data unencrypted over HTTP, not hashing passwords, using weak hash functions like MD5 or SHA-1, no input validation to stiff thst goes into your database, trusting user input blindly, buffer overflows due to unchecked input, lack of access control, no user authentication, using default admin credentials, running all code as administrator/root without dropping priviledges, relying on client-side validation for security, using self-rolled cryptographic algorithms, mixing authentication and authorization logic, no session expiration or timeout, predictable session IDs, no patch management or updates, wide-open network shares, exposing internal services to the internet, trusting data from cookies or query strings without verification, etc
bloqs 1y ago
This sites performance is so bad i can barely scroll
nyrikki 1y ago
No mention why this temp token had rights to do things like create a new deployments and generate artifact attestations? For their fix, they disabled debug logs...but didn't answer if they changed the temp tokens permissions to something more appropriate for a code analysis engine.
- Elucalidavah 1y ago
  > For their fix, they disabled debug logs For their quick fix, hopefully not for their final fix.
- arccy 1y ago
  just goes to show how lax microsoft is about their security. nobody should trust them.
- beaugunderson 1y ago
  Temporary action tokens have full write by default; you have to explicitly opt for a read-only version. > Read and write permissions > Workflows have read and write permissions in the repository for all scopes. If you read this line of the documentation (https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token https://docs.github.com/en/actions/security-for-github-actio...) you might think otherwise: > If the default permissions for the GITHUB_TOKEN are restrictive, you may have to elevate the permissions to allow some actions and commands to run successfully. But I can confirm that in our GitHub organization "Read and write permissions" was the default, and thus that line of documentation makes no sense.
- Pathogen-David 1y ago
  If the GitHub Actions temporary token does not have workflow-defined permissions scope, it defaults either to a permissive or restricted default scope based on the repository's setting. This setting can also be configured at the organization level to restrict all repos owned by the org. Historically the only choice was permissive by default, so this is unfortunately the setting used by older organizations and repos. When a new repo is created, the default is inherited from the parent organization, so this insecure default tends to stick around if nobody bothers to change it. (There is no user-wide setting, so new repos owned by a user will use the restricted default. I believe newly created orgs use the better default.) [0]: https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#permissions-for-the-github_token https://docs.github.com/en/actions/security-for-github-actio...
ryao 1y ago
I put CodeQL in use in OpenZFS PRs. This is not an issue for OpenZFS. None of our code is secret. :)
- deleted 1y ago
  [deleted]
- asmosoinio 1y ago
  I don't think this is a good take: Even if your code is not secret, the attack could add anything to your code or release artifacts. Luckily it was quickly remedied at least.
atxtechbro 1y ago
Is this fixed?
- lsllc 1y ago
  It's in the article (and the comments here) -- yes, it was remediated within 3 hours of being reported back in January by GitHub.

297 points by cyberbender 1y ago

helsinki 1y ago

As someone with the last name Prater—derived from Praetorian—I really wish I owned praetorian.com.

ratg13 1y ago
You would have had to have this thought prior to the release of the movie “The Net” in 1995
smoyer 1y ago
Their gokart project was awesome!

ashishb 1y ago

I am getting more and more convinced that CI and CD should be completely separate environments. Compromise of CI should not lead to token leaks related to CD.

mdaniel 1y ago
This area is near and dear to my heart, and I would offer that the solution isn't to decouple CD over into its own special little thing but rather to make the CD "multi factor" in that it must be "sub":"repo:octo-org/octo-repo:environment:prod"[1] and feel free to sprinkle in any other [fun claims][] you'd like to harden that system 1: https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services#:~:text=you%20can%20configure%20deployment%20rules%20on%20an%20environment%20to%20restrict%20which%20branches%20and%20tags%20can%20deploy%20to%20the%20environment%20or%20access%20environment%20secrets https://docs.github.com/en/actions/security-for-github-actio... fun claims: https://github.com/github/actions-oidc-debugger#readme https://github.com/github/actions-oidc-debugger#readme
- ashishb 1y ago
  Doable but I would prefer a complete isolation for simplicity.
  - thund 1y ago
    there are ways to isolate code from CI from CD, it's just not as easy as setting up the classic repo. One can use multiple repos for example, or run CI and CD with different products.
nrvn 1y ago
This is essentially how separation of duties(and concerns) looks like. And this is how some of the good examples of projects work. Specific techniques and tooling and specific boundaries of CI and CD vary depending on the nature of the end product but conceptually you are absolutely right.

udev4096 1y ago

Using public github actions is just asking for trouble and more so without analyzing the workflow's procedure. Instead, just host one yourself using woodpecker or countless other great CI builders (circle, travis, gitlab, etc)

Sytten 1y ago

An again this would not be so bad an impact if github finally pushed their immutable actions [1]. I sound like a broken record since I keep repeating that this would solve like 70%+ of the scope of attacks on gha today. You would think that the weekly disaster they have would finally make them launch it. [1] https://github.com/features/preview/immutable-actions https://github.com/features/preview/immutable-actions

thund 1y ago
They probably have good reasons if it's still in preview, that could be serious bugs, security gaps, potential breaking changes that would cause more harm than good if rushed, etc
- intelVISA 1y ago
  Too much stakeholder alignment?
  - tanepiper 1y ago
    More like last year they laid off a whole bunch of people. We've been waiting for several open tickets on GitHub to be picked up, some were but seem to be abandoned and others just ignored.
- 1oooqooq 1y ago
  the only reason any company does or don't anything: not required for sales. in 2019 i saw a fortune500 tech company put in place their own vulnerability scanner internal application which included this feature for our enterprise github repos. the tool was built and deployed to an old Linux docker image that was never updated to not be the target of the attack they were preventing... they never vetted to random version they started with either. i guess one can still use zip bomb or even the xz backdoor for extra irony points when attacking that system. anyway, the people signing github checks also get promoted by pretending to implement that feature internally.

junto 1y ago

They weren’t kidding on the response time. Very impressive from GitHub.

belter 1y ago
Not very impressive to have an exposed public token with full write credentials...
- 1a527dd5 1y ago
  Trying my best not to break the no snark rule [1], but I'm sure your code is 100% bullet proof against all current and future-yet-invented-attacks. [1] _and failing_.
  - belter 1y ago
    [flagged]
  - atoav 1y ago
    Nobody is immune against mistakes, but a certain class of mistakes¹ should never ever happen to anyone who should know better. And that in my book is anybody who has their code used by more people than themselves. I am not saying devs aren't allowed to make stupid mistakes, but if we let civil engineers have their bridges collapse with an "shit happens" -attitude trust in civil engineering would be questionable at best. So yeah shit happens to us devs, but we should be shamed if it was preventable by simply knowing the basics. So my opinion is anybody who writes code that is used by others should feel a certain danger-tingle whenever a secret or real user data is put literally anywhere. To all beginners that just means that when handling secrets, instead of pressing on, you should pause and make an exhaustive list of who would have read/write access to the secret under which conditions and whether that is intended. And with things that are world-readable like a public repo, this is especially crucial. Another one may or may not be your shells history, the context of your environment variables, whatever you copy-paste into the browser-searchbar/application/LLM/chat/comment section of your choice etc. If you absolutely have to store secrets/private user data in files within a repo it is a good idea to add the following to your .gitignore: *.private *.private.* And then every such file has to have ".private." within the filename (e.g. credentials.private.json), this not only marks it to yourself, it also prevents you to mix up critical with mundane configuration. But better is to spend a day to think about where secrets/user data really should be stored and how to manage them properly. ¹: a non-exhaustive list of other such mistakes: mistaking XOR for encryption, storing passwords in plaintext, using hardcoded credentials, relying on obscurity for security, sending data unencrypted over HTTP, not hashing passwords, using weak hash functions like MD5 or SHA-1, no input validation to stiff thst goes into your database, trusting user input blindly, buffer overflows due to unchecked input, lack of access control, no user authentication, using default admin credentials, running all code as administrator/root without dropping priviledges, relying on client-side validation for security, using self-rolled cryptographic algorithms, mixing authentication and authorization logic, no session expiration or timeout, predictable session IDs, no patch management or updates, wide-open network shares, exposing internal services to the internet, trusting data from cookies or query strings without verification, etc

bloqs 1y ago

This sites performance is so bad i can barely scroll

nyrikki 1y ago

No mention why this temp token had rights to do things like create a new deployments and generate artifact attestations? For their fix, they disabled debug logs...but didn't answer if they changed the temp tokens permissions to something more appropriate for a code analysis engine.

Elucalidavah 1y ago
> For their fix, they disabled debug logs For their quick fix, hopefully not for their final fix.
arccy 1y ago
just goes to show how lax microsoft is about their security. nobody should trust them.
beaugunderson 1y ago
Temporary action tokens have full write by default; you have to explicitly opt for a read-only version. > Read and write permissions > Workflows have read and write permissions in the repository for all scopes. If you read this line of the documentation (https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token https://docs.github.com/en/actions/security-for-github-actio...) you might think otherwise: > If the default permissions for the GITHUB_TOKEN are restrictive, you may have to elevate the permissions to allow some actions and commands to run successfully. But I can confirm that in our GitHub organization "Read and write permissions" was the default, and thus that line of documentation makes no sense.
Pathogen-David 1y ago
If the GitHub Actions temporary token does not have workflow-defined permissions scope, it defaults either to a permissive or restricted default scope based on the repository's setting. This setting can also be configured at the organization level to restrict all repos owned by the org. Historically the only choice was permissive by default, so this is unfortunately the setting used by older organizations and repos. When a new repo is created, the default is inherited from the parent organization, so this insecure default tends to stick around if nobody bothers to change it. (There is no user-wide setting, so new repos owned by a user will use the restricted default. I believe newly created orgs use the better default.) [0]: https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#permissions-for-the-github_token https://docs.github.com/en/actions/security-for-github-actio...

ryao 1y ago

I put CodeQL in use in OpenZFS PRs. This is not an issue for OpenZFS. None of our code is secret. :)

deleted 1y ago
[deleted]
asmosoinio 1y ago
I don't think this is a good take: Even if your code is not secret, the attack could add anything to your code or release artifacts. Luckily it was quickly remedied at least.

atxtechbro 1y ago

Is this fixed?

lsllc 1y ago
It's in the article (and the comments here) -- yes, it was remediated within 3 hours of being reported back in January by GitHub.