6 ms·

I really like Caddy and use it on a few projects. Its a good piece of software that generally just works. Their latest security incident leaves me feeling some

I really like Caddy and use it on a few projects. Its a good piece of software that generally just works. Their latest security incident leaves me feeling somewhat uneasy though. The only announcement for it was on twitter[1] and a footnote on one of the releases[2]. I don't believe there was ever a mention of the incident on the official forum so if you weren't following them on Twitter you'd never find out. The general messaging around it was "a GitHub bug caused it" and its never really been followed up on publicly :\ [1]: https://twitter.com/caddyserver/status/1338324878441603073 https://twitter.com/caddyserver/status/1338324878441603073 [2]: https://github.com/caddyserver/caddy/releases/tag/v2.2.1 https://github.com/caddyserver/caddy/releases/tag/v2.2.1

francislavoie 5y ago
It wasn't a security incident, actually. It's true that "a GitHub bug caused it". It wasn't malicious. TLDR, a contributor made a tag on their own fork of Caddy, and for some reason our next release used their tag, because it turns out forks in GitHub aren't actual separate repos, but rather "still technically the same repo". It's really strange. Wasn't that contributor's fault either, they had no idea that would happen. All that happened is that the v2.2.2 git tag wasn't properly signed with Matt's signing key. There was no problem with the code at all. We've put in place checks during our CI actions to ensure that releases are always verified to be signed by Matt's key. See https://github.com/caddyserver/caddy/pull/3932 https://github.com/caddyserver/caddy/pull/3932 Specifically, commit https://github.com/caddyserver/caddy/commit/1d473ae924f0d52cb5a8f525e3e02b5167a4a5f1 https://github.com/caddyserver/caddy/commit/1d473ae924f0d52c... (which you'll notice is _not_ part of the Caddy repo, it was actually from a fork which was later deleted) triggered this GitHub Actions job for the release https://github.com/caddyserver/caddy/actions/runs/392345801 https://github.com/caddyserver/caddy/actions/runs/392345801 which we cancelled when we noticed it was happening. But we can't remove the tag from Go's caching server https://pkg.go.dev/github.com/caddyserver/caddy/v2@v2.2.2 https://pkg.go.dev/github.com/caddyserver/caddy/v2@v2.2.2 so it's kinda there forever. More context: https://twitter.com/mholt6/status/1337879764317564928 https://twitter.com/mholt6/status/1337879764317564928, https://twitter.com/danlsgiga/status/1338859470227529732 https://twitter.com/danlsgiga/status/1338859470227529732, that whole twitter thread has many branches, so click around to get the whole conversation.
- itsjloh 5y ago
  It’s great you’ve got some protections in place to prevent it from happening again. That inspires some confidence. Thanks for all the work yourself and Matt do on Caddy.
  - francislavoie 5y ago
    Thanks for the kind words :) Shoutout to https://github.com/mohammed90 https://github.com/mohammed90 who's been a huge part of our CI/CD efforts on Caddy (among many other things), including that tag verification PR linked above.

francislavoie 5y ago

It wasn't a security incident, actually. It's true that "a GitHub bug caused it". It wasn't malicious. TLDR, a contributor made a tag on their own fork of Caddy, and for some reason our next release used their tag, because it turns out forks in GitHub aren't actual separate repos, but rather "still technically the same repo". It's really strange. Wasn't that contributor's fault either, they had no idea that would happen. All that happened is that the v2.2.2 git tag wasn't properly signed with Matt's signing key. There was no problem with the code at all. We've put in place checks during our CI actions to ensure that releases are always verified to be signed by Matt's key. See https://github.com/caddyserver/caddy/pull/3932 https://github.com/caddyserver/caddy/pull/3932 Specifically, commit https://github.com/caddyserver/caddy/commit/1d473ae924f0d52cb5a8f525e3e02b5167a4a5f1 https://github.com/caddyserver/caddy/commit/1d473ae924f0d52c... (which you'll notice is _not_ part of the Caddy repo, it was actually from a fork which was later deleted) triggered this GitHub Actions job for the release https://github.com/caddyserver/caddy/actions/runs/392345801 https://github.com/caddyserver/caddy/actions/runs/392345801 which we cancelled when we noticed it was happening. But we can't remove the tag from Go's caching server https://pkg.go.dev/github.com/caddyserver/caddy/v2@v2.2.2 https://pkg.go.dev/github.com/caddyserver/caddy/v2@v2.2.2 so it's kinda there forever. More context: https://twitter.com/mholt6/status/1337879764317564928 https://twitter.com/mholt6/status/1337879764317564928, https://twitter.com/danlsgiga/status/1338859470227529732 https://twitter.com/danlsgiga/status/1338859470227529732, that whole twitter thread has many branches, so click around to get the whole conversation.

itsjloh 5y ago
It’s great you’ve got some protections in place to prevent it from happening again. That inspires some confidence. Thanks for all the work yourself and Matt do on Caddy.
- francislavoie 5y ago
  Thanks for the kind words :) Shoutout to https://github.com/mohammed90 https://github.com/mohammed90 who's been a huge part of our CI/CD efforts on Caddy (among many other things), including that tag verification PR linked above.