4 ms·

etcd has had numerous liveness and safety bugs, with one happening as recently as December of 2025. Would you consider that a correct implementation? You may b

etcd has had numerous liveness and safety bugs, with one happening as recently as December of 2025. Would you consider that a correct implementation? You may be interested in knowing that the largest managed Kubernetes service in the world (AWS EKS) ripped out etcd for in favor of their homegrown consensus service for large scale EKS clusters: https://aws.amazon.com/blogs/containers/under-the-hood-amazon-eks-ultra-scale-clusters/ https://aws.amazon.com/blogs/containers/under-the-hood-amazo...

fl0ki 1mo ago
etcd is some of the most amateur code I've ever seen, despite being one of the oldest and presumably most mature "infrastructure" projects written in Go. goBGP is arguably even worse. I don't have a third place in mind that's even worth mentioning relative to these two.
- alfons_foobar 1mo ago
  just curious, what problems do you see with gobgp? (I have only a rather basic familiarity with go, but was considering gobgp for an infra project...)
  - iscoelho 1mo ago
    Not speaking to their code, but to start GoBGP has the worst performance of any BGP daemon by a large margin [1]. [1] https://elegantnetwork.github.io/posts/comparing-open-source-bgp-stacks/ https://elegantnetwork.github.io/posts/comparing-open-source...
    - preisschild 1mo ago
      Garbage collected languages like Go will always have worse performance than lower level languages like C (frr and bird are implemented in C). Gobgp is great if you want to embed it directly into a Go app though. Talos Linux has done that recently.
      - tsss 1mo ago
        No they won't. They are generally faster in throughput than any non-gc application that isn't heavily hand optimized. Their problems are higher memory usage and unpredictable latency, not speed.
      - pjmlp 1mo ago
        Nope, it is a matter of skill as well. https://github.com/ixy-languages/ixy-languages https://github.com/ixy-languages/ixy-languages
  - fl0ki 1mo ago
    Just look at the code. Anywhere you like. There's nothing I can say that will be a substitute for seeing it yourself. I've had to use it as a library sometimes, and it's also really poorly designed as a library. There's no consistent principle for what is exported and what remains internal, so almost anything non-trivial you do may require you to copy-paste parts of the library code because something happened to not be exported. So you must think, okay sure, but in return for that you get a minimal and stable API. Nope! It also has some of the most API churn of any Go module, already up to /v4 and that's only counting the semver-major breakage they bothered to acknowledge, not the many semver papercuts along the way.
camkego 1mo ago
Sorry to pile on, but yeah, I wanted to use etcd during 2021 and 2022, around v3.5, but etcd had serious issues including silent data corruption. If you are curious, ask gemini flash "there were a number of etcd releases years ago where it seems a new wave of developers came in and started breaking everything"
- Cthulhu_ 1mo ago
  Why type out a prompt instead of link to an authoritative source? Gemini is not a source.
Thaxll 1mo ago
I'd like to know what you base your statement on that the Raft implementations in etcd or CockroachDB are incorrect. Your original paper does not mention those implementations, so where does that claim come from?
- gyesxnuibh 1mo ago
  Having run a fleet of 100s of etcd clusters for 10000s of rps, and the fact that upstream runs tests similar to antithesis and recently partnered with antithesis [0], and jepsen has tested it long ago as well [1]. Etcd's raft algorithm is fine. Someone even did a TLA+ proof on it in the last couple years[2]. Yes there was a correctness issue a few years ago but otherwise the person you're replying to doesn't know what they're talking about. Also those bugs have nothing to do with the raft implementation, but instead the state machine implemented on top. 0: https://etcd.io/blog/2025/autonomus_testing_with_antithesis/ https://etcd.io/blog/2025/autonomus_testing_with_antithesis/ 1: https://jepsen.io/analyses/etcd-3.4.3 https://jepsen.io/analyses/etcd-3.4.3 2: https://github.com/etcd-io/raft/pull/113 https://github.com/etcd-io/raft/pull/113
  - cobbzilla 1mo ago
    Is the correctness of its implementation of the algorithm unaffected by bugs in this state machine? Maybe I missed something.
    - gyesxnuibh 1mo ago
      The raft algorithm works and if you implemented a less complex state machine (like using a simpler kv store that doesn't need global event ordering via revisions and watches) it would work. That's what antithesis said they did to test the raft algorithms in the other article linked
  - iscoelho 1mo ago
    "there was a correction issue" is downplaying it. Etcd is truly the worst example of Raft. Etcd corruption and loss of quorum is extremely common in practice and the GitHub issues sit for years. The design is simple, the performance is modest, yet it still has still never been reliable, despite being marketed as so. I can't speak to whether this is specifically due to their Raft implementation, but I'd argue the entire codebase is over-engineered and questionable.

fl0ki 1mo ago

etcd is some of the most amateur code I've ever seen, despite being one of the oldest and presumably most mature "infrastructure" projects written in Go. goBGP is arguably even worse. I don't have a third place in mind that's even worth mentioning relative to these two.

alfons_foobar 1mo ago
just curious, what problems do you see with gobgp? (I have only a rather basic familiarity with go, but was considering gobgp for an infra project...)
- iscoelho 1mo ago
  Not speaking to their code, but to start GoBGP has the worst performance of any BGP daemon by a large margin [1]. [1] https://elegantnetwork.github.io/posts/comparing-open-source-bgp-stacks/ https://elegantnetwork.github.io/posts/comparing-open-source...
  - preisschild 1mo ago
    Garbage collected languages like Go will always have worse performance than lower level languages like C (frr and bird are implemented in C). Gobgp is great if you want to embed it directly into a Go app though. Talos Linux has done that recently.
    - tsss 1mo ago
      No they won't. They are generally faster in throughput than any non-gc application that isn't heavily hand optimized. Their problems are higher memory usage and unpredictable latency, not speed.
    - pjmlp 1mo ago
      Nope, it is a matter of skill as well. https://github.com/ixy-languages/ixy-languages https://github.com/ixy-languages/ixy-languages
- fl0ki 1mo ago
  Just look at the code. Anywhere you like. There's nothing I can say that will be a substitute for seeing it yourself. I've had to use it as a library sometimes, and it's also really poorly designed as a library. There's no consistent principle for what is exported and what remains internal, so almost anything non-trivial you do may require you to copy-paste parts of the library code because something happened to not be exported. So you must think, okay sure, but in return for that you get a minimal and stable API. Nope! It also has some of the most API churn of any Go module, already up to /v4 and that's only counting the semver-major breakage they bothered to acknowledge, not the many semver papercuts along the way.

camkego 1mo ago

Sorry to pile on, but yeah, I wanted to use etcd during 2021 and 2022, around v3.5, but etcd had serious issues including silent data corruption. If you are curious, ask gemini flash "there were a number of etcd releases years ago where it seems a new wave of developers came in and started breaking everything"

Cthulhu_ 1mo ago
Why type out a prompt instead of link to an authoritative source? Gemini is not a source.

Thaxll 1mo ago

I'd like to know what you base your statement on that the Raft implementations in etcd or CockroachDB are incorrect. Your original paper does not mention those implementations, so where does that claim come from?

gyesxnuibh 1mo ago
Having run a fleet of 100s of etcd clusters for 10000s of rps, and the fact that upstream runs tests similar to antithesis and recently partnered with antithesis [0], and jepsen has tested it long ago as well [1]. Etcd's raft algorithm is fine. Someone even did a TLA+ proof on it in the last couple years[2]. Yes there was a correctness issue a few years ago but otherwise the person you're replying to doesn't know what they're talking about. Also those bugs have nothing to do with the raft implementation, but instead the state machine implemented on top. 0: https://etcd.io/blog/2025/autonomus_testing_with_antithesis/ https://etcd.io/blog/2025/autonomus_testing_with_antithesis/ 1: https://jepsen.io/analyses/etcd-3.4.3 https://jepsen.io/analyses/etcd-3.4.3 2: https://github.com/etcd-io/raft/pull/113 https://github.com/etcd-io/raft/pull/113
- cobbzilla 1mo ago
  Is the correctness of its implementation of the algorithm unaffected by bugs in this state machine? Maybe I missed something.
  - gyesxnuibh 1mo ago
    The raft algorithm works and if you implemented a less complex state machine (like using a simpler kv store that doesn't need global event ordering via revisions and watches) it would work. That's what antithesis said they did to test the raft algorithms in the other article linked
- iscoelho 1mo ago
  "there was a correction issue" is downplaying it. Etcd is truly the worst example of Raft. Etcd corruption and loss of quorum is extremely common in practice and the GitHub issues sit for years. The design is simple, the performance is modest, yet it still has still never been reliable, despite being marketed as so. I can't speak to whether this is specifically due to their Raft implementation, but I'd argue the entire codebase is over-engineered and questionable.