5 ms·

Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from o

Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from open source components. Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been devilishly difficult before then without significant upfront work. Fetishising complexity might be a bit strong here. Seeing more posts like this is a good thing.

mLuby 7y ago
You make a good counterpoint: it's awesome that this can be done concisely enough these days to fit in a hobby blog post. That's new and demonstrates the approachability of these tools. At the same time, OP is right: a business doing this is making poor engineering decisions.
- kjeetgill 7y ago
  We don't know that yet! The post explains what they did but not the why of the problem. Maybe they have a good reason, maybe they don't. But no need to jump to conclusions. I've seen plenty of bs fragile ducktape databases. Some of them exist because because someone wanted to hand-roll everything. And some exist because they were afraid to invent anything and ended up hacking together a bunch of garbage MySQL nodes. Because they didn't want "reinvent the wheel". Specifics matter!
humbleMouse 7y ago
I agree my comment may be a knee-jerk reaction, and it is quite a good technical article with clear examples. I also agree that the open source tech used in this solution is very powerful and it is great we are at a point where a solution like this could exist. I love avro schema registries and kafka just as much as the next guy.
- kjeetgill 7y ago
  So question: is there a good reason to use Avro over protobufs? Just from a hand wavy aesthetic perspective I like protobufs but I'd like to get a clear picture.

mLuby 7y ago

You make a good counterpoint: it's awesome that this can be done concisely enough these days to fit in a hobby blog post. That's new and demonstrates the approachability of these tools. At the same time, OP is right: a business doing this is making poor engineering decisions.

kjeetgill 7y ago
We don't know that yet! The post explains what they did but not the why of the problem. Maybe they have a good reason, maybe they don't. But no need to jump to conclusions. I've seen plenty of bs fragile ducktape databases. Some of them exist because because someone wanted to hand-roll everything. And some exist because they were afraid to invent anything and ended up hacking together a bunch of garbage MySQL nodes. Because they didn't want "reinvent the wheel". Specifics matter!

humbleMouse 7y ago

I agree my comment may be a knee-jerk reaction, and it is quite a good technical article with clear examples. I also agree that the open source tech used in this solution is very powerful and it is great we are at a point where a solution like this could exist. I love avro schema registries and kafka just as much as the next guy.

kjeetgill 7y ago
So question: is there a good reason to use Avro over protobufs? Just from a hand wavy aesthetic perspective I like protobufs but I'd like to get a clear picture.