7 ms·

What are good alternatives when you need a common "single source of truth" schema shared between multiple languages? We use protobuf between c# and Python.

ltbarcly3 1mo ago
The goal is not to have a single source of truth schema. That is a means to some other goal, and it's not even a good means. If you never change the schema then you don't have to worry about it, get things working and never look back. If you do change your schema from time to time, you need testing between the two systems. If you have good tests again a single source of truth is fully redundant, both systems are talking just fine. If you don't have tests things can and will break all the time even using protobuf.
- afavour 1mo ago
  > The goal is not to have a single source of truth schema. That is a means to some other goal, and it's not even a good means. It’s about data transmission. Being able to encode and decode in a type safe manner between different languages (and so, different platforms) is a goal that makes a lot of sense. > If you do change your schema from time to time, you need testing between the two systems Or you could just use a defined format that doesn’t require testing. I rarely use protobuf but I can see why people do. The guaranteed backwards compatibility is huge for people who can’t just publish a new web frontend at the drop of a hat.
  - ltbarcly3 1mo ago
    afavour it's been a long day already, maybe I can explain why you still need tests another day.
    - afavour 1mo ago
      You could always try being less condescending when replying to users and you might find there was no need for anyone to reply in the first place.
- kccqzy 1mo ago
  If you understand how to evolve protobuf schema definitions, then you don’t really need testing. You instinctively know how the parser works when it is parsing data with a different schema from what it expects. And that’s a powerful thing. If your things break even when using protobuf then you don’t grok protobuf. It’s probably not an exaggeration to say that being able to avoid tests between different systems who have different versions of the schema is a core goal of protobuf. Why? These two different systems are probably owned by different teams, and introducing explicit tests between different versions of them increases coupling between them.
  - ltbarcly3 1mo ago
    "you don't really need testing" Protobuf only allows you to add optional fields after the initial version, right? Because otherwise it would not be backwards compatible. Protobuf does not allow you to define contingent logic between fields. Optional fields are always nullable (or you must provide a default). This forces you to know all of this and use a method to see if it was actually set or just defaulted. So you have a ton of nullable/defaulted fields that likely are required to be filled in or not filled in, contingent on other values in the same struct. For example if the charge type is "purchase" then price must be not null and > 0. That sort of thing. This is so common you should just assume your app has a million little rules like this that are assumed. Protobuf does not help you here at all. You need some other logic to validate the data on top of protobuf. once you have that anyway protobuf's value is that it's expensive, requires build steps, is actually not fast, and forces you to distribute the schema between different apps somehow. If it's not obvious yet, lets say you are sending your charge structs and you have a bug where sometimes you don't set price. It's defaulted to 0 or -1 or whatever nonsense value the default is, or null, it doesn't matter it's not correct sometimes. That is why you need tests my guy. Protobuf can't fix this. If you use json the tests make sure everything protobuf does for you is done too. In a world where you have to write tests because protobuf can't force you to correctly set fields, you have tests already, and protobuf didn't help you at all. Whether you call it tests or input validation or whatever, protobuf definitions are insufficient, and when you have what is sufficient it 100% covers everything protobuf does 'for you'. If you don't get it at this point then lets just agree that you will never get it.
- andai 1mo ago
  Both sibling comments say one type of assurance makes the other irrelevant, but I would wager they cover different territory.
throw1234567891 1mo ago
Json schema?
IshKebab 1mo ago
I quite like the look of Typespec though I haven't used it much. I always thought Thrift was waaay better than any of the alternatives, but it always had terrible documentation and I think it died mainly because of that.

ltbarcly3 1mo ago

The goal is not to have a single source of truth schema. That is a means to some other goal, and it's not even a good means. If you never change the schema then you don't have to worry about it, get things working and never look back. If you do change your schema from time to time, you need testing between the two systems. If you have good tests again a single source of truth is fully redundant, both systems are talking just fine. If you don't have tests things can and will break all the time even using protobuf.

afavour 1mo ago
> The goal is not to have a single source of truth schema. That is a means to some other goal, and it's not even a good means. It’s about data transmission. Being able to encode and decode in a type safe manner between different languages (and so, different platforms) is a goal that makes a lot of sense. > If you do change your schema from time to time, you need testing between the two systems Or you could just use a defined format that doesn’t require testing. I rarely use protobuf but I can see why people do. The guaranteed backwards compatibility is huge for people who can’t just publish a new web frontend at the drop of a hat.
- ltbarcly3 1mo ago
  afavour it's been a long day already, maybe I can explain why you still need tests another day.
  - afavour 1mo ago
    You could always try being less condescending when replying to users and you might find there was no need for anyone to reply in the first place.
kccqzy 1mo ago
If you understand how to evolve protobuf schema definitions, then you don’t really need testing. You instinctively know how the parser works when it is parsing data with a different schema from what it expects. And that’s a powerful thing. If your things break even when using protobuf then you don’t grok protobuf. It’s probably not an exaggeration to say that being able to avoid tests between different systems who have different versions of the schema is a core goal of protobuf. Why? These two different systems are probably owned by different teams, and introducing explicit tests between different versions of them increases coupling between them.
- ltbarcly3 1mo ago
  "you don't really need testing" Protobuf only allows you to add optional fields after the initial version, right? Because otherwise it would not be backwards compatible. Protobuf does not allow you to define contingent logic between fields. Optional fields are always nullable (or you must provide a default). This forces you to know all of this and use a method to see if it was actually set or just defaulted. So you have a ton of nullable/defaulted fields that likely are required to be filled in or not filled in, contingent on other values in the same struct. For example if the charge type is "purchase" then price must be not null and > 0. That sort of thing. This is so common you should just assume your app has a million little rules like this that are assumed. Protobuf does not help you here at all. You need some other logic to validate the data on top of protobuf. once you have that anyway protobuf's value is that it's expensive, requires build steps, is actually not fast, and forces you to distribute the schema between different apps somehow. If it's not obvious yet, lets say you are sending your charge structs and you have a bug where sometimes you don't set price. It's defaulted to 0 or -1 or whatever nonsense value the default is, or null, it doesn't matter it's not correct sometimes. That is why you need tests my guy. Protobuf can't fix this. If you use json the tests make sure everything protobuf does for you is done too. In a world where you have to write tests because protobuf can't force you to correctly set fields, you have tests already, and protobuf didn't help you at all. Whether you call it tests or input validation or whatever, protobuf definitions are insufficient, and when you have what is sufficient it 100% covers everything protobuf does 'for you'. If you don't get it at this point then lets just agree that you will never get it.
andai 1mo ago
Both sibling comments say one type of assurance makes the other irrelevant, but I would wager they cover different territory.

throw1234567891 1mo ago

Json schema?

IshKebab 1mo ago

I quite like the look of Typespec though I haven't used it much. I always thought Thrift was waaay better than any of the alternatives, but it always had terrible documentation and I think it died mainly because of that.