5 ms·

If that is really true, then we will of course update the comparison page. However, we have put it together from what we were able to find in Google Protocol Bu

by VStack 11y ago

If that is really true, then we will of course update the comparison page. However, we have put it together from what we were able to find in Google Protocol Buffer's own docs + stack overflow + googling. It is entirely possible that we made mistakes. Sending schema over the wire is not a good solution for anything else than point-to-point communication. An intermediate node would need every single schema transmitted along with every single messsage, or have another way to keep the schemas cached. That becomes complicated. The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. Then a protobuf message is not fully self describing. This might be easy to add, but it doesn't have it (according to Protobuf's own docs). We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level. However, we need to make space for some IAP specific fields coming later in the process (like cache references, column stores and more). Stuff that is IAP specific. That is why we chose to roll with our own encoding in the first place.

kentonv 11y ago
> Sending schema over the wire is not a good solution for anything else than point-to-point communication. OK, let's back up a moment. I am not entirely sure what "Schema / Class Id" in your table means. Your table claims Protobuf doesn't support it, but the text below is unclear on what you think Protobuf doesn't support. You frequently use the term "self-describing", but this could have two meanings: 1) Like JSON, where the names of all fields appear in the message, so that a human can read the message easily without external information. 2) Limited self-description in which field values can be identified and parsed, but their names are not available (perhaps replaced by numeric tags or indexes). Protobuf can support (1) by including the schema in the payload. I agree this is not commonly useful. Protobuf supports (2) natively, by virtue of being a TLV format (just like ION). Re-reading the page, it sounds like you are assuming the Protobuf format cannot be deciphered at all without the schema, but this simply isn't true. If you meant something else, please explain. > The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. I wrote that documentation. It doesn't mean what you think (my fault, perhaps). What it's saying is that the top-level message is a series of tag-value pairs with no explicit indication of where that series ends (on the assumption that you already know, e.g. based on EOF). Thus, if you concatenate two whole messages without adding any delimiter then it will look like one big message containing all the fields from both. However, each field within the message is clearly delimited and sub-messages are length-delimited therefore skipable. > We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level. ION is a TLV encoding like Protobuf. Cap'n Proto is fixed offsets + pointers. These are vastly different styles of encoding that enable different modes of use. You can certainly debate which is better but I don't think it's correct to describe the formats as "pretty close", unless you consider all binary formats to be "pretty close" to each other.
- jjenkov 11y ago
  You are right, the term "self describing" as used in our docs could be more clear. Being self describing means that you do not need a schema to make sense of a stream of data of that format. However, there is also a degree to which a data format can be self describing. A CSV file is reasonably self describing because you can see where one field ends and the next begins (at the comma / separator), and where one record ends and the next begins (new line). With a header line of column names a CSV file becomes more self describing, as you now also have a name indicating the semantic meaning of fields in that column. If a CSV file could somehow contain a specification of the data type of each column, it would be even more self describing etc. This is what we are trying to achieve with ION. If you need speed, you can omit most of the meta data like property names etc. If you need messages to be self describing, you can add a lot of meta data (like class / schema names + version, property names etc.). I apologize for having written incorrect documentation. If you wrote those docs for Google Protocol Buffers, part of that is on you. They are not exactly crystal clear ;-) (our doc's aren't either - still working on them!) Thank you for clearing up that Protobuf fields can be distinguished in a stream of Protobuf fields, even without schema. That was unclear to me before now. By the way, that is pretty clear in Cap'n Proto - your invention right? So - better docs already! And - thank you for clearing up the difference in the encoding of Cap'n Proto. Any link to where I can read about that encoding style in more details?
  - kentonv 11y ago
    > Any link to where I can read about that encoding style in more details? Hmm, I'm not aware of any literature other than what's on the Cap'n Proto web site. You can of course find the Cap'n Proto encoding documented here: https://capnproto.org/encoding.html https://capnproto.org/encoding.html The format is, of course, a lot like how in-memory data structures are laid out in C (fields of a struct have fixed offsets; variable-size fields are behind pointers). Unlike native pointers, though, Cap'n Proto's pointers are designed to be relocatable and easy to bounds-check, and they contain just enough type information for the message to be minimally self-describing (so that you can e.g. make a copy of a particular sub-object without knowing its schema).

by VStack 11y ago

kentonv 11y ago

> Sending schema over the wire is not a good solution for anything else than point-to-point communication. OK, let's back up a moment. I am not entirely sure what "Schema / Class Id" in your table means. Your table claims Protobuf doesn't support it, but the text below is unclear on what you think Protobuf doesn't support. You frequently use the term "self-describing", but this could have two meanings: 1) Like JSON, where the names of all fields appear in the message, so that a human can read the message easily without external information. 2) Limited self-description in which field values can be identified and parsed, but their names are not available (perhaps replaced by numeric tags or indexes). Protobuf can support (1) by including the schema in the payload. I agree this is not commonly useful. Protobuf supports (2) natively, by virtue of being a TLV format (just like ION). Re-reading the page, it sounds like you are assuming the Protobuf format cannot be deciphered at all without the schema, but this simply isn't true. If you meant something else, please explain. > The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. I wrote that documentation. It doesn't mean what you think (my fault, perhaps). What it's saying is that the top-level message is a series of tag-value pairs with no explicit indication of where that series ends (on the assumption that you already know, e.g. based on EOF). Thus, if you concatenate two whole messages without adding any delimiter then it will look like one big message containing all the fields from both. However, each field within the message is clearly delimited and sub-messages are length-delimited therefore skipable. > We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level. ION is a TLV encoding like Protobuf. Cap'n Proto is fixed offsets + pointers. These are vastly different styles of encoding that enable different modes of use. You can certainly debate which is better but I don't think it's correct to describe the formats as "pretty close", unless you consider all binary formats to be "pretty close" to each other.

jjenkov 11y ago
You are right, the term "self describing" as used in our docs could be more clear. Being self describing means that you do not need a schema to make sense of a stream of data of that format. However, there is also a degree to which a data format can be self describing. A CSV file is reasonably self describing because you can see where one field ends and the next begins (at the comma / separator), and where one record ends and the next begins (new line). With a header line of column names a CSV file becomes more self describing, as you now also have a name indicating the semantic meaning of fields in that column. If a CSV file could somehow contain a specification of the data type of each column, it would be even more self describing etc. This is what we are trying to achieve with ION. If you need speed, you can omit most of the meta data like property names etc. If you need messages to be self describing, you can add a lot of meta data (like class / schema names + version, property names etc.). I apologize for having written incorrect documentation. If you wrote those docs for Google Protocol Buffers, part of that is on you. They are not exactly crystal clear ;-) (our doc's aren't either - still working on them!) Thank you for clearing up that Protobuf fields can be distinguished in a stream of Protobuf fields, even without schema. That was unclear to me before now. By the way, that is pretty clear in Cap'n Proto - your invention right? So - better docs already! And - thank you for clearing up the difference in the encoding of Cap'n Proto. Any link to where I can read about that encoding style in more details?
- kentonv 11y ago
  > Any link to where I can read about that encoding style in more details? Hmm, I'm not aware of any literature other than what's on the Cap'n Proto web site. You can of course find the Cap'n Proto encoding documented here: https://capnproto.org/encoding.html https://capnproto.org/encoding.html The format is, of course, a lot like how in-memory data structures are laid out in C (fields of a struct have fixed offsets; variable-size fields are behind pointers). Unlike native pointers, though, Cap'n Proto's pointers are designed to be relocatable and easy to bounds-check, and they contain just enough type information for the message to be minimally self-describing (so that you can e.g. make a copy of a particular sub-object without knowing its schema).