6 ms·
Show HN: ION, a JSON alternative – Versatile, compact, fast, binary data format
- leeoniya 11y agoi have found that most of my huge json data is from uniform recordsets. there's a great json-compatible encoder for such cases that stores them in a format that's CSV-esque: https://github.com/WebReflection/JSONH https://github.com/WebReflection/JSONH
- VStack 11y agoThat might work for a web app, but not for mobile apps, or backend to backend communication.
- leeoniya 11y agoanything that used json before would benefit from JSONH. things that abused json before, would be better off switching to a different format that is less restrictive than json, like ION or msgpack [1] [1] http://msgpack.org/index.html http://msgpack.org/index.html
- efaref 11y agoOr you could use an actual RFC standard[1] that has numerous high-quality implementations[2]. [1] https://tools.ietf.org/html/rfc7049 https://tools.ietf.org/html/rfc7049 [2] http://cbor.io/ http://cbor.io/
- robalfonso 11y agoIt wouldn't be fun if you didn't re-invent your own wheel but this time did it with the features you want instead of the ones that are almost-identical-but-you-didn't-invent!
- VStack 11y agoWe believe that while ION is similar in encoding to other formats, it is different in aspects that make ION a better all-round data and network message format. See our ION vs.Other Data Formats Page too. http://tutorials.jenkov.com/iap/ion-vs-other-formats.html http://tutorials.jenkov.com/iap/ion-vs-other-formats.html Not every innovation is from scratch, or 100% unique. MessagePack was designed as a binary encoding of JSON. CBOR too. ION was designed to be a binary general purpose data format able to mimic both JSON, XML, CSV and raw bytes (files), plus the most commonly used data types. Of course there will be similarities when we try to tackle the same problems, but we also believe we had added something new and useful to the mix.
- VStack 11y agoWe have already look at CBOR and MessagePack. They miss the ION tables for compact arrays of data. Here is a link to our comparison to other data formats. http://tutorials.jenkov.com/iap/ion-vs-other-formats.html http://tutorials.jenkov.com/iap/ion-vs-other-formats.html
- efaref 11y agoI don't really see how ION tables are an improvement over arrays of arrays, e.g.: { "headers": [ "a", "b", "c" ], "rows": [ [ 1, 2, 3 ], [ 4, 5, 6 ], ] } Furthermore, ION appears to require you to know the length of your data up front, whereas you could use CBOR unspecified length arrays to stream data from your database without precalculating the table length. It seems like quite a niche format, though. Most data is not truly tabular. Also, in the table it claims "Yes" under support for "Cyclic references", and yet further down the page: > ION has support for expressing cyclic references between objects. At this point this support is not 100% finalized. So surely this should be "Yes(*)"?
- VStack 11y agoFirst of all, lots of results sent back from backend services (or databases) are arrays of objects. So no, tables are not a "niche" format - tables are heavily used. Second, an array of arrays could mean anything. You have not semantics telling whether the arrays are independent or if the first array is an array of columns for the following arrays. ION Tables add that semantic information. Third, yes, we could encode everything as text or as raw bytes and leave it up to the user to make sense of it. But that is exactly what we are trying to avoid with ION. We want to give devs a decent standard data format to use, that doesn't require a lot of data encoding choices up front. The encoding options have been thought through already, and sensible choices already made which you can just follow. Fourth, you can nest ION tables inside ION tables, and thus create a more compact representation of an object graph. Using JSON / CBOR you would need a lot of nested arrays inside arrays to emulate that. Possible, but not exactly pretty.
- VStack 11y ago
- GordonS 11y agoObligatory xkcd: https://xkcd.com/927/ https://xkcd.com/927/
- kevinSuttle 11y agoIs there a spec? Is this a Java API for it? See also: https://github.com/edn-format/edn https://github.com/edn-format/edn
- VStack 11y agoYes please check: Specification: http://tutorials.jenkov.com/iap/index.html http://tutorials.jenkov.com/iap/index.html Benchmarks: http://tutorials.jenkov.com/iap/ion-performance-benchmarks.html http://tutorials.jenkov.com/iap/ion-performance-benchmarks.h... Tutorials: http://tutorials.jenkov.com/iap-tools-java/index.html http://tutorials.jenkov.com/iap-tools-java/index.html Here is also a recent Infoq.com article http://www.infoq.com/articles/IAP-Fast-HTTP-Alternative?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global http://www.infoq.com/articles/IAP-Fast-HTTP-Alternative?utm_....
- profeta 11y agothe "message structure" page is 404ing. all the rest hints to a designed-by-committee buzzword thingy.
- jjenkov 11y agoHere is the Message Structure doc http://tutorials.jenkov.com/iap/iap-message-structure.html http://tutorials.jenkov.com/iap/iap-message-structure.html . We are not as far with the core IAP protocol as we are with ION, but we will get there. Designed-by-committee - really? You pull that?
- fnordsensei 11y agoTo be fair, Transit is probably a better message format than edn. https://github.com/cognitect/transit-java https://github.com/cognitect/transit-java
- deleted 11y ago[deleted]
- umanwizard 11y agoNote for anyone who was as confused as I was: this is not the same thing as Amazon's internal typed JSON format, which is also called Ion.
- rix0r 11y agoWell, why would it be? That's internal. Hasn't been published as far as I can Google.
- umanwizard 11y agoI don't work at Amazon anymore so I thought maybe they had open-sourced it
- jjenkov 11y agoYes, a guy told us that Amazon has an internal data format called ION. We googled for it, but didn't find it, so we assumed Amazon wants to keep it internal.
- umanwizard 11y agoNot that you are under any obligation to care, but this will now be very annoying for Amazon employees. "Oh, you need to call that service using Ion, not Json. No, the other Ion..."
- pinkunicorn 11y agoNot to mention if they try to use it internally, then might end up with package name conflicts..
- pinkunicorn 11y agoComing to think of it, ION was actually pretty great. Chris Suver's brain child!
- dmitrygr 11y agoHTTP ERROR 404 Problem accessing /iap/message-structure.html. Reason: Page not found
- k__ 11y agoI consider no need for a schema a plus of MessagePack. But the Nos are all red, haha.
- woah 11y agoOn one of these pages, the claim is made that Protobufs is not self-describing, and therefore cannot be used for "network applications". It seems that "self-describing" here means that the format includes key names, instead of compressing them by using numbers like protobufs does. I can't understand why having field names is going to make a difference for anyone. Once you are setting up a system to deal with a specific format of data, why not just include a protobufs schema?
- umanwizard 11y agoIt's nice to not have to assume the client and server are running the exact same version of the protocol. If you use ordinal numbers instead of names, you can never remove or reorder things without completely breaking backward compatibility. You can only append new fields to the end.
- ambrice 11y agoThe tag numbers are part of the message definitions, they're not assigned arbitrarily. So absolutely you can reorder them. And as long as you described them as "optional" in the original message definition you can remove them without breaking backwards compatibility.
- ctz 11y agoHere's a use case where protobuf is terrible because it isn't self-describing: write a wireshark plugin which parses and pretty-prints protobuf messages for human consumption. You can't, because such a plugin would have to have a-priori knowledge of the schema in use.
- VikingCoder 11y agoYou mean something like this, which does exactly that? https://code.google.com/archive/p/protobuf-wireshark/ https://code.google.com/archive/p/protobuf-wireshark/ Sure, you don't know the NAME of the field, but you can see the ID of it.
- VStack 11y ago
- dplgk 11y agoBinary format makes me believe it's not human-readable. How doesn't this compare in size to gzipped JSON? JSON overhead is fairly small (some quotes, colons, brackets and keys) - it's no XML.
- Cyph0n 11y agoI think it can make quite a difference in long messages, as well as when serving a high number of requests.
- VStack 11y agoYes, binary formats are not easily readable in a text editor. But, it is actually possible to convert ION to an XML format and back again without loss of information (we have not implemented this yet). This should make it easier to read messages during debug - especially because you don't need to know the schema for the given message to conver it to XML. Regarding GZipped JSON, it is true that GZiped JSON is small. But, due to the CRIME and BREACH attacks it is not recommended to compress data sent over encrypted connections (TLS). If you look at our performance benchmarks page you can see a list of serialized length comparisons. As you can see, as soon as you send a few objects in an ION table, the difference is big. More than what you normally can gain with GZip (except perhaps for String).http://tutorials.jenkov.com/iap/ion-performance-benchmarks.html#serialized-length http://tutorials.jenkov.com/iap/ion-performance-benchmarks.h... Furthermore, GZip only helps with transfer time, and actually slows down parsing time. If you look at our performance benchmarks you will see that ION parsing time is a lot faster than JSON. Additionally, if you really, really want high speed you do not parse ION (or JSON) into Java objects. You process the data directly in its binary form. If you look at our read-and-use benchmark you can see just how big a speed difference that gives. ION is designed for being processed directly. JSON isn't as good for that purpose. Finally, ION is designed for fast arbitrary hierarchical navigation. JSON is not.
- efaref 11y agoOne thing that I like about CBOR is that with very little knowledge it's surprisingly readable in a hex-dump. Low value positive integers are the value of the byte itself, strings all have the form "0x6L [string]" or "0x7X [len] [string]". Arrays and maps are similarly obvious. Of course, anything more than a simple construct you're better off using a decoder (e.g. a Wireshark one). Also, the fact that it's compatible with JSON means that you can use JSON in your development, and then switch to CBOR at the end for the reduction in packet size. In python it's as simple as changing: import json as encoder to import cbor as encoder
- blakecallens 11y agohttps://media.giphy.com/media/1M9fmo1WAFVK0/giphy.gif https://media.giphy.com/media/1M9fmo1WAFVK0/giphy.gif
- deleted 11y ago[deleted]
- klodolph 11y ago> As you can see, an ION field can contain values that are up to 2^120 bytes long. If you need to encode larger blocks of data than that, you would need to break it up into multiple fields. Har, har.
- shockzzz 11y agoIs there an ION vs. Thrift comparison?
- VStack 11y agoNot yet. We have been asked to compare ION to Flatbuffers, Cap'n Proto, Thrift, Avro, Transit, BSON and several other encodings. However, writing the benchmarks and going through the features systematically is a lot of work, so we have not yet had the time to go through them all.
- Matthias247 11y agoI think the most interesting difference to the usual serialization formats might be the copy and reference types. I'm a little bit undecided whether they might be a brilliant idea or not. The decision whether to support copy or not puts some extra effort in the serializer and deserializer, but the total result is the same that you can have as without a copy field mechanism. The support of cyclic references makes a big change, because you can't directly model them with technologies. You might also have trouble using these data structures in some programming languages or libraries (e.g. if you are only using immutable types or want to use only value types). However for some kind of data it seems to make sense to support cyclic data, as GraphQL and Falcor have also added support for that. I also don't see that many use cases for the table structure. I have deployed thousands of RPC APIs into production, and I can't recall having the need for it. And even if you need it, using an object with 2 arrays in it would be just fine. I also looked through the IAP documentation (btw. bad name => ipod accessory protocol) because it's quite related to what I'm working on. I think that the shown basic communication patterns are correct, but from the documentation I can't really get a feeling what I could expect from an IAP library. Would it be some low level messaging system (like MQTT, ZeroMQ, etc.) or would higher level communication patterns (request/response, notifications) also be built in. There are no predifined message formats for RPC listed in the documentation which would outline that. The WAMP specification (http://wamp.ws http://wamp.ws) e.g. makes it clearer what I could expect from such a protocol. I'm not sure whether we need a new low level messaging protocol or if the work should be more focused on adding higher level semantics on top if it. E.g. I think some pattern that I really need in my domain is remote object synchronization, which means the status of an object on the server gets automatically pushed towards all interested client and is continously updated during changes (=> e.g. to build something like Firebase). Of course one can built something like that on top of basic messages by defining subscribe and update messages in the API, but I'm wondering if it's worthwhile to add something like that directly in the protocol. On the one hand this is also a special case of the subscription pattern which is also listed here, on the other hand it can not directly be implemented with the subscription possibilites of many message broker systems, because they won't send you the current state of an object after subscription but will only forward you a message after the value changes for the next time. The connection and sequence definition in IAP looks a little bit redudant to me on the first look. I really think there is a need for message ordering and you must support it. The question for me is then if you don't need message ordering, why not put the message into a seperate channel and let channels/streams always be ordered (like in HTTP/2)? Overhead for channel creation? Or to setup channels during creation either as ordered or unordered and keep that for the lifetime of the channel?
- kentonv 11y agoI'm looking at: http://tutorials.jenkov.com/iap/ion-vs-other-formats.html http://tutorials.jenkov.com/iap/ion-vs-other-formats.html As the author of Protobuf v2 (the version that was open sourced by Google), I object to some of the "no"s in the protobuf column. (Note: I no longer work on Protobuf, and I did not invent the format. I do work on and did invent Cap'n Proto.) > Protobuf apparently isn't great at encoding raw bytes either (according to their own website). Protobuf can handle raw bytes just fine, using the "bytes" type. There is no special encoding done on bytes; parsing and encoding is done by memcpy(). I'm curious to know what part of the web site you interpret as saying otherwise. It's entirely possible that the web site contains confusing language, but a citation would have been a good idea here. > Schema / Class Id > Self describing The Protobuf libraries have extensive support for manipulating dynamic schemas and transmitting schemas over the wire. See the "Descriptor" and "DynamicMessage" APIs. This is mentioned on the web site: https://developers.google.com/protocol-buffers/docs/techniques#self-description https://developers.google.com/protocol-buffers/docs/techniqu... > Even if these compact objects do not contain any property names, they are still self describing enough that you can see where fields start and end, plus their data type, without an external schema. You cannot do that with Protobuf (as far as we know). You absolutely can do that with Protobuf. This is what the "protoc --decode_raw" flag does, and it should be clear enough from reading the encoding. https://developers.google.com/protocol-buffers/docs/encoding https://developers.google.com/protocol-buffers/docs/encoding > Cyclic references While it's true that Protobuf doesn't support these, I hope you've considered the denial-of-service vulnerabilities they tend to create if the receiver is not expecting them. Please ensure that cyclic references are only allowed in cases where the app opted into it. Relatedly, overlapping references / backreferences ("Copy" in your table) potentially leads to an amplification attack where a small message on the wire turns out to be much, much larger when traversed. If applications cannot defend themselves from huge payloads by setting a message size limit, then you'll need to give them some other way. > All of the formats (except perhaps Protobuf) supports arbitrary hierarchical navigation of the encoded data, without first converting it to objects. Protobuf supports this, and in fact should be an unqualified "Yes" rather than "Yes(*)" like the others. Protobuf encoding is very similar to ION's. Sub-messages are length-delimited, which seems to be exactly the advantage you're claiming that ION has. Note that none of these formats support random access in the way that Cap'n Proto does. In summary, I believe Protobuf deserves a "yes" in: "Raw bytes", "Good at raw bytes", "Schema / Class Id", "Arbitrary hierarchical navigation", and "Self describing".