creationix

7 ms·

1.
by creationix 6mo ago
I see. We have different definitions of serialized. The way I typically see it used is to describe how something is transmitted and stored, not how it is read. Yes, by your definition, this is random and not serially read.
2.
by creationix 6mo ago
So you're saying that random access formats that are encoded to disk as a stream of bytes are not "serialized" because you don't alway read them in order? Yes, many formats are read start-to-end, but I don't think t
3.
by creationix 6mo ago
> Just copy and paste files If all your workflows allow copying as binary files, more power to you! But there are a lot of workflows where that is not possible. This was inspired by years of hands-on operational incident handling in pr
4.
by creationix 6mo ago
> what sort of storage device does not allow your computers to use all 256 byte values - clipboards - logs - terminal output - alerts - yaml configs - JSON configs - hacker news comments - markdown documentation - etc... I assure you, th
5.
by creationix 6mo ago
And don't worry about railroad diagrams. I already intended to create them, I've just been extra busy this week with other things.
6.
by creationix 6mo ago
Tha main reason for the reverse encoding is it makes it easier on the writer. You simply do a depth-first traversal of the data graph and emit data on the way back up the stack. Zero buffering is needed since this naturally means you writ
7.
by creationix 6mo ago
Also good luck parsing 10 MiB of JSON in a loop that can't tolerate blocking the CPU for more than 10ms. What's expensive is very relative to the use case.
8.
by creationix 6mo ago
Very true. I had forgotten about bencode, I should read up on that again. It makes sense they need a canonical form because they want same values to have same content hashes.
9.
by creationix 6mo ago
I meant computers can read it without any preprocessing. It's random access. You don't need to parse it, you don't need to decompress it. You just start at the end and follow pointers till you get to the desired value. Eve
10.
by creationix 6mo ago
yeah, LuaJIT is one of the use cases I had in mind working on this. JSON is pretty fast in modern JS engines, but in Lua land, JSON kinda sucks and doesn't really match the language without using virtual tables. JSON has `null` values
11.
by creationix 6mo ago
> it only has a text encoding as long as you can guarantee you don't have any unicode? The format is technically a binary format in that length prefixes are counts of bytes. But in practice it is a textual format since you can almo
12.
by creationix 6mo ago
How does CBOR retain JSON compatibility more than RX? RX can represent any value JSON can represent. It doesn't even lose key order like some random-access formats do. In fact, RX is closer to JSON than CBOR. Take decimals as an examp
13.
by creationix 6mo ago
I've rewritten the framing in the README to first explain when you should use RX and when you should not. Most uses of JSON should probably stay JSON. Let me know what you think https://github.com/creationix/rx&#x
14.
by creationix 6mo ago
Initial format docs are now here: https://github.com/creationix/rx/blob/main/docs/rx-format.md Railroad diagrams will come later when I have more time.
15.
by creationix 6mo ago
Thanks for the feedback. I've improved the framing to make the purpose/value more clear. What do you think about "RX is a read-only embedded store for JSON-shaped data"? https://www.npmjs.com/package&#x
16.
by creationix 6mo ago
That benchmark is a fair comparison for a real-world production workload and use case. Sadly I can't share the details. But suffice it to say that the dataset is a huge object with tens of thousands of paths as keys and moderately la
17.
by creationix 6mo ago
the project framing needs some help perhaps. JSON is really good at a lot of use cases that this will never replace. But there are cases where JSON is currently used where this is much better. In particular large unstructured datasets wh
18.
by creationix 6mo ago
I'm happy to hear suggestions. This format was actually the internal .rexc bytecode for Rex (routing expressions), but when I realized it was actually a pretty good standalone format, I renamed it `.rx` for short. I am aware of RxJS
19.
by creationix 6mo ago
You're right. Some important differences: sick is binary, rx is textual (this matters for tooling) sick has size limits (65534 max keys for example. I have real-world rx datasets reaching this size already) rx uses arbitrary precisio
20.
by creationix 6mo ago
> Does this duplicate the name of keys? Yes, the format allows for objects to be stored with a pointer to a shared schema (either an array of keys or another object that has the desired keys) The current implementation is pretty close to
21.
by creationix 6mo ago
The current format version is the exact same feature set as JSON. I even encode numbers as arbitrary precision decimals (which JSON also does). This is quite different from CBOR which stores floats in binary as powers of 2. I could techni
22.
by creationix 6mo ago
it's not really possible to stay human readable and get the compression levels and random access properties I was going for. But it is as human tooling friendly as possible given the constraints.
23.
by creationix 6mo ago
yes, this would work very well for any case where you have embedded databases of unstructured data that you want to query in a website or edge server
24.
by creationix 6mo ago
yep. I built custom JSON parsers as a first solution. The problem is you can't get away from scanning at least half the document bytes on average. With RX and other truly random-access formats you could even optimize to the point of
25.
by creationix 6mo ago
I did seriously consider SQLite, but my existing datasets don't map easily to relational database tables. This is essentially no-sql for sqlite.
26.
by creationix 6mo ago
right, the jq query language is much more complex and featureful than the simple selector syntax I added to the rx-cli. But more could be added later as needed or it could just stream JSON output. It would be pretty trivial to hook up a s
27.
by creationix 6mo ago
> What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the
28.
by creationix 6mo ago
> 2x the size of json after compression Right and that makes sense. There is more information in here. The entire thing is length prefixed and even indexed for O(1) array lookups and O(log2 N) object lookups. If you don't care abo
29.
by creationix 6mo ago
It is also a format that can be read as-is without any preprocessing. In some cases base64 can do that, and this format does make heavy use of base64 varints. Sure, you can encode as JSON, then compress with gzip and then base64 encode. Y
30.
by creationix 6mo ago
yep, this is exactly the kind of use case that caused me to design this format.

More ›