6 ms·

I regularly deal with JSON documents several MB in size, but do developers frequently deal with JSON documents several GB in size? If so, where do you encounter

I regularly deal with JSON documents several MB in size, but do developers frequently deal with JSON documents several GB in size? If so, where do you encounter something like that? Surely "processing" that much data (for whatever definition of process you have) is orders of magnitude slower than parsing it. I love the idea of a library trying to squeeze every last bit of performance out of the CPU, but I'm genuinely curious at the problems it solves in the real world.

Groxx 5y ago
Depends. I've had multi-gigabyte `[{..}, {..}, ...]` json arrays from database dumps, and doing even basic things with that with jq takes ages unless you use the (highly obtuse IMO) streaming methods. Sometimes you can pre-grep to filter the results to something trivial to process, but sometimes the structure is not unique enough to let you do that, or it depends on multiple field values - filtering that with a json parser makes perfect sense, and then speed can matter. That said, a 2x+ improvement for a couple megabytes, especially if done many times per second, is still a significant improvement.
nerdponx 5y ago
Huge log files are one use case. Large datasets are another, in data analysis, machine learning, and ETL tasks.
- mayama 5y ago
  Aren't log files processed a line at a time? Last time I had to deal with some structured log, I streamed lines concurrently into json parser and it went pretty fast.
jeffbee 5y ago
GeoJSON data can be just as big as you’d like. I have a big archive of GIS data where the files are enormous.
magicalhippo 5y ago
Not JSON, but we process XMLs on the order of a GB. Largest ones are consolidated invoices (ie a lot of separate invoices in one file). Other large ones contain rules and codelists in multiple languages.
jonstewart 5y ago
All the time. It's not there's a single record that is that size, but all sorts of things log in json, so you wind up with multi-GB jsonl files. As an example: AWS CloudTrail logs.
thechao 5y ago
Hardware config file for a small SOC. It faithfully describes every addressable physical byte.
hobs 5y ago
Monthly general ledger entries for the largest real estate companies, tried XML and JSON, eventually landed on compressed CSV for best trade off between human readable large files (~1-3GB) and compressibility.

Groxx 5y ago

Depends. I've had multi-gigabyte `[{..}, {..}, ...]` json arrays from database dumps, and doing even basic things with that with jq takes ages unless you use the (highly obtuse IMO) streaming methods. Sometimes you can pre-grep to filter the results to something trivial to process, but sometimes the structure is not unique enough to let you do that, or it depends on multiple field values - filtering that with a json parser makes perfect sense, and then speed can matter. That said, a 2x+ improvement for a couple megabytes, especially if done many times per second, is still a significant improvement.

nerdponx 5y ago

Huge log files are one use case. Large datasets are another, in data analysis, machine learning, and ETL tasks.

mayama 5y ago
Aren't log files processed a line at a time? Last time I had to deal with some structured log, I streamed lines concurrently into json parser and it went pretty fast.

jeffbee 5y ago

GeoJSON data can be just as big as you’d like. I have a big archive of GIS data where the files are enormous.

magicalhippo 5y ago

Not JSON, but we process XMLs on the order of a GB. Largest ones are consolidated invoices (ie a lot of separate invoices in one file). Other large ones contain rules and codelists in multiple languages.

jonstewart 5y ago

All the time. It's not there's a single record that is that size, but all sorts of things log in json, so you wind up with multi-GB jsonl files. As an example: AWS CloudTrail logs.

thechao 5y ago

Hardware config file for a small SOC. It faithfully describes every addressable physical byte.

hobs 5y ago

Monthly general ledger entries for the largest real estate companies, tried XML and JSON, eventually landed on compressed CSV for best trade off between human readable large files (~1-3GB) and compressibility.