ianmcook

7 ms·

1.
Stop Paying the JSON Tax(columnar.tech)
2 points by ianmcook 5mo ago|0 comments
2.
Show HN: ADBC for COBOL – modern database access meets 1959(columnar.tech)
6 points by ianmcook 6mo ago|0 comments
3.
by ianmcook 2y ago
@1egg0myegg0 that's great to hear. I'll check to see if it applies to Arrow. Another performance issue with DuckDB/Arrow integration that we've been working to solve is that Arrow lacked a canonical way to pass statistic
4.
by ianmcook 2y ago
Arrow developer here, we've invested a lot in seamless DuckDB interop, great to see it getting traction. Recent blog post here that breaks down why the Arrow format (which underlies Arrow Flight) is so fast in applications like this:
5.
by ianmcook 3y ago
Anyone know what format they are serializing the data in to move it between Excel and Python? Are they using Apache Arrow?
6.
by ianmcook 6y ago
Thanks for the heads up. The post is intended to be up but there's an intermittent error happening. It's been reported to the Apache infrastructure team.
7.
by ianmcook 6y ago
Parquet is not based on Arrow. The Parquet libraries are built into Arrow, but the two projects are separate and Arrow is not a dependency of Parquet.
8.
by ianmcook 6y ago
From https://arrow.apache.org/faq/ : "Parquet files cannot be directly operated on but must be decoded in large chunks... Arrow is an in-memory format meant for direct and efficient use for computational purposes.
9.
by ianmcook 6y ago
The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a Feather file so you can operate on it without reading the who
10.
by ianmcook 6y ago
Re this second point: Arrow opens up a great deal of language and framework flexibility for data engineering-type tasks. Pre-Arrow, common kinds of data warehouse ETL tasks like writing Parquet files with explicit control over column types,