mwlon

6 ms·

1.
Dtype_dispatch: a macro that defines macros that define enums and match them(graphallthethings.com)
17 points by mwlon 2y ago|3 comments
2.
Zero-Copy Reads in Rust(graphallthethings.com)
3 points by mwlon 2y ago|0 comments
3.
by mwlon 3y ago
Numerical data is full of rich patterns, but the general-purpose compressors we've historically used on them (e.g. snappy, gzip, zstd) are designed for unstructured, string-like data. Pcodec (or pco) is a new approach for numerical seq
4.
The Parquet We Could Have(graphallthethings.com)
2 points by mwlon 3y ago|0 comments
5.
Reducing Parquet 71% with Quantile Compression(graphallthethings.com)
2 points by mwlon 4y ago|0 comments
6.
by mwlon 5y ago
PancakeDB is an event ingestion solution, an important part of most tech companies' data stacks. Write to it one event at a time, and process billions at a time with the Spark connector or other tools. I've released it under BSL s
7.
PancakeDB Is Now Free(pancakedb.com)
2 points by mwlon 5y ago|1 comments
8.
by mwlon 5y ago
It is a new startup I'm building. It's a new type of database that can take in streaming data with very fast (~10ms) response times and output batch data with very fast throughput. To do that, it uses a new columnar file format an
9.
PancakeDB offers columnar reads 30% faster than Parquet(pancakedb.com)
20 points by mwlon 5y ago|3 comments
10.
New, better compression for columns of numerical data(github.com)
4 points by mwlon 5y ago|1 comments
11.
by mwlon 5y ago
I made this open source compression algorithm as part of a database I'm creating. It typically compresses columns of numerical data to ~25% smaller than alternatives (think .snappy.parquet or .gzip.parquet) at similar or cheaper comput