robert3005

8 ms·

1.
by robert3005 8mo ago
Not to be confused with Velox a compute engine https://github.com/facebookincubator/velox/
2.
by robert3005 10mo ago
What you can do if you have gpu friendly format is you send compressed data over PCI-E and then decompress on the gpu. Thus your overall throughput will increase since PCI-E bandwidth is the limiting factor of the overall system.
3.
by robert3005 10mo ago
Can’t wait for https://github.com/apache/iceberg/issues/12225 to merge so there’s an api to integrate against
4.
by robert3005 10mo ago
The default writer will decompress the values, however, right now you can implement your own write strategy that will avoid doing it. We plan on adding that as an option since it’s quite common.
5.
by robert3005 1y ago
Highly recommend https://www.vldb.org/pvldb/vol16/p2132-afroozeh.pdf for a comparable algorithm. It generalizes to arbitrary input and output bit widths.
6.
by robert3005 1y ago
You might be interested in https://github.com/spiraldb/fastlanes . It gives you fast bit packed vectors. It needs padding to 1024 elements though for the performance.
7.
by robert3005 1y ago
You don’t have to wait for the doors to close to be able to scan your ticket in London Underground. The gate will stay open and let you through. It’s a little bit awkward since you have to approach as you scan your ticket leading to your ha
8.
Data Layouts: Where Bytes Find Their Forever Home(blog.spiraldb.com)
5 points by robert3005 2y ago|0 comments
9.
by robert3005 2y ago
The thing we are trying to achieve is to be able to experiment and tune the way data is groupped on disk. Parquet has one way of laying data out, csv is another (though it's a text format so a bit moot), ORC is another, Lance has yet a
10.
by robert3005 2y ago
Fulcrum | Software Engineer | London or New York | ONSITE | Full-Time Fulcrum is building next generation storage platform for diverse data of the future. We believe users will need to process non tabular and tabular data together and we ne