eddyxu

44 ms·

1. 1.▲
   by eddyxu 3y ago

   No worries at all. For the teams which are happy on CSV/JSON, i'd admit that lance is not ideal alternative for them. > It seems you build a few indexes, such that I'm guessing those are the main drivers on the benefits? Y
2. 2.▲
   by eddyxu 3y ago

   That's a good idea. The format itself is self-contained. We need a little bit work to expose our file-level read/write API to public.
3. 3.▲
   by eddyxu 3y ago

   The use cases for such random access was from our experiences to maintain large-scale training data, with the needs to debugging model performance against to data, which requires fast slicing and dicing over the dataset, filtering into subs
4. 4.▲
   by eddyxu 3y ago

   We have not done benchmarks against deeplake yet. Deeplake has some interesting concepts in their design, I'd be very interested to do a benchmark soon.
5. 5.▲
   by eddyxu 3y ago

   Hey, co-author of Lance here. Lance is faster in random access because the layout / encodings were designed to be fast in both scan and random access case. We borrowed many ideas from Google's Procella paper, and Arrow's in-m