4 ms·
As I understand, arrow is particularly interesting since it’s wire format can be immediately queried/operated on without deserialization. Would saving an Arrow
by gbrits 6y ago
As I understand, arrow is particularly interesting since it’s wire format can be immediately queried/operated on without deserialization. Would saving an Arrow-structure as parquet not defeat that purpose, since your would need the costly deserialization step again on read? Honest question
- BadInformatics 6y agoThe FAQ [1] and this SO answer [2] explain it better than I can, but basically yes. However, the (de)serialization overhead is probably better than most alternative formats you could save to. [1] https://arrow.apache.org/faq/ https://arrow.apache.org/faq/ [2] https://stackoverflow.com/questions/56472727/difference-between-apache-parquet-and-arrow/56481636#56481636 https://stackoverflow.com/questions/56472727/difference-betw...