If you're already using parquet, it might be worth looking at the concept of datasets e.g. https://arrow.apache.org/docs/python/generated/pyarrow.parquet.Parque

If you're already using parquet, it might be worth looking at the concept of datasets e.g. https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetDataset.html https://arrow.apache.org/docs/python/generated/pyarrow.parqu...