6 ms·

Your issue is that you're using the default (old) binding to GDAL, based on Fiona [0]. You need to use pyogrio [1], its vectorized counterpart, instead. Make s

Your issue is that you're using the default (old) binding to GDAL, based on Fiona [0]. You need to use pyogrio [1], its vectorized counterpart, instead. Make sure you use `engine="pyogrio"` when calling `to_file` [2]. Fiona does a loop in Python, while pyogrio is exclusively compiled. So pyogrio is usually about 10-15x faster than fiona. Soon, in pyogrio version 0.8, it will be another ~2-4x faster than pyogrio is now [3]. [0]: https://github.com/Toblerity/Fiona https://github.com/Toblerity/Fiona [1]: https://github.com/geopandas/pyogrio https://github.com/geopandas/pyogrio [2]: https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.to_file.html https://geopandas.org/en/stable/docs/reference/api/geopandas... [3]: https://github.com/geopandas/pyogrio/pull/346 https://github.com/geopandas/pyogrio/pull/346

culebron21 2y ago
Still CSV is 2x smaller than GPKG with this kind of data. And CSV.gz is 7x smaller.
- kylebarron 2y ago
  That's why I'm working on the GeoParquet spec [0]! It gives you both compression-by-default and super fast reads and writes! So it's usually as small as gzipped CSV, if not smaller, while being faster to read and write than GeoPackage. Try using `GeoDataFrame.to_parquet` and `GeoPandas.read_parquet` [0]: https://github.com/opengeospatial/geoparquet https://github.com/opengeospatial/geoparquet
culebron21 2y ago
CSV is still faster than geo-formats with pyogrio. From what I saw, it writes most of the file quickly, then spends a lot of time, I think, building the index. > %%timeit -n 1 > d.to_csv('/tmp/test.csv') 10.8 s ± 1.05 s per loop (mean ± std. dev. of 7 runs, 1 loop each) > %%timeit -n 1 > d2.to_file('/tmp/test.gpkg', engine='pyogrio') 1min 15s ± 5.96 s per loop (mean ± std. dev. of 7 runs, 1 loop each) > %%timeit -n 1 > d.to_csv('/tmp/test.csv.gz') 35.3 s ± 1.37 s per loop (mean ± std. dev. of 7 runs, 1 loop each) > %%timeit -n 1 > d2.to_file('/tmp/test.fgb', driver='FlatGeobuf', engine='pyogrio') 19.9 s ± 512 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) > ls -lah /tmp/test* -rw-rw-r-- 1 culebron culebron 228M мар 27 11:02 /tmp/test.csv -rw-rw-r-- 1 culebron culebron 63M мар 27 11:27 /tmp/test.csv.gz -rw-rw-r-- 1 culebron culebron 545M мар 27 11:52 /tmp/test.fgb -rw-r--r-- 1 culebron culebron 423M мар 27 11:14 /tmp/test.gpkg
culebron21 2y ago
...but this has spared me today some irritation at work. Thanks!

culebron21 2y ago

Still CSV is 2x smaller than GPKG with this kind of data. And CSV.gz is 7x smaller.

kylebarron 2y ago
That's why I'm working on the GeoParquet spec [0]! It gives you both compression-by-default and super fast reads and writes! So it's usually as small as gzipped CSV, if not smaller, while being faster to read and write than GeoPackage. Try using `GeoDataFrame.to_parquet` and `GeoPandas.read_parquet` [0]: https://github.com/opengeospatial/geoparquet https://github.com/opengeospatial/geoparquet

culebron21 2y ago

CSV is still faster than geo-formats with pyogrio. From what I saw, it writes most of the file quickly, then spends a lot of time, I think, building the index. > %%timeit -n 1 > d.to_csv('/tmp/test.csv') 10.8 s ± 1.05 s per loop (mean ± std. dev. of 7 runs, 1 loop each) > %%timeit -n 1 > d2.to_file('/tmp/test.gpkg', engine='pyogrio') 1min 15s ± 5.96 s per loop (mean ± std. dev. of 7 runs, 1 loop each) > %%timeit -n 1 > d.to_csv('/tmp/test.csv.gz') 35.3 s ± 1.37 s per loop (mean ± std. dev. of 7 runs, 1 loop each) > %%timeit -n 1 > d2.to_file('/tmp/test.fgb', driver='FlatGeobuf', engine='pyogrio') 19.9 s ± 512 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) > ls -lah /tmp/test* -rw-rw-r-- 1 culebron culebron 228M мар 27 11:02 /tmp/test.csv -rw-rw-r-- 1 culebron culebron 63M мар 27 11:27 /tmp/test.csv.gz -rw-rw-r-- 1 culebron culebron 545M мар 27 11:52 /tmp/test.fgb -rw-r--r-- 1 culebron culebron 423M мар 27 11:14 /tmp/test.gpkg

culebron21 2y ago

...but this has spared me today some irritation at work. Thanks!