6 ms·

I’m super exited about the planned support for multidimensional data, I know zarr is there but a long term storage format other than netcdf would be interesting

I’m super exited about the planned support for multidimensional data, I know zarr is there but a long term storage format other than netcdf would be interesting maybe also something to replace grib, geotiff etc to share files

Upitor 2y ago
Where do you see the upcoming support for multidim data? Link?
- bitschubser_ 2y ago
  > This is just the beginning of modernizing geospatial data storage. We’re already looking ahead to other types of geospatial data such as raster, point cloud, spatial indexes… its not far from raster to full multidimensional
tomnicholas1 2y ago
Surely Zarr is already a long-term storage format for multidimensional data? It can even be mapped directly to netCDF, GRIB and geoTIFF via VirtualiZarr[0]. Also if you like Iceberg and you like arrays you will really like Icechunk[1], which is Version-controlled Zarr! [0] https://github.com/zarr-developers/VirtualiZarr https://github.com/zarr-developers/VirtualiZarr [1] https://icechunk.io/en/latest/ https://icechunk.io/en/latest/
- bitschubser_ 2y ago
  I know icechunk and I’m a huge fan of earthmover. But a common binary format like parquet seems nice… with interop for e.g duckdb and geo queries, you can “just load” era5 and do something like get wind direction/speed along the following path for the last 5 years group by day etc…
- lysecret 2y ago
  If you know the exact tensor shape of your data ahead of time Zarr works well (we use it as the dataformat for our ml experiments). If you have dynamically growing data or irregular shapes zarr doesn't work as well.
  - tomnicholas1 2y ago
    Icechunk can handle growing dimensions with ACID transactions! For irregular shapes in some cases using multiple groups + xarray.DataTree can help you, but in general yeah ragged data is hard.
deleted 2y ago
[deleted]
Mosarwat 2y ago
Beside Zarr, there are also efforts to support different types of raster (sort of multidimensional) data such as geotiff and NetCDF. The Iceberg Geo spec was heavily influenced by the Havasu project proposed by Wherobots, which also supports that type of raster data. However, the Iceberg geo spec still only supports only geometry for now. https://wherobots.com/building-a-spatial-data-lakehouse/ https://wherobots.com/building-a-spatial-data-lakehouse/

Upitor 2y ago

Where do you see the upcoming support for multidim data? Link?

bitschubser_ 2y ago
> This is just the beginning of modernizing geospatial data storage. We’re already looking ahead to other types of geospatial data such as raster, point cloud, spatial indexes… its not far from raster to full multidimensional

tomnicholas1 2y ago

Surely Zarr is already a long-term storage format for multidimensional data? It can even be mapped directly to netCDF, GRIB and geoTIFF via VirtualiZarr[0]. Also if you like Iceberg and you like arrays you will really like Icechunk[1], which is Version-controlled Zarr! [0] https://github.com/zarr-developers/VirtualiZarr https://github.com/zarr-developers/VirtualiZarr [1] https://icechunk.io/en/latest/ https://icechunk.io/en/latest/

bitschubser_ 2y ago
I know icechunk and I’m a huge fan of earthmover. But a common binary format like parquet seems nice… with interop for e.g duckdb and geo queries, you can “just load” era5 and do something like get wind direction/speed along the following path for the last 5 years group by day etc…
lysecret 2y ago
If you know the exact tensor shape of your data ahead of time Zarr works well (we use it as the dataformat for our ml experiments). If you have dynamically growing data or irregular shapes zarr doesn't work as well.
- tomnicholas1 2y ago
  Icechunk can handle growing dimensions with ACID transactions! For irregular shapes in some cases using multiple groups + xarray.DataTree can help you, but in general yeah ragged data is hard.

deleted 2y ago

[deleted]

Mosarwat 2y ago

Beside Zarr, there are also efforts to support different types of raster (sort of multidimensional) data such as geotiff and NetCDF. The Iceberg Geo spec was heavily influenced by the Havasu project proposed by Wherobots, which also supports that type of raster data. However, the Iceberg geo spec still only supports only geometry for now. https://wherobots.com/building-a-spatial-data-lakehouse/ https://wherobots.com/building-a-spatial-data-lakehouse/