tomnicholas1

6 ms·

1.
by tomnicholas1 16d ago
> The problem in climatology is that when the models don't fit the data they just change the data and claim victory That is... not remotely true, nor supported by either of the links you shared. (Nor by any cursory look at the state
2.
by tomnicholas1 16d ago
Whilst I weakly agree with this characterisation, it's not at all fair or accurate to put epidemiology and climatology in the same category as your other examples. Yes they have some weaknesses in replication practices (especially for
3.
by tomnicholas1 1mo ago
> Data from physics, chemistry, biology experiments There are PetaBytes of important scientific data locked in archival file formats. The first step is to make this efficiently readable. https://www.earthmover.io/blog&#x2
4.
by tomnicholas1 2mo ago
Yes! But if you want to share it with anyone else that would be great, since we're advocating for fairly radical changes within a big bureaucracy here, as I'm sure you will appreciate :)
5.
by tomnicholas1 2mo ago
Anyone interested in accessing GOES data at scale will find this interesting - I created a Zarr index over the 7 billion chunks of data in the GOES-16 archive. https://www.earthmover.io/blog/virtual-zarr
6.
Cloud-optimizing the GOES-16 satellite data archive without copying data(earthmover.io)
10 points by tomnicholas1 3mo ago|1 comments
7.
by tomnicholas1 3mo ago
The last 2 years of my professional life have in some sense all been working towards this blog post. It shows how you can make an enormous archive of public scientific data[1] much easier to use for everyone, without having to copy all the
8.
by tomnicholas1 6mo ago
Awesome to see this project here - it was partly inspired by my blog post (original is linked from the OP, but there's a slightly newer version on my personal site here[0]). [0]: https://tom-nicholas.com/blog/2025&
9.
Periodic Labs(periodic.com)
2 points by tomnicholas1 6mo ago|0 comments
10.
by tomnicholas1 7mo ago
What you describe is very similar to how Icechunk[1] works. It works beautifully for transactional writes to "repos" containing PBs of scientific array data in object storage. [1]: https://icechunk.io/en/lates
11.
by tomnicholas1 8mo ago
People have literally used Zarr for this - at one point Gemini used Zarr for checkpointing model weights. Not sure what the current fashion in that space is though. It's definitely one of many fields that see convergent evolution towar
12.
by tomnicholas1 8mo ago
IMO Zarr is that newer format. It abstracts over the features of all these other formats so neatly that it can literally subsume them. I feel that we no longer really need TIFF etc. - for scientific use cases in the cloud Zarr is all that&#
13.
by tomnicholas1 8mo ago
The generalized form of this range-request-based streaming approach looks something like my project VirtualiZarr [0]. Many of these scientific file formats (HDF5, netCDF, TIFF/COG, FITS, GRIB, JPEG and more) are essentially just contig
14.
by tomnicholas1 9mo ago
God this article is 10000% better than the posted one. This is great: > Names should not describe what you currently think the thing you’re naming is for. Imagine naming your newborn child "Doctor", or "SupportsMeInMyOldAg
15.
by tomnicholas1 1y ago
Thank you for the explanation! But what a mess. I would love to bring these benefits to the multidimensional array world, via integration with the Zarr/Icechunk formats somehow (which I work on). But this fragmentation of formats makes
16.
by tomnicholas1 1y ago
The pitch for this sounds very similar to the pitch for Vortex (i.e. obviating the need to create a new format every time a shift occurs in data processing and computing by providing a data organization structure and a general-purpose API t
17.
by tomnicholas1 1y ago
The really depressing part is if you plot rate of new delays against real time elapsed, the projected finishing date is even further. This is why much of the fusion research community feel disillusioned with ITER, and so are more interested
18.
by tomnicholas1 1y ago
Presumably because everyone in MCF has been waiting for ITER for decades, and JET is being decommissioned after a last gasp. Every other tokamak is considerably smaller (or similar size like DIII-D or JT-60SA). Much of the interesting tokam
19.
What Is Cloud-Optimized Scientific Data?(earthmover.io)
3 points by tomnicholas1 1y ago|1 comments
20.
by tomnicholas1 1y ago
I wrote the article I wish I could have read back when I first heard of Zarr and cloud-native science back in 2018. This explains how object storage and conventional filesystems are different, and the key properties that make Zarr work so w
21.
by tomnicholas1 1y ago
Isn't that more about enumerating the microstates? The Pauli exclusion principle just ends up forbidding some of the microstates (forbidding a significant fraction of them if you're in the low-temperature regime).
22.
by tomnicholas1 1y ago
Yes, that assumption is called the Ergodic Hypothesis, and generally justified in undergraduate statistical mechanics courses by proving and appealing to Liouville's theorem. [1] https://en.wikipedia.org/wiki/Ergod
23.
Tensors vs. Tables: Why tabular tools trip over gridded data(earthmover.io)
3 points by tomnicholas1 1y ago|1 comments
24.
by tomnicholas1 1y ago
The scientific community works primarily with array (or "tensor") data, using tools like numpy, xarray, and zarr. People familiar with modern relational database tools such as DuckDB and Parquet often ask why can't we just us
25.
by tomnicholas1 2y ago
I think the posted article was generated from this one - the structure of the content is so similar.
26.
by tomnicholas1 2y ago
This entire stack also now exists for arrays as well as for tabular data. It's still S3 for storage, but Zarr instead of parquet, Icechunk instead of Iceberg, and Xarray for queries in python.
27.
by tomnicholas1 2y ago
Icechunk can handle growing dimensions with ACID transactions! For irregular shapes in some cases using multiple groups + xarray.DataTree can help you, but in general yeah ragged data is hard.
28.
by tomnicholas1 2y ago
Surely Zarr is already a long-term storage format for multidimensional data? It can even be mapped directly to netCDF, GRIB and geoTIFF via VirtualiZarr[0]. Also if you like Iceberg and you like arrays you will really like Icechunk[1], whic
29.
by tomnicholas1 2y ago
> The future of Python's main open source data science ecosystem, numfocus, does not seem bright. Despite performance improvements, Python will always be a glue language. Your first sentence is a scorching hot take, but I don't
30.
by tomnicholas1 2y ago
I’ve been thinking a lot recently about how one thing science needs is a social network for sharing big data. One thing the post gets at is that providing a decentralized global subscribable data catalog is fundamentally a network protocol

More ›