PolarizedPoutin

6 ms·

1.
by PolarizedPoutin 8mo ago
https://aliramadhan.me/ I recently redesigned my personal website to mimic Earth’s layers and also started blogging. Thanks for putting this together, I love browsing through unique personal websites!
2.
Show HN: Matplotloom: Weave frames into Matplotlib animations simply and quickly(github.com)
2 points by PolarizedPoutin 2y ago|0 comments
3.
by PolarizedPoutin 2y ago
Thank you for reading and for the links! I'm trying out Clickhouse for the next post. Definitely excited for sub 24 hour data loading! I haven't heard of VictoriaMetrics but that's some impressive performance. Will check it o
4.
by PolarizedPoutin 2y ago
The data is publicly available! The data is freely available from the Climate Change Service [1] which has a nice API but download speeds can be a bit slow. You'll have to sign up for this. NCAR's Research Data Archive [2] provide
5.
by PolarizedPoutin 2y ago
What do you mean by the heat distribution of energy? Do you mean like how much heat is received from the sun at a particular location and particular time? If so, then temperature is only the result of this plus other factors. Factors like c
6.
by PolarizedPoutin 2y ago
GraphCast was trained on this exact same data! From https://deepmind.google/discover/blog/graphcast-ai-model-for... > Crucially, GraphCast and traditional approaches go hand-in-hand: we trained GraphCast on fou
7.
by PolarizedPoutin 2y ago
The full dataset is quite huge (~9 petabytes and growing) out of which I'm using just ~8 terabytes. The data is freely available from the Climate Change Service [1] which has a nice API but download speeds can be a bit slow. NCAR'
8.
by PolarizedPoutin 2y ago
The full dataset is quite huge (~9 petabytes and growing) out of which I'm using just ~8 terabytes. Still quite big to upload. The data is freely available from the Climate Change Service [1] which has a nice API but download speeds ca
9.
by PolarizedPoutin 2y ago
Yeah I was thinking about this and hoped that Postgres had a `float2` data type but `int2` would have to work. I could scale the numbers to fit them into 2 bytes with minimal loss of precision, but decided I'd rather take the storage s
10.
by PolarizedPoutin 2y ago
It's a good question! It's true that the output is massive, I believe ~9 petabytes and growing. But running the model is super expensive. It runs on ECMWF's supercomputer. Not sure how many cores but I would guesstimate in th
11.
by PolarizedPoutin 2y ago
I'm hoping to compare TimescaleDB and Clickhouse to see how big the difference is for different queries! My impression is that TimescaleDB gives you some columnar features, but maybe Clickhouse is a true columnar database.
12.
by PolarizedPoutin 2y ago
Had a read through parts 1 and 2, thank you for the engaging reads! Love how you've formatted your posts with the margin notes too. Thank you for providing the function to write numpy structured arrays to Postgres binary, I couldn&#x27
13.
by PolarizedPoutin 2y ago
Thank you for reading through and for your feedback! Excited to try your settings to disable the WAL and other overhead and see if I get even faster inserts. Also glad to hear an expert say that WAL data isn't really necessary for bulk
14.
by PolarizedPoutin 2y ago
Hoping to find out when I compare TimescaleDB vs. Clickhouse!
15.
by PolarizedPoutin 2y ago
Haha simplistic but probably faster and more space-efficient than a relational database. Sounds like rabernat and open-meteo who commented here do something similar to you and find it fast as well!
16.
by PolarizedPoutin 2y ago
Thank you for mentioning paleoclimatology! Not sure if what I'm doing with ERA5 data is that rigorous haha, but one of my favorite plots is https://commons.wikimedia.org/wiki/File:All_palaeotemps.svg (still have i
17.
by PolarizedPoutin 2y ago
The data is definitely sparse in the earlier decades, especially over unpopulated areas and in developing nations, due to the lack of regularly recorded weather observations (sensors, balloons, etc.). Especially since weather satellites sta
18.
by PolarizedPoutin 2y ago
Yes you are correct of course. ERA5 is climate model output constrained to match weather observations, not actual observations, which I do note in the post.
19.
by PolarizedPoutin 2y ago
Are you suggesting compressing the data using like Fourier series or wavelets? I know some of the variables have sharp gradients, especially variables like precipitation, which probably wouldn't compress super well. And some applicatio
20.
by PolarizedPoutin 2y ago
Thank you! I've relied on that USGS Projects manual multiple times haha. Working with satellite data some of it was in somewhat obscure projections and the manual always told you how to convert back to latitude-longitude.
21.
by PolarizedPoutin 2y ago
Thanks for the link on the trillion row challenge, interesting read! I'm looking at queries and indexes next and I'm hoping to include Clickhouse in that comparison.
22.
by PolarizedPoutin 2y ago
Thank you for the link! I've mostly used TablePlus (not free I think) and matplotlib (via psycopg3) to plot data returned by queries, but this looks like it'll be faster to use. I've only inserted data so far haha but will be
23.
by PolarizedPoutin 2y ago
Haha thank you for reading and glad you found it engaging! Maybe it's the benefit of being a beginner and not having any skin in the game. I did a lot of searching but couldn't find any conclusive answers for my use case so figure
24.
by PolarizedPoutin 2y ago
Thanks for pointing out the status of ERA5 on GCP and AWS. I'll have to try the binning and compare it with listing all the values and finding the 99th percentile! But yeah the spatial part of that query might be the tougher part. And
25.
by PolarizedPoutin 2y ago
Hey Ryan and thank you for the feedback! I agree that storing the data is appropriately chunked Zarr files is almost surely going to be faster, simpler to set up, and take up less space. Could even put up an API in front of it to get "
26.
by PolarizedPoutin 2y ago
The main reason why was that it's a personal project and I wanted to do everything on my home server so that I wouldn't have to pay for cloud resources, and so that I could learn Postgres, TimescaleDB, and eventuallly PostGIS. But
27.
by PolarizedPoutin 2y ago
Thank you for reading and for your kind words! Ah I did not know about the `create_default_indexes=>false` and that a time index is created by default for hypertables. I'll add a note to explain this! Also curious to benchmark inser
28.
by PolarizedPoutin 2y ago
Wish I saw this before I started haha! I left a footnote about why I didn't try binary copy (basically someone else found its performance disappointing) but it sounds like I should give it a try. footnote: https://aliramadha
29.
by PolarizedPoutin 2y ago
Thank you for reading through it thoroughly and pointing this out! I'm still new and learning Postgres so this is good to know. I will update the post.
30.
by PolarizedPoutin 2y ago
Thank you for the insights! Yeah I'm still not sure how Postgres/PostGIS will scale for me, but good to know that BigQuery does this nicely. This is not something I'm productionizing (at least not yet?) and I'm giving my

More ›