6 ms·

Historical weather data API for machine learning, free for non-commercial

196 points by meteo-jeff 4y ago

aarreedd 4y ago
Does not seem accurate. This is telling me it snowed 1.33cm on June 17, 2022 in New York City. https://api.open-meteo.com/v1/forecast?latitude=40.71&longitude=-74.01&daily=rain_sum,snowfall_sum&temperature_unit=fahrenheit&windspeed_unit=mph&timezone=America%2FNew_York&past_days=92 https://api.open-meteo.com/v1/forecast?latitude=40.71&longit...
- meteo-jeff 4y ago
  Thanks for the info. Snowfall was recently added. I am afraid there could be a bug with that particular variable. Temperature, clouds, etc, seem fine EDIT: Issue identified and will be fixed in the next days! Thanks!
bernulli 4y ago
Hi meteo-jeff, this looks really cool! I have two questions: 1) How does the spatial resolution come into this? Is it constant data all across the 2kmx2km (?) parcel with an abrupt change, or is it interpolated in some way? Can I query the coordinates of the mesh? 2) How 'historical' does it get? How far back can I go with this? Thank you!
- flak48 4y ago
  The tool itself seems to provide upto 3 months of history: https://open-meteo.com/en/docs#latitude=52.52&longitude=13.41&hourly=temperature_2m https://open-meteo.com/en/docs#latitude=52.52&longitude=13.4...
- meteo-jeff 4y ago
  Hi bernulli, 1) Data are coming from multiple weather models. Primary data source is the German Weather service DWD with the ICON weather model. In my past experiences, the DWD ICON model performs best for many regions. DWD ICON has a global (~13 km), European (7 km) and a Central Europe (1-2 km) "domain". A higher resolution can improve forecast accuracy, but this is not guaranteed. For Open-Meteo APIs, multiple models are mixed together. Typically high resolution domains only provide 3-5 days of forecast, afterwards they are combined with a global model. For North American locations, I am going to add high resolution domains from NOAA as-well. 2) For now, only couple of months archive are available. There will be no limit of how much data can be stored. Data is fairly well compressed while still maintaining good read performance. I am working on a long term archive as well. ECMWF provides a reanalysis dataset called ERA5 [1] with data from 1959. It will still take me a couple of weeks to process it. With 23 weather variables, it requires around 20 TB disk space (Gridded float32 with deflate compression). [1] https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysi...
  - bernulli 4y ago
    Thank you! I don’t think I understand the resolution then, could you explain a bit more? Say, I request data along a 100km line, every 10m. Do I get the same numbers if it’s in the same mesh cell with a sudden change when it’s crossed, or do I get some (bilinear?) interpolation?
farmin 4y ago
Do you have a commercial option? Does anyone know good alternatives? What forecast models do you use for Australia?
- boulos 4y ago
  Looks like "please contact us" (https://open-meteo.com/en/features#terms https://open-meteo.com/en/features#terms). So you're in the right place? :)
- ericd 4y ago
  I think the issue is probably that NOAA data is gathered via a bunch of agreements with other countries, and at least some of them stipulate non-commercial use. Unsure about how vigorously that’s actually enforced. In you’re curious to read the license text: https://gourdian.net/g/eric/noaa_gsod.global_summary_of_day#summary https://gourdian.net/g/eric/noaa_gsod.global_summary_of_day#...
- meteo-jeff 4y ago
  Hi farmin, so far I do not offer commercial options, just to keep me out of any potential legal issues. In the next weeks I will review everything, make sure attributions and licenses are correct, and remove the non-commercial limitation. Australia is currently only covered by a global weather model from German Weather service DWD. I will check if BOM offers some open-data models
  - farmin 4y ago
    Thanks for your reply. I think BoM do offer some stuff. But check out SILO for easy access to historical data. https://www.longpaddock.qld.gov.au/silo/ https://www.longpaddock.qld.gov.au/silo/
    - hugh-avherald 4y ago
      BoM means $$ even for the last 72hrs
  - chipsa 4y ago
m0llusk 4y ago
This is how we could defeat a rogue AI: Distract it by talking about the weather.
ricksunny 4y ago
Original open-meteo HN thread for background (9 months back) https://news.ycombinator.com/item?id=28499910 https://news.ycombinator.com/item?id=28499910
melony 4y ago
You can get these for free from the government websites.
- deleted 4y ago
  [deleted]
- dx034 4y ago
  Are there any governments publishing world wide data over API? Couldn't find any last time I checked.
  - melony 4y ago
    Not as an API, but for ML purposes static data (with a quick cleaning script) is perfectly usable. The amount of storage needed for weather data isn't worth spending money on an API. This isn't like financial data where storage could potentially become a problem for the casual user.
    - testrun 4y ago
      Did you train on weather data? And if so, how accurate was/is your forecasts?
  - gspr 4y ago
    https://cds.climate.copernicus.eu/ https://cds.climate.copernicus.eu/
MeteorMarc 4y ago
It would also be fun to have the historical weather *forecasts* so that you can compare the forecasts with the eventually measured data.
- meteo-jeff 4y ago
  Actually, they are historical weather forecasts, but assembled to a continuous time-series. Storing each weather forecast individually to a performance evaluation for "how good a forecast in 5 days is", would require a lot of storage. Some local weather models update every 6 hours. But even with a continuous time-series, you can already tell how good or bad a forecast compared to measurements are. Assuming, your measurements are correct ;-)
  - indoor47 4y ago
    This would still be a remarkable dataset for learning. And worth the storage. Though it might need other inputs as well (like pressure zone etc.) to escape potential biases.
meteo-jeff 4y ago
Some technical background: Open-Meteo offers free weather APIs for a while now. Archiving data was not an option, because forecast data alone required 300 GB storage. In the past couple of weeks, I started to look for fast and efficient compression algorithms like zstd, brotli or lz4. All of them, performed rather poor with time-series weather data. After a lot of trial and error, I found a couple of pre-processing steps, that improve compression ratio a lot: 1) Scaling data to reasonable values. Temperature has an accuracy of 0.1° at best. I simply round everything to 0.05 instead of keeping the highest possible floating point precision. 2) A temperature time-series increases and decreases by small values. 0.4° warmer, then 0.2° colder. Only storing deltas improves compression performance. 3) Data are highly spatially correlated. If the temperature is rising in one "grid-cell", it is rising in the neighbouring grid cells as well. Simply subtract the time-series from one grid-cell to the next grid-cell. Especially this yielded a large boost. 4) Although zstd performs quite well with this encoded data, other integer compression algorithms have far better compression and decompression speeds. Namely I am using FastPFor. With that compression approach, an archive became possible. One week of weather forecast data should be around 10 GB compressed. With that, I can easily maintain a very long archive.
- subleq 4y ago
  Would flac work for compression? Given the weather data is a time series of numbers it could be represented as audio. It would then automatically do the difference encoding thing you’re doing. If you encoded nearby grid cells as audio channels, flac would even handle the correlation like it does for stereo audio.
- Brometheus 4y ago
  Hello, good work. Please investigate if you would like to work for this company: https://www.energymeteo.de/ueber_uns/jobs.php https://www.energymeteo.de/ueber_uns/jobs.php
  - O__________O 4y ago
    Feedback: (1) Maybe it’s just me, but the “current jobs” are only available in German, if you switch to English, Spanish or French — the page gets translated, but the three “current jobs” drop down lists get removed; super confusing, since it gets reset to German if you click “current jobs” from any of the other pages; (2) HN is an English site, would be nice if you were linking to the English page, not German; (3) if you’re affiliated with the company, which I believe you are, you should say so and noting it in your profile with contact information would be nice too. (4) Reminder that HN has free job postings every month if you are affiliated with the company: https://news.ycombinator.com/submitted?id=whoishiring https://news.ycombinator.com/submitted?id=whoishiring
m3kw9 4y ago
Still trying to predict weather using historical is like trying to predict the next number on a roulette using historical numbers
- bronco21016 4y ago
  I think historical data isn’t necessarily applied to forecasting the weather. For example, my first thought with this data is to comb through and build a model where when X conditions exist, then Y airport delays are likely. The FAA doesn’t give the data for their end of the model though.
- hansel_der 4y ago
  it's really not
- chipsa 4y ago
  If yesterday, the weather was clear skies, and the temperature followed a given curve, then today if the weather was clear skies, the temperature is going to follow a very similar curve. Same rough curve if it was clear skies a year ago. The exact values might be a bit different, but the high for the day will probably be about the same amount over the sunrise temp for all 3 of these scenarios. Throw in wind direction, and you could be more accurate with this. This is why the 14th Weather Squadron creates Wind Stratified Conditional Climotology tables. Past performance is indicative of future results, especially when you're not under the influence of a frontal system.
mhalle 4y ago
Thanks for offering this service! You explain your API offers historic data using the "past_days" parameter. Could you also offer a "date" parameter for a given day, or are you only keeping a rolling window of data?
- meteo-jeff 4y ago
  Sure. What do you think about "&start_date=20220701" and "&end_date=20220714"? If end_date is not specified, it would return start_date with 7 days forecast
Kalanos 4y ago
check out https://docs.aiqc.io https://docs.aiqc.io for easy walk-forward, multivariate deep learning: https://docs.aiqc.io/notebooks/gallery/tensorflow/tab_forecast.html https://docs.aiqc.io/notebooks/gallery/tensorflow/tab_foreca... excited to play w some of this data

196 points by meteo-jeff 4y ago

Does not seem accurate. This is telling me it snowed 1.33cm on June 17, 2022 in New York City. https://api.open-meteo.com/v1/forecast?latitude=40.71&longitude=-74.01&daily=rain_sum,snowfall_sum&temperature_unit=fahrenheit&windspeed_unit=mph&timezone=America%2FNew_York&past_days=92 https://api.open-meteo.com/v1/forecast?latitude=40.71&longit...

meteo-jeff 4y ago
Thanks for the info. Snowfall was recently added. I am afraid there could be a bug with that particular variable. Temperature, clouds, etc, seem fine EDIT: Issue identified and will be fixed in the next days! Thanks!

bernulli 4y ago

Hi meteo-jeff, this looks really cool! I have two questions: 1) How does the spatial resolution come into this? Is it constant data all across the 2kmx2km (?) parcel with an abrupt change, or is it interpolated in some way? Can I query the coordinates of the mesh? 2) How 'historical' does it get? How far back can I go with this? Thank you!

flak48 4y ago
The tool itself seems to provide upto 3 months of history: https://open-meteo.com/en/docs#latitude=52.52&longitude=13.41&hourly=temperature_2m https://open-meteo.com/en/docs#latitude=52.52&longitude=13.4...
meteo-jeff 4y ago
Hi bernulli, 1) Data are coming from multiple weather models. Primary data source is the German Weather service DWD with the ICON weather model. In my past experiences, the DWD ICON model performs best for many regions. DWD ICON has a global (~13 km), European (7 km) and a Central Europe (1-2 km) "domain". A higher resolution can improve forecast accuracy, but this is not guaranteed. For Open-Meteo APIs, multiple models are mixed together. Typically high resolution domains only provide 3-5 days of forecast, afterwards they are combined with a global model. For North American locations, I am going to add high resolution domains from NOAA as-well. 2) For now, only couple of months archive are available. There will be no limit of how much data can be stored. Data is fairly well compressed while still maintaining good read performance. I am working on a long term archive as well. ECMWF provides a reanalysis dataset called ERA5 [1] with data from 1959. It will still take me a couple of weeks to process it. With 23 weather variables, it requires around 20 TB disk space (Gridded float32 with deflate compression). [1] https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysi...
- bernulli 4y ago
  Thank you! I don’t think I understand the resolution then, could you explain a bit more? Say, I request data along a 100km line, every 10m. Do I get the same numbers if it’s in the same mesh cell with a sudden change when it’s crossed, or do I get some (bilinear?) interpolation?

farmin 4y ago

Do you have a commercial option? Does anyone know good alternatives? What forecast models do you use for Australia?

boulos 4y ago
Looks like "please contact us" (https://open-meteo.com/en/features#terms https://open-meteo.com/en/features#terms). So you're in the right place? :)
ericd 4y ago
I think the issue is probably that NOAA data is gathered via a bunch of agreements with other countries, and at least some of them stipulate non-commercial use. Unsure about how vigorously that’s actually enforced. In you’re curious to read the license text: https://gourdian.net/g/eric/noaa_gsod.global_summary_of_day#summary https://gourdian.net/g/eric/noaa_gsod.global_summary_of_day#...
meteo-jeff 4y ago
Hi farmin, so far I do not offer commercial options, just to keep me out of any potential legal issues. In the next weeks I will review everything, make sure attributions and licenses are correct, and remove the non-commercial limitation. Australia is currently only covered by a global weather model from German Weather service DWD. I will check if BOM offers some open-data models
- farmin 4y ago
  Thanks for your reply. I think BoM do offer some stuff. But check out SILO for easy access to historical data. https://www.longpaddock.qld.gov.au/silo/ https://www.longpaddock.qld.gov.au/silo/
  - hugh-avherald 4y ago
    BoM means $$ even for the last 72hrs
- chipsa 4y ago

m0llusk 4y ago

This is how we could defeat a rogue AI: Distract it by talking about the weather.

ricksunny 4y ago

Original open-meteo HN thread for background (9 months back) https://news.ycombinator.com/item?id=28499910 https://news.ycombinator.com/item?id=28499910

melony 4y ago

You can get these for free from the government websites.

deleted 4y ago
[deleted]
dx034 4y ago
Are there any governments publishing world wide data over API? Couldn't find any last time I checked.
- melony 4y ago
  Not as an API, but for ML purposes static data (with a quick cleaning script) is perfectly usable. The amount of storage needed for weather data isn't worth spending money on an API. This isn't like financial data where storage could potentially become a problem for the casual user.
  - testrun 4y ago
    Did you train on weather data? And if so, how accurate was/is your forecasts?
- gspr 4y ago
  https://cds.climate.copernicus.eu/ https://cds.climate.copernicus.eu/

MeteorMarc 4y ago

It would also be fun to have the historical weather *forecasts* so that you can compare the forecasts with the eventually measured data.

meteo-jeff 4y ago
Actually, they are historical weather forecasts, but assembled to a continuous time-series. Storing each weather forecast individually to a performance evaluation for "how good a forecast in 5 days is", would require a lot of storage. Some local weather models update every 6 hours. But even with a continuous time-series, you can already tell how good or bad a forecast compared to measurements are. Assuming, your measurements are correct ;-)
- indoor47 4y ago
  This would still be a remarkable dataset for learning. And worth the storage. Though it might need other inputs as well (like pressure zone etc.) to escape potential biases.

meteo-jeff 4y ago

Some technical background: Open-Meteo offers free weather APIs for a while now. Archiving data was not an option, because forecast data alone required 300 GB storage. In the past couple of weeks, I started to look for fast and efficient compression algorithms like zstd, brotli or lz4. All of them, performed rather poor with time-series weather data. After a lot of trial and error, I found a couple of pre-processing steps, that improve compression ratio a lot: 1) Scaling data to reasonable values. Temperature has an accuracy of 0.1° at best. I simply round everything to 0.05 instead of keeping the highest possible floating point precision. 2) A temperature time-series increases and decreases by small values. 0.4° warmer, then 0.2° colder. Only storing deltas improves compression performance. 3) Data are highly spatially correlated. If the temperature is rising in one "grid-cell", it is rising in the neighbouring grid cells as well. Simply subtract the time-series from one grid-cell to the next grid-cell. Especially this yielded a large boost. 4) Although zstd performs quite well with this encoded data, other integer compression algorithms have far better compression and decompression speeds. Namely I am using FastPFor. With that compression approach, an archive became possible. One week of weather forecast data should be around 10 GB compressed. With that, I can easily maintain a very long archive.

subleq 4y ago
Would flac work for compression? Given the weather data is a time series of numbers it could be represented as audio. It would then automatically do the difference encoding thing you’re doing. If you encoded nearby grid cells as audio channels, flac would even handle the correlation like it does for stereo audio.
Brometheus 4y ago
Hello, good work. Please investigate if you would like to work for this company: https://www.energymeteo.de/ueber_uns/jobs.php https://www.energymeteo.de/ueber_uns/jobs.php
- O__________O 4y ago
  Feedback: (1) Maybe it’s just me, but the “current jobs” are only available in German, if you switch to English, Spanish or French — the page gets translated, but the three “current jobs” drop down lists get removed; super confusing, since it gets reset to German if you click “current jobs” from any of the other pages; (2) HN is an English site, would be nice if you were linking to the English page, not German; (3) if you’re affiliated with the company, which I believe you are, you should say so and noting it in your profile with contact information would be nice too. (4) Reminder that HN has free job postings every month if you are affiliated with the company: https://news.ycombinator.com/submitted?id=whoishiring https://news.ycombinator.com/submitted?id=whoishiring

m3kw9 4y ago

Still trying to predict weather using historical is like trying to predict the next number on a roulette using historical numbers

bronco21016 4y ago
I think historical data isn’t necessarily applied to forecasting the weather. For example, my first thought with this data is to comb through and build a model where when X conditions exist, then Y airport delays are likely. The FAA doesn’t give the data for their end of the model though.
hansel_der 4y ago
it's really not
chipsa 4y ago
If yesterday, the weather was clear skies, and the temperature followed a given curve, then today if the weather was clear skies, the temperature is going to follow a very similar curve. Same rough curve if it was clear skies a year ago. The exact values might be a bit different, but the high for the day will probably be about the same amount over the sunrise temp for all 3 of these scenarios. Throw in wind direction, and you could be more accurate with this. This is why the 14th Weather Squadron creates Wind Stratified Conditional Climotology tables. Past performance is indicative of future results, especially when you're not under the influence of a frontal system.

mhalle 4y ago

Thanks for offering this service! You explain your API offers historic data using the "past_days" parameter. Could you also offer a "date" parameter for a given day, or are you only keeping a rolling window of data?

meteo-jeff 4y ago
Sure. What do you think about "&start_date=20220701" and "&end_date=20220714"? If end_date is not specified, it would return start_date with 7 days forecast

Kalanos 4y ago

check out https://docs.aiqc.io https://docs.aiqc.io for easy walk-forward, multivariate deep learning: https://docs.aiqc.io/notebooks/gallery/tensorflow/tab_forecast.html https://docs.aiqc.io/notebooks/gallery/tensorflow/tab_foreca... excited to play w some of this data