11 ms·

Hi, hobbyist here! This is a huge area where government meteorologists and "Big Weather" differ and you can help close that gap! For context: The governments

Hi, hobbyist here! This is a huge area where government meteorologists and "Big Weather" differ and you can help close that gap! For context: The governments of the world provides these big weather companies (weather.com (cough IBM), Accuweather (cough IBM cough), etc) a metric shit ton of their data completely for free (by law) including data transfer. These are things like radar, satellite, ground station data, forecasts, composite models, etc. These companies profit substantially on it, as in billions of dollars. You as citizens also can get this data completely for free as well! MADIS is a system the government is working on to make that data access easier by bringing together many of these systems together and removing the bureaucratic redundancy and abstracting out the aging infrastructure. This is literally terabytes of data per day you can grab with almost no questions asked. That data is then processed privately and resold and repackaged to the end user, and you probably interact with this privatized data the most. The frustration I have much of the additional "value" these weather data brokers provide is by linking up with each other with data contracts. These private companies have a much much higher detail on the ground than the government by being able to partner with companies that make common internet-connected personal radar stations and reselling that data to each other. The government doesn't have that privilege to buy limitless data. NOAA/NWS, for example, is extremely underfunded so if they had to privilege to buy it they probably couldn't come to an agreement to buy it. As a result, they can't use that data to improve the accuracy of alerts/warnings/forcasts, the same exact tools that the big weather companies make all their money from. It's a shit cycle and totally unfair IMO. So please contribute if you can!! Sidebar: I'm a founder of a self-bootstrapped startup to build a better weather data broker that doesn't cost an arm and a leg. If that's something you are passionate about solving, feel free to reach out :)

snowfield 2y ago
You can always use yr.no This is the Norwegian government weather service. It's global and free for everyone. Also has fully open apis
- lukan 2y ago
  I second this. It is incredibly fast, no bloat, no ads. Just not as accurate as a local service as their main focus is - norway.
  - snowfield 2y ago
    Idk about that. It's used by farmers all over the world. They do purchase local data.
positr0n 2y ago
Do you have a source for sites like weather.com and AccuWeather being part of a billions of dollars industry? That is surprising to me.
- datadrivenangel 2y ago
  https://en.wikipedia.org/wiki/AccuWeather https://en.wikipedia.org/wiki/AccuWeather
  - positr0n 2y ago
    > AccuWeather's estimated annual revenue is currently $156.6M per year.
- mikeortman 2y ago
  It's hard to get accurate numbers as many weather sites are privately held. But to give some info: Weather Company was sold for $2B to IBM in 2015 (and recently sold to private equity for undisclosed amount). Tomorrow.IO has a $1B+ valuation pre-IPO SPAC. https://www.forbes.com/sites/johnkoetsier/2021/04/08/ibms-weather-company-has-a-path-to-billions-in-new-revenue-thanks-to-the-rise-of-the-subscription-economy/ https://www.forbes.com/sites/johnkoetsier/2021/04/08/ibms-we...
  - WrongAssumption 2y ago
    That’s not really indicative of a large industry. If the weather company was bought for 2 billion, at standard valuations they were probably pulling at most 100 million per year in net income.
    - keithalewis 2y ago
      User name checks out.
- akira2501 2y ago
  You can't safely fly or sail unless you know and can predict the weather at your destination.
2Gkashmiri 2y ago
I have always wanted to build a cheap ESP based weather station I can give to a friend, have them set up their WiFi and it would transmit data to the central server and I do not want to submit data to AccuWeather or wunderground or someone else. Is there a Foss server around that I can set up on my own ?
- mikeortman 2y ago
  https://www.weewx.com https://www.weewx.com
  - zikduruqe 2y ago
    My weewx station has been running since 2013 on a RPi.
- imoverclocked 2y ago
  FWIW, purple air is basically this minus wind information.
_xivi 2y ago
> weather.com (cough IBM) Didn't IBM sell weather.com (and all of the assets) to a private equity? The deal was announced last year and closed in Feb 2024 https://finance.yahoo.com/news/francisco-partners-completes-acquisition-weather-140000998.html https://finance.yahoo.com/news/francisco-partners-completes-...
- mikeortman 2y ago
  Yeah, they did... but being blunt, IBM had 8 years to do a lot of damage. Not speaking of the engineers or the tech behind it, I'm talking about escalating the ad revenue and general junk on their website. MOST of the page load time on weather.com and AccuWeather is ad and tracking. It's an ungodly number of requests and will drain a data plan's usage limits surprisingly quickly. Just opening weather.com will send almost 1000 requests , transfer 10.3MB. Every 30 seconds or so it will make about 300 requests + 2MB of transfer for new ads. It's... insane
  - jeffbee 2y ago
    Ads and cookies are hardly the worst thing that AccuWeather has done. They have been lobbying the federal government for 30 years to forbid NOAA from issuing forecasts at all. They want the government to pay for all the satellites and supercomputers, and they want all that data for free, but they want you to have to pay AccuWeather for forecasts.
    - Teever 2y ago
      In situations like this I feel that it is important to put a face to a name. AccuWeather, a corporation like any other is made of people. These shortsighted decisions and shitty behavior are directly attributable to people who do them for selfish reasons. So let's start talking about these people in the first person, not in the abstract. Who is directly responsible for these atrocious actions and why do they stand to benefit from them?
      - jonathankoren 2y ago
tcmart14 2y ago
This sounds really cool. Have you looked into see if you can get permission from the guys over at aprs.fi to scrape their data from ham radio weather stations running over APRS? [1] [0] https://aprs.fi/page/api https://aprs.fi/page/api
- thcipriani 2y ago
  My rough understanding is CWOP was started by ham nerds and forwards all data to MADIS via FindU[0] [0]: <http://www.findu.com/ http://www.findu.com/>
DavidPeiffer 2y ago
> These private companies have a much much higher detail on the ground than the government by being able to partner with companies that make common internet-connected personal radar stations and reselling that data to each other. I haven't heard of personal radar stations, and wasn't hitting anything in a quick web search. Are you able to provide an example of these systems?
- semi-extrinsic 2y ago
  I'm pretty sure this was a typo, so s/radar/weather/ . For reference, a weather radar operates in Doppler mode with return signal coming from Rayleigh scattering of raindrops, so it's on the 3cm - 10cm wavelength. You are talking about something like a 5 meter diameter antenna dish that weighs half a tonne, which is on an elevation-azimuth motorized mount, in a 7 meter diameter radome, with peak transmit power of 250 000 W. Of course you can buy one yourself, if you have the space, electrical power and money for it - ballpark 1.5 mill. USD.
  - jonathankoren 2y ago
    I don't know how much this costs, but it's only a meter, 65 kg, and runs on a 110 volt outlet. I'm thinking high ones of thousands of dollars, maybe low tens of thousands. https://www.furuno.com/files/Brochure/448/upload/WR110_EN.pdf https://www.furuno.com/files/Brochure/448/upload/WR110_EN.pd...
    - defrost 2y ago
      They only list agents and the agents don't list prices and few feature the doppler weather radar on ther website .. That said, I'd guess $10,000 US < price < $50,000 US for that product given the pricing of $8K US for smaller commercial fishing radars from the same company. More on the two compact weather radar systems: https://www.furuno.com/en/systems/meteorological-monitoring/ https://www.furuno.com/en/systems/meteorological-monitoring/
      - closewith 2y ago
8bitsrule 2y ago
Thanks for mentioning. When I looked up MADIS, Wiki pointed me here: https://www.weather.gov/ncep/ https://www.weather.gov/ncep/ There, selecting 'Weather Prediction Center' goes to https://www.wpc.ncep.noaa.gov/#page=ovw https://www.wpc.ncep.noaa.gov/#page=ovw There I searched in 'Local forecast' for cities and see most of what 'weather com' delivers, with 3 domains in Noscript instead of the couple-dozen from weather.com
hejdufufjrj 2y ago
Big Weather does provide the data for "free" to the weather widgets on millions of phones and computers. So they do provide a public service, even if maybe they get too much money from it.
theyinwhy 2y ago
I don't understand why the quality of the specific weather station does not matter. Isn't bad data impacting the overall model quality?
zer00eyz 2y ago
If you see this post, You should go share your story and probably this effort with the Home automation and Home assistant crowd. As consumers and creators of plenty of weather data you might see a fair bit of traction there!
szvsw 2y ago
> Sidebar: I'm a founder of a self-bootstrapped startup to build a better weather data broker that doesn't cost an arm and a leg. If that's something you are passionate about solving, feel free to reach out :) Would love to hear more about this. I’m a researcher and a lot of my work revolves around machine learning applications to building energy modeling, and one of my projects actually revolves around the difference between using TMY vs AMY EPW files in automatic calibration of models. Would be great to chat more. What’s the name of your startup? I can shoot you an email at the official email.
zorm 2y ago
Since the Weather Research and Forecasting Innovation Act of 2017, Congress is requiring NOAA to start acquiring data via commercial partnerships. NOAA has already made some contracts with Spire [1] and Saildrone [2]. I am sure there are more but these are the ones most familiar to me. Your weather data broker startup sounds very interesting! [1]: https://spire.com/press-release/spire-global-awarded-national-oceanic-and-atmospheric-administration-contract-for-satellite-weather-data/ https://spire.com/press-release/spire-global-awarded-nationa... [2]: https://research.noaa.gov/2022/08/03/noaa-and-saildrone-team-up-to-deploy-seven-hurricane-tracking-surface-drones/ https://research.noaa.gov/2022/08/03/noaa-and-saildrone-team...
- counters 2y ago
  The entire slate of commercial acquisitions planned or in progress can be found at [1]. It's pretty anemic; NOAA has spent far less than what folks were hoping they would. I think a major part of this is that the private sector really didn't have very many high-TRL observation systems that could readily be integrated into NOAA's assimilation and forecasting systems. Lots of planned constellations and ideas about things to do in the future, but just not that much stuff that was ready to package-up and deliver to NOAA. The most successful acquisitions for GPS/GNSS-RO and buoy/drone data seems strongly bolstered by the fact that these data were already readily assimilated by existing infrastructure. The private sector has really embellished its capabilities to the detriment of the CDP and other programs. I think too many industry players saw NOAA's expansion here as a potential slush fund to fully subsidize their R&D, but again the TRL of planned observation systems was too low and so the system didn't really work efficiently. Classic policy failure - would make a fantastic case study or Master's thesis for someone studying weather in an STS program! [1]: https://www.space.commerce.gov/business-with-noaa/commercial-weather-data-pilot-cwdp/ https://www.space.commerce.gov/business-with-noaa/commercial...
jnurmine 2y ago
For the data, do you know meteostat.net? They get input from e.g. NOAA (US), DWD (DE), YR (NO) and so on. I'm not affiliated, just needed some historical data and (finally!) discovered them.
open-meteo 2y ago
Hi, I am building an Open-Source Weather API aggregating open-data weather models from NOAA, ECMWF, DWD, MeteoFrance, JMA, CMA, CMCC and others. I agree that many weather companies basically redistribute NWS data at a premium. There is a free API service available on https://open-meteo.com https://open-meteo.com and all databases are redistributed via an AWS Open-Data Sponsorship. Feel free to reach out if you need help building your weather data broker startup
- seabass-labrax 2y ago
  A sincere personal "thank you" from me! I use Breezy Weather[1] on my smartphone, which gets most of its data from your aggregating service, Open-Meteo. It's the perfect combination of accurate, international and open. [1]: https://f-droid.org/en/packages/org.breezyweather/ https://f-droid.org/en/packages/org.breezyweather/
- just_testing 2y ago
  First of all, thank you! What a treasure trove of data. If I may ask a question, do you have historical air quality data?
  - open-meteo 2y ago
    You can access historical air quality data from August 2022 onwards: https://open-meteo.com/en/docs/air-quality-api#start_date=2022-07-29&end_date=2024-06-09&time_mode=time_interval https://open-meteo.com/en/docs/air-quality-api#start_date=20.... Data is based on the Copernicus air quality forecast. All are references listed the documentation
    - just_testing 2y ago
      Thank you!
- CraigGilmore 2y ago
  I am working on this kind of solution - please email - see about
kmbfjr 2y ago
I became so annoyed at the tracking, advertising and poor notifications of many apps, I wrote my own for severe weather alerts. My hobby will include a C band antenna this fall, and I’m on the hunt for radar data sources in which to create my own mosaics.
ptero 2y ago
Thank you for doing this! As my own sidebar, I spent many years at a national lab working with distributed sensor networks (primarily ATC and other radar for detecting non-weather stuff :) ). I thought about using ADS-B as input for weather state and forecasts, but never got around to trying it. Now that I am working on my own startup (self-funded and without revenue so far), so this again will likely languish in my todo list. If someone wants to try it, great, and feel free to reach out as I can probably save you some time selecting and interpreting the right ADS-B fields: We have a lot of aircraft blasting ADS-B reports whenever they fly. Most reports contain (1) accurate 3D position, including altitude and (2) barometric altitude measurements, which gives you (after some minimal work) air pressure. So you have millions freely available pressure reports not just on the ground, but throughout 0-40000 ft altitude band. You also get measured airspeed and groundspeed, so in addition to pressure you get wind vectors at thousands of points in the air, updating in real time. I suspect this can provide some non-trivial information and I am not aware of anyone actually using it for this purpose.
- counters 2y ago
  These data are already consumed operationally by the major global weather modeling centers - e.g., check out the AMDAR [1] or ACARS programs. There are commercial agreements which restrict third-party access to these data in real-time, but they are widely disseminate with a 24-48 hour lag in the observational archives that NOAA curates for weather modeling. These data are a very important input for operational weather forecasting. On a global basis, we've seen how losing these data during the pandemic due to reductions in air travel decreased forecast model skill [2]. Furthermore, ACARS profiles derived from aircraft landing at airports where severe weather is expected are regularly used to complement SPECI weather balloon launches. [1]: https://community.wmo.int/en/activity-areas/aircraft-based-observations/amdar/amdar-description https://community.wmo.int/en/activity-areas/aircraft-based-o... [2]: https://journals.ametsoc.org/view/journals/apme/59/11/JAMC-D-20-0010.1.xml https://journals.ametsoc.org/view/journals/apme/59/11/JAMC-D...
  - ptero 2y ago
    Thank you for the info, I am glad to hear that the data is being collected and used. Getting it from raw ADS-B reports makes all fields available without the lag (and thus more useful for weather reporting and forecasting), but the recorded data is definitely good to have!
fallingknife 2y ago
Why is this unfair? With these private companies I get access to all this data for free. If the government did it I would have to pay for it out of my taxes. Why do we need the government to provide more accurate forecasts when this is already taken care of by the private sector?
- chiph 2y ago
  > With these private companies I get access to all this data for free. These companies are already receiving government data for free. And then enriching it with private weather station data (and ads). So your taxes are already funding some part of it. As computing power being used for forecasting increases (14.5 petaflops each for both Dogwood and Cactus NOAA supercomputers), having more granular data is going to be useful in improving their models. https://www.noaa.gov/news-release/noaa-completes-upgrade-to-weather-and-climate-supercomputer-system https://www.noaa.gov/news-release/noaa-completes-upgrade-to-...
- desert_rue 2y ago
  It is in the public interest to have both weather and forecasts. Maybe you are like me and look out the window to see the weather for that day, but millions need accurate weather and forecasts. They include farmers, those that run power grids, air traffic controllers, and more.
- Workaccount2 2y ago
  You get that data for "free" or your hand over your browser data - and likely location data, to unknown third party brokers that know more about you than you know about yourself. Weather apps in particular are notorious for selling your location data, since everyone does "show weather for my current location". It's "free" in the worst sense of the word.
  - radicality 2y ago
    Do you know much about Windy, and whether they have any ties to selling user location data? All the plots and features seemed nice so I paid for a year of subscription (after Dark Sky was discontinued ugh), but if they do that I’ll just stick to the default apple Weather app.
counters 2y ago
> NOAA/NWS, for example, is extremely underfunded so if they had to privilege to buy it they probably couldn't come to an agreement to buy it. As a result, they can't use that data to improve the accuracy of alerts/warnings/forcasts, the same exact tools that the big weather companies make all their money from. It's a shit cycle and totally unfair IMO. Huh? This is kind of an odd take for a few reasons. For starters, NOAA isn't "extremely underfunded"; with the possible exception of the current budgeting cycle, NOAA generally does pretty well and has strong bipartisan support. It could always use more money, but I wouldn't call it "underfunded. The reason NOAA doesn't buy more data is because most of the available data has limited value. Personal weather stations have substantial quality issues and add almost no value in areas where we already have high-quality surface observations. We thin out and throw away a ton of surface observations already during the data assimilation process to initialize our forecast models anyways - data from aloft is far more valuable and impactful from a forecast impact perspective. For what it's worth, few if any companies use proprietary observations to improve their forecasts. It's an open secret that the vast majority of companies out there are just applying proprietary statistical modeling / bias correction on top of publicly available data. Only a handful of companies actually have novel observations, and there's limited evidence it makes a significant difference in the forecast. At best, it can result in the way that those statistical corrections are applied to existing forecasts and ensembles - you can count on one hand the number of companies that actually run a vertically-integrated stack including data assimilation of proprietary observations and end-to-end numerical modeling. That isn't to say there isn't unique value in the observations. It's just that the industry flagrantly misleads about how they use them.
- pinkmuffinere 2y ago
  Referring to the parent comment, the data which NOAA isn’t able to buy is the government’s data, which is freely provided to non-government organizations. The parent comment doesn’t discuss personal data very much. I think this misunderstanding might be the root cause of your disagreement.
  - counters 2y ago
    Parent comment is mis-informed. NOAA doesn't have to buy any data from other government agencies or organizations. It's all open and publicly available. There are challenges around the reliability and quality of data that NOAA doesn't take efforts itself to curate and maintain which limit their utility, but that's a separate issue. More importantly, NOAA explicitly funds the National Mesonet Program [1] to actively identify, acquire, consume, and ingest data from a wide variety of state and federal agencies across the country. The NMP itself partners with a major private sector company, Synoptic Data PBC [2] to perform the engineering necessary to acquire all this data. Synoptic actively maintains the infrastructure which consumes and publishes this data to MADIS for use by NOAA and any other stakeholder. [1]: https://nationalmesonet.us/ https://nationalmesonet.us/ [2]: https://synopticdata.com/ https://synopticdata.com/
mozman 2y ago
what’s good weather station hw to get? thinking ultrasonic wind instrument and some type of radar
- FireBeyond 2y ago
  Doesn't encompass radar (and no home weather station will) but NetAtmo sells one with ultrasonic wind, smart rain gauge, and has a full RESTful API available: https://www.netatmo.com/smart-weather-station https://www.netatmo.com/smart-weather-station I use this and love it.
  - VENTS 2y ago
    You can watch it here: https://lcjcapteurs.com/en/girouette-anemometres-capteur-vent/ultrasonic-anemometer-cv7-hr-oem/ https://lcjcapteurs.com/en/girouette-anemometres-capteur-ven... or https://www.davisinstruments.com/products/sonic-anemometer https://www.davisinstruments.com/products/sonic-anemometer
HumblyTossed 2y ago
Don't forget, AccuWeather wanted to make it illegal for you to have access to that data for free from the government.
- widerporst 2y ago
  Same thing in Germany, with WetterOnline ruminating data from DWD (the German meteorological service) and then suing them when they offered their DWD WarnWetter app for free. Unfortunately, they were successful: https://www.heise.de/news/BGH-Urteil-Staatlicher-Wetterdienst-muss-Gratis-App-beschraenken-4681828.html https://www.heise.de/news/BGH-Urteil-Staatlicher-Wetterdiens... (Couldn't find an English source, sorry.)
  - Rinzler89 2y ago
    >Unfortunately, they were successful Unbelievable.
- Wistar 2y ago
  Alas, there are yet ongoing attempts to break-up and/or privatize the NWS. Project 2025 specifically calls for the break-up of NOAA.
  - HumblyTossed 2y ago
    Oh for crying out loud! WTF??? I think someone spilled idiot juice in the water.
    - wormius 2y ago
      Swamp Water Kool-Aid for the cultists.
supportengineer 2y ago
What is the best option for someone who wants to contribute but doesn’t have a weather station currently or a lot of free time to take on another project?
alexpotato 2y ago
"The governments of the world provides these big weather companies (weather.com (cough IBM), Accuweather (cough IBM cough), etc) a metric shit ton of their data completely for free (by law) including data transfer. These are things like radar, satellite, ground station data, forecasts, composite models, etc. These companies profit substantially on it, as in billions of dollars" Michael Lewis' book The Fifth Risk goes IN DEPTH into how Accuweather/Weather.com and the government were interacting (particularly during the Trump administration). I highly recommend the book in general and for this particular story in particular.
just_testing 2y ago
If I may use your expertise a bit: I need air quality data for two cities, one of which doesn't have any air quality monitoring stations. Is there anything I can use as a proxy for air pollution, like satellite data? Also, do you know any sources for that data? Thanks! P.S.: I'm passionate about air quality, given I have alergies and was an active member of the bicycle activism movement for about 10 years.
- toomuchtodo 2y ago
  Might find useful info here: https://www.nasa.gov/missions/tempo/nasa-releases-new-high-quality-near-real-time-air-quality-data/ https://www.nasa.gov/missions/tempo/nasa-releases-new-high-q... https://news.ycombinator.com/item?id=40530798 https://news.ycombinator.com/item?id=40530798 https://news.ycombinator.com/item?id=40534632 https://news.ycombinator.com/item?id=40534632
  - just_testing 2y ago
    Thank you!!
CraigGilmore 2y ago
Hi Mike- I am very interested in this and doing some work - please email

snowfield 2y ago

You can always use yr.no This is the Norwegian government weather service. It's global and free for everyone. Also has fully open apis

lukan 2y ago
I second this. It is incredibly fast, no bloat, no ads. Just not as accurate as a local service as their main focus is - norway.
- snowfield 2y ago
  Idk about that. It's used by farmers all over the world. They do purchase local data.

positr0n 2y ago

Do you have a source for sites like weather.com and AccuWeather being part of a billions of dollars industry? That is surprising to me.

datadrivenangel 2y ago
https://en.wikipedia.org/wiki/AccuWeather https://en.wikipedia.org/wiki/AccuWeather
- positr0n 2y ago
  > AccuWeather's estimated annual revenue is currently $156.6M per year.
mikeortman 2y ago
It's hard to get accurate numbers as many weather sites are privately held. But to give some info: Weather Company was sold for $2B to IBM in 2015 (and recently sold to private equity for undisclosed amount). Tomorrow.IO has a $1B+ valuation pre-IPO SPAC. https://www.forbes.com/sites/johnkoetsier/2021/04/08/ibms-weather-company-has-a-path-to-billions-in-new-revenue-thanks-to-the-rise-of-the-subscription-economy/ https://www.forbes.com/sites/johnkoetsier/2021/04/08/ibms-we...
- WrongAssumption 2y ago
  That’s not really indicative of a large industry. If the weather company was bought for 2 billion, at standard valuations they were probably pulling at most 100 million per year in net income.
  - keithalewis 2y ago
    User name checks out.
akira2501 2y ago
You can't safely fly or sail unless you know and can predict the weather at your destination.

2Gkashmiri 2y ago

I have always wanted to build a cheap ESP based weather station I can give to a friend, have them set up their WiFi and it would transmit data to the central server and I do not want to submit data to AccuWeather or wunderground or someone else. Is there a Foss server around that I can set up on my own ?

mikeortman 2y ago
https://www.weewx.com https://www.weewx.com
- zikduruqe 2y ago
  My weewx station has been running since 2013 on a RPi.
imoverclocked 2y ago
FWIW, purple air is basically this minus wind information.

_xivi 2y ago

> weather.com (cough IBM) Didn't IBM sell weather.com (and all of the assets) to a private equity? The deal was announced last year and closed in Feb 2024 https://finance.yahoo.com/news/francisco-partners-completes-acquisition-weather-140000998.html https://finance.yahoo.com/news/francisco-partners-completes-...

mikeortman 2y ago
Yeah, they did... but being blunt, IBM had 8 years to do a lot of damage. Not speaking of the engineers or the tech behind it, I'm talking about escalating the ad revenue and general junk on their website. MOST of the page load time on weather.com and AccuWeather is ad and tracking. It's an ungodly number of requests and will drain a data plan's usage limits surprisingly quickly. Just opening weather.com will send almost 1000 requests , transfer 10.3MB. Every 30 seconds or so it will make about 300 requests + 2MB of transfer for new ads. It's... insane
- jeffbee 2y ago
  Ads and cookies are hardly the worst thing that AccuWeather has done. They have been lobbying the federal government for 30 years to forbid NOAA from issuing forecasts at all. They want the government to pay for all the satellites and supercomputers, and they want all that data for free, but they want you to have to pay AccuWeather for forecasts.
  - Teever 2y ago
    In situations like this I feel that it is important to put a face to a name. AccuWeather, a corporation like any other is made of people. These shortsighted decisions and shitty behavior are directly attributable to people who do them for selfish reasons. So let's start talking about these people in the first person, not in the abstract. Who is directly responsible for these atrocious actions and why do they stand to benefit from them?
    - jonathankoren 2y ago

tcmart14 2y ago

This sounds really cool. Have you looked into see if you can get permission from the guys over at aprs.fi to scrape their data from ham radio weather stations running over APRS? [1] [0] https://aprs.fi/page/api https://aprs.fi/page/api

thcipriani 2y ago
My rough understanding is CWOP was started by ham nerds and forwards all data to MADIS via FindU[0] [0]: <http://www.findu.com/ http://www.findu.com/>

DavidPeiffer 2y ago

> These private companies have a much much higher detail on the ground than the government by being able to partner with companies that make common internet-connected personal radar stations and reselling that data to each other. I haven't heard of personal radar stations, and wasn't hitting anything in a quick web search. Are you able to provide an example of these systems?

semi-extrinsic 2y ago
I'm pretty sure this was a typo, so s/radar/weather/ . For reference, a weather radar operates in Doppler mode with return signal coming from Rayleigh scattering of raindrops, so it's on the 3cm - 10cm wavelength. You are talking about something like a 5 meter diameter antenna dish that weighs half a tonne, which is on an elevation-azimuth motorized mount, in a 7 meter diameter radome, with peak transmit power of 250 000 W. Of course you can buy one yourself, if you have the space, electrical power and money for it - ballpark 1.5 mill. USD.
- jonathankoren 2y ago
  I don't know how much this costs, but it's only a meter, 65 kg, and runs on a 110 volt outlet. I'm thinking high ones of thousands of dollars, maybe low tens of thousands. https://www.furuno.com/files/Brochure/448/upload/WR110_EN.pdf https://www.furuno.com/files/Brochure/448/upload/WR110_EN.pd...
  - defrost 2y ago
    They only list agents and the agents don't list prices and few feature the doppler weather radar on ther website .. That said, I'd guess $10,000 US < price < $50,000 US for that product given the pricing of $8K US for smaller commercial fishing radars from the same company. More on the two compact weather radar systems: https://www.furuno.com/en/systems/meteorological-monitoring/ https://www.furuno.com/en/systems/meteorological-monitoring/
    - closewith 2y ago

8bitsrule 2y ago

Thanks for mentioning. When I looked up MADIS, Wiki pointed me here: https://www.weather.gov/ncep/ https://www.weather.gov/ncep/ There, selecting 'Weather Prediction Center' goes to https://www.wpc.ncep.noaa.gov/#page=ovw https://www.wpc.ncep.noaa.gov/#page=ovw There I searched in 'Local forecast' for cities and see most of what 'weather com' delivers, with 3 domains in Noscript instead of the couple-dozen from weather.com

hejdufufjrj 2y ago

Big Weather does provide the data for "free" to the weather widgets on millions of phones and computers. So they do provide a public service, even if maybe they get too much money from it.

theyinwhy 2y ago

I don't understand why the quality of the specific weather station does not matter. Isn't bad data impacting the overall model quality?

zer00eyz 2y ago

If you see this post, You should go share your story and probably this effort with the Home automation and Home assistant crowd. As consumers and creators of plenty of weather data you might see a fair bit of traction there!

szvsw 2y ago

> Sidebar: I'm a founder of a self-bootstrapped startup to build a better weather data broker that doesn't cost an arm and a leg. If that's something you are passionate about solving, feel free to reach out :) Would love to hear more about this. I’m a researcher and a lot of my work revolves around machine learning applications to building energy modeling, and one of my projects actually revolves around the difference between using TMY vs AMY EPW files in automatic calibration of models. Would be great to chat more. What’s the name of your startup? I can shoot you an email at the official email.

zorm 2y ago

Since the Weather Research and Forecasting Innovation Act of 2017, Congress is requiring NOAA to start acquiring data via commercial partnerships. NOAA has already made some contracts with Spire [1] and Saildrone [2]. I am sure there are more but these are the ones most familiar to me. Your weather data broker startup sounds very interesting! [1]: https://spire.com/press-release/spire-global-awarded-national-oceanic-and-atmospheric-administration-contract-for-satellite-weather-data/ https://spire.com/press-release/spire-global-awarded-nationa... [2]: https://research.noaa.gov/2022/08/03/noaa-and-saildrone-team-up-to-deploy-seven-hurricane-tracking-surface-drones/ https://research.noaa.gov/2022/08/03/noaa-and-saildrone-team...

counters 2y ago
The entire slate of commercial acquisitions planned or in progress can be found at [1]. It's pretty anemic; NOAA has spent far less than what folks were hoping they would. I think a major part of this is that the private sector really didn't have very many high-TRL observation systems that could readily be integrated into NOAA's assimilation and forecasting systems. Lots of planned constellations and ideas about things to do in the future, but just not that much stuff that was ready to package-up and deliver to NOAA. The most successful acquisitions for GPS/GNSS-RO and buoy/drone data seems strongly bolstered by the fact that these data were already readily assimilated by existing infrastructure. The private sector has really embellished its capabilities to the detriment of the CDP and other programs. I think too many industry players saw NOAA's expansion here as a potential slush fund to fully subsidize their R&D, but again the TRL of planned observation systems was too low and so the system didn't really work efficiently. Classic policy failure - would make a fantastic case study or Master's thesis for someone studying weather in an STS program! [1]: https://www.space.commerce.gov/business-with-noaa/commercial-weather-data-pilot-cwdp/ https://www.space.commerce.gov/business-with-noaa/commercial...

jnurmine 2y ago

For the data, do you know meteostat.net? They get input from e.g. NOAA (US), DWD (DE), YR (NO) and so on. I'm not affiliated, just needed some historical data and (finally!) discovered them.

open-meteo 2y ago

Hi, I am building an Open-Source Weather API aggregating open-data weather models from NOAA, ECMWF, DWD, MeteoFrance, JMA, CMA, CMCC and others. I agree that many weather companies basically redistribute NWS data at a premium. There is a free API service available on https://open-meteo.com https://open-meteo.com and all databases are redistributed via an AWS Open-Data Sponsorship. Feel free to reach out if you need help building your weather data broker startup

seabass-labrax 2y ago
A sincere personal "thank you" from me! I use Breezy Weather[1] on my smartphone, which gets most of its data from your aggregating service, Open-Meteo. It's the perfect combination of accurate, international and open. [1]: https://f-droid.org/en/packages/org.breezyweather/ https://f-droid.org/en/packages/org.breezyweather/
just_testing 2y ago
First of all, thank you! What a treasure trove of data. If I may ask a question, do you have historical air quality data?
- open-meteo 2y ago
  You can access historical air quality data from August 2022 onwards: https://open-meteo.com/en/docs/air-quality-api#start_date=2022-07-29&end_date=2024-06-09&time_mode=time_interval https://open-meteo.com/en/docs/air-quality-api#start_date=20.... Data is based on the Copernicus air quality forecast. All are references listed the documentation
  - just_testing 2y ago
    Thank you!
CraigGilmore 2y ago
I am working on this kind of solution - please email - see about

kmbfjr 2y ago

I became so annoyed at the tracking, advertising and poor notifications of many apps, I wrote my own for severe weather alerts. My hobby will include a C band antenna this fall, and I’m on the hunt for radar data sources in which to create my own mosaics.

ptero 2y ago

Thank you for doing this! As my own sidebar, I spent many years at a national lab working with distributed sensor networks (primarily ATC and other radar for detecting non-weather stuff :) ). I thought about using ADS-B as input for weather state and forecasts, but never got around to trying it. Now that I am working on my own startup (self-funded and without revenue so far), so this again will likely languish in my todo list. If someone wants to try it, great, and feel free to reach out as I can probably save you some time selecting and interpreting the right ADS-B fields: We have a lot of aircraft blasting ADS-B reports whenever they fly. Most reports contain (1) accurate 3D position, including altitude and (2) barometric altitude measurements, which gives you (after some minimal work) air pressure. So you have millions freely available pressure reports not just on the ground, but throughout 0-40000 ft altitude band. You also get measured airspeed and groundspeed, so in addition to pressure you get wind vectors at thousands of points in the air, updating in real time. I suspect this can provide some non-trivial information and I am not aware of anyone actually using it for this purpose.

counters 2y ago
These data are already consumed operationally by the major global weather modeling centers - e.g., check out the AMDAR [1] or ACARS programs. There are commercial agreements which restrict third-party access to these data in real-time, but they are widely disseminate with a 24-48 hour lag in the observational archives that NOAA curates for weather modeling. These data are a very important input for operational weather forecasting. On a global basis, we've seen how losing these data during the pandemic due to reductions in air travel decreased forecast model skill [2]. Furthermore, ACARS profiles derived from aircraft landing at airports where severe weather is expected are regularly used to complement SPECI weather balloon launches. [1]: https://community.wmo.int/en/activity-areas/aircraft-based-observations/amdar/amdar-description https://community.wmo.int/en/activity-areas/aircraft-based-o... [2]: https://journals.ametsoc.org/view/journals/apme/59/11/JAMC-D-20-0010.1.xml https://journals.ametsoc.org/view/journals/apme/59/11/JAMC-D...
- ptero 2y ago
  Thank you for the info, I am glad to hear that the data is being collected and used. Getting it from raw ADS-B reports makes all fields available without the lag (and thus more useful for weather reporting and forecasting), but the recorded data is definitely good to have!

fallingknife 2y ago

Why is this unfair? With these private companies I get access to all this data for free. If the government did it I would have to pay for it out of my taxes. Why do we need the government to provide more accurate forecasts when this is already taken care of by the private sector?

chiph 2y ago
> With these private companies I get access to all this data for free. These companies are already receiving government data for free. And then enriching it with private weather station data (and ads). So your taxes are already funding some part of it. As computing power being used for forecasting increases (14.5 petaflops each for both Dogwood and Cactus NOAA supercomputers), having more granular data is going to be useful in improving their models. https://www.noaa.gov/news-release/noaa-completes-upgrade-to-weather-and-climate-supercomputer-system https://www.noaa.gov/news-release/noaa-completes-upgrade-to-...
desert_rue 2y ago
It is in the public interest to have both weather and forecasts. Maybe you are like me and look out the window to see the weather for that day, but millions need accurate weather and forecasts. They include farmers, those that run power grids, air traffic controllers, and more.
Workaccount2 2y ago
You get that data for "free" or your hand over your browser data - and likely location data, to unknown third party brokers that know more about you than you know about yourself. Weather apps in particular are notorious for selling your location data, since everyone does "show weather for my current location". It's "free" in the worst sense of the word.
- radicality 2y ago
  Do you know much about Windy, and whether they have any ties to selling user location data? All the plots and features seemed nice so I paid for a year of subscription (after Dark Sky was discontinued ugh), but if they do that I’ll just stick to the default apple Weather app.

counters 2y ago

> NOAA/NWS, for example, is extremely underfunded so if they had to privilege to buy it they probably couldn't come to an agreement to buy it. As a result, they can't use that data to improve the accuracy of alerts/warnings/forcasts, the same exact tools that the big weather companies make all their money from. It's a shit cycle and totally unfair IMO. Huh? This is kind of an odd take for a few reasons. For starters, NOAA isn't "extremely underfunded"; with the possible exception of the current budgeting cycle, NOAA generally does pretty well and has strong bipartisan support. It could always use more money, but I wouldn't call it "underfunded. The reason NOAA doesn't buy more data is because most of the available data has limited value. Personal weather stations have substantial quality issues and add almost no value in areas where we already have high-quality surface observations. We thin out and throw away a ton of surface observations already during the data assimilation process to initialize our forecast models anyways - data from aloft is far more valuable and impactful from a forecast impact perspective. For what it's worth, few if any companies use proprietary observations to improve their forecasts. It's an open secret that the vast majority of companies out there are just applying proprietary statistical modeling / bias correction on top of publicly available data. Only a handful of companies actually have novel observations, and there's limited evidence it makes a significant difference in the forecast. At best, it can result in the way that those statistical corrections are applied to existing forecasts and ensembles - you can count on one hand the number of companies that actually run a vertically-integrated stack including data assimilation of proprietary observations and end-to-end numerical modeling. That isn't to say there isn't unique value in the observations. It's just that the industry flagrantly misleads about how they use them.

pinkmuffinere 2y ago
Referring to the parent comment, the data which NOAA isn’t able to buy is the government’s data, which is freely provided to non-government organizations. The parent comment doesn’t discuss personal data very much. I think this misunderstanding might be the root cause of your disagreement.
- counters 2y ago
  Parent comment is mis-informed. NOAA doesn't have to buy any data from other government agencies or organizations. It's all open and publicly available. There are challenges around the reliability and quality of data that NOAA doesn't take efforts itself to curate and maintain which limit their utility, but that's a separate issue. More importantly, NOAA explicitly funds the National Mesonet Program [1] to actively identify, acquire, consume, and ingest data from a wide variety of state and federal agencies across the country. The NMP itself partners with a major private sector company, Synoptic Data PBC [2] to perform the engineering necessary to acquire all this data. Synoptic actively maintains the infrastructure which consumes and publishes this data to MADIS for use by NOAA and any other stakeholder. [1]: https://nationalmesonet.us/ https://nationalmesonet.us/ [2]: https://synopticdata.com/ https://synopticdata.com/

mozman 2y ago

what’s good weather station hw to get? thinking ultrasonic wind instrument and some type of radar

FireBeyond 2y ago
Doesn't encompass radar (and no home weather station will) but NetAtmo sells one with ultrasonic wind, smart rain gauge, and has a full RESTful API available: https://www.netatmo.com/smart-weather-station https://www.netatmo.com/smart-weather-station I use this and love it.
- VENTS 2y ago
  You can watch it here: https://lcjcapteurs.com/en/girouette-anemometres-capteur-vent/ultrasonic-anemometer-cv7-hr-oem/ https://lcjcapteurs.com/en/girouette-anemometres-capteur-ven... or https://www.davisinstruments.com/products/sonic-anemometer https://www.davisinstruments.com/products/sonic-anemometer

HumblyTossed 2y ago

Don't forget, AccuWeather wanted to make it illegal for you to have access to that data for free from the government.

widerporst 2y ago
Same thing in Germany, with WetterOnline ruminating data from DWD (the German meteorological service) and then suing them when they offered their DWD WarnWetter app for free. Unfortunately, they were successful: https://www.heise.de/news/BGH-Urteil-Staatlicher-Wetterdienst-muss-Gratis-App-beschraenken-4681828.html https://www.heise.de/news/BGH-Urteil-Staatlicher-Wetterdiens... (Couldn't find an English source, sorry.)
- Rinzler89 2y ago
  >Unfortunately, they were successful Unbelievable.
Wistar 2y ago
Alas, there are yet ongoing attempts to break-up and/or privatize the NWS. Project 2025 specifically calls for the break-up of NOAA.
- HumblyTossed 2y ago
  Oh for crying out loud! WTF??? I think someone spilled idiot juice in the water.
  - wormius 2y ago
    Swamp Water Kool-Aid for the cultists.

supportengineer 2y ago

What is the best option for someone who wants to contribute but doesn’t have a weather station currently or a lot of free time to take on another project?

alexpotato 2y ago

"The governments of the world provides these big weather companies (weather.com (cough IBM), Accuweather (cough IBM cough), etc) a metric shit ton of their data completely for free (by law) including data transfer. These are things like radar, satellite, ground station data, forecasts, composite models, etc. These companies profit substantially on it, as in billions of dollars" Michael Lewis' book The Fifth Risk goes IN DEPTH into how Accuweather/Weather.com and the government were interacting (particularly during the Trump administration). I highly recommend the book in general and for this particular story in particular.

just_testing 2y ago

If I may use your expertise a bit: I need air quality data for two cities, one of which doesn't have any air quality monitoring stations. Is there anything I can use as a proxy for air pollution, like satellite data? Also, do you know any sources for that data? Thanks! P.S.: I'm passionate about air quality, given I have alergies and was an active member of the bicycle activism movement for about 10 years.

toomuchtodo 2y ago
Might find useful info here: https://www.nasa.gov/missions/tempo/nasa-releases-new-high-quality-near-real-time-air-quality-data/ https://www.nasa.gov/missions/tempo/nasa-releases-new-high-q... https://news.ycombinator.com/item?id=40530798 https://news.ycombinator.com/item?id=40530798 https://news.ycombinator.com/item?id=40534632 https://news.ycombinator.com/item?id=40534632
- just_testing 2y ago
  Thank you!!

CraigGilmore 2y ago

Hi Mike- I am very interested in this and doing some work - please email