15 ms·

GraphCast: AI model for weather forecasting

• haolez 3y ago
  Are there any experts around that can chime in on the possible impacts of this technology if widely adopted?
  • supdudesupdude 3y ago
    It doesnt predict rainfall so i doubt most of us will actually care about it until then. Still it depends on input data (the current state of weather etc). How are we supposed to accurately model the weather at every point in the world? Especially when tech bro Joe living in San Fran expects things to be accurate to a meter within his doorstep
    • counters 3y ago
      GraphCast does predict rainfall - see https://charts.ecmwf.int/products/graphcast_medium-rain-acc?base_time=202311141200&projection=opencharts_europe&valid_time=202311141800 https://charts.ecmwf.int/products/graphcast_medium-rain-acc?... for example.
  • _visgean 3y ago
    It will get adopted, eventually we will have more accurate weather forecasts. Thats good for anything that depends on weather - e.g. energy consumption and production, transportation costs...
• miserableuse 3y ago
  Does anybody know if its possible to initialize the model using GFS initial conditions used for the GFS HRES model? If so, where can I find this file and how can I use it? Any help would be greatly appreciated!
  • counters 3y ago
    You can try, but other models in this class have struggled when initialized using model states pulled from other analysis systems. ECMWF publishes a tool that can help bootstrap simple inference runs with different AI models [1] (they have plugins for several). You could write a tool that re-maps a GDAS analysis to "look like" ERA-5 or IFS analysis, and then try feeding it into GraphCast. But YMMV if the integration is stable or not - models like PanguWx do not work off-the-shelf with this approach. [1]: https://github.com/ecmwf-lab/ai-models https://github.com/ecmwf-lab/ai-models
    • miserableuse 3y ago
      Thank you for your response. Are these ML models initialized by gridded initial conditions measurements (such as the GDAS pointed out) or by NWP model forecast results (such as hour-zero forecast from the GFS)? Or are those one and the same?
      • counters 3y ago
        They're more-or-less the same thing.
• Gys 3y ago
  I live in an area which regularly has a climate differently then forecasted: often less rain and more sunny. Would be great if I can connect my local weather station (and/or its history) to some model and have more accurate forecasts.
  • speps 3y ago
    Because weather data is interpolated between multiple stations, you wouldn't even need the local station position, your own position would be more accurate as it'd take a lot more parameters into account.
  • tash9 3y ago
    One piece of context to note here is that models like ECMWF are used by forecasters as a tool to make predictions - they aren't taken as gospel, just another input. The global models tend to consistently miss in places that have local weather "quirks" - which is why local forecasters tend to do better than, say, accuweather, where it just posts what the models say. Local forecasters might have learned over time that, in early Autumn, the models tend to overpredict rain, and so when they give their forecasts, they'll tweak the predictions based on the model tendencies.
  • dist-epoch 3y ago
    There are models which take as input both global forecasts and local ones, and which then can transpose a global forecast into a local one. National weather institutions sometimes do this, since they don't have the resources to run a massive supercomputer model.
    • Gys 3y ago
      Interesting. So what I am looking for is probably an even more scaled down version? Or something that runs in the cloud with an api to upload my local measurements.
      • supdudesupdude 3y ago
        Hate to break it but one weather station wont improve a forecast? What are they supposed to do? Ignore the output of our state of the art forecast models and add an if statement for your specific weather station??
• deleted 3y ago
  [deleted]
• hackitup7 3y ago
  I've been really impressed at how much better weather forecasting has become already. I remember weather forecasts feeling like a total crapshoot as recently as 15 years ago or so.
  • patall 3y ago
    Isn't that highly subjective to where you live? Because I moved to Scandinavia and the forecast here is so incredibly bad, compared to central europe.
    • tomesco 3y ago
      Yes, driven by local data collection. More tightly packed ground stations and the availability of atmospheric measurement at various altitudes will improve accuracy.
      • mike-cardwell 3y ago
        Also, the weather is just a lot more predictable in some areas than others.
        • londons_explore 3y ago
          I think it's mostly this. If you look at a weather radar map, sometimes you see a speckled pattern of rain where there is heavy rain in places, and 100 yards away there is no rain at all. No way you can predict that multiple days out.
        • obscurette 3y ago
          This. Just some days ago I had a conversation with meteorologist who said exactly this - the weather has never been easy to predict in northen Europe and it has become even less predictable with climate change and global warming.
        • Kye 3y ago
          I feel this living in the path of moisture coming from the Gulf of Mexico. My phone has gotten good at letting me know when the rain will start and stop to within a few minutes, but whatever data source Apple uses still struggles with near-term prediction (day+) in the summer when there are random popup storms all the time.
• animous 3y ago
  [flagged]
• xnx 3y ago
  I continue to be a little confused by the distinction between Google, Google Research and DeepMind. Google Research, had made this announcement about 24-hour forecasting just 2 weeks ago: https://blog.research.google/2023/11/metnet-3-state-of-art-neural-weather.html https://blog.research.google/2023/11/metnet-3-state-of-art-n... (which is also mentioned in the GraphCast announcement from today)
  • mukara 3y ago
    DeepMind recently merged with the Brain team from Google Research to form `Google DeepMind`. It seems this was done to have Google DeepMind focused primarily (only?) on AI research, leaving Google Research to work on other things in more than 20 research areas. Still, some AI research involves both orgs, including MetNet in weather forecasting. In any case, GraphCast is a 10-day global model, whereas MetNet is a 24-hour regional model, among other differences.
    • xnx 3y ago
      Good explanation. Now that both the 24-hour regional and 10-day global models have been announced in technical/research detail, I supposed there might still be a general blog post about how improved forecasting is when you search for "weather" or check the forecast on Android.
      • mnky9800n 3y ago
        That would require your local weather service to use these models
      • kridsdale3 3y ago
        IIRC the MetNet announcement a few weeks ago said that their model is now used when you literally Google your local weather. I don't think it's available yet to any API that third party weather apps pull from, so you'll have to keep searching "weather in Seattle" to see it.
        • 3y ago
• robertlagrant 3y ago
  This is fascinating: > For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
  • broast 3y ago
    Weather is markovian
    • deleted 3y ago
      [deleted]
    • hakuseki 3y ago
      That is not strictly true. The weather at time t0 may affect non-weather phenomena at time t1 (e.g. traffic), which in turn may affect weather at time t2. Furthermore, a predictive model is not working with a complete picture of the weather, but rather some limited-resolution measurements. So, even ignoring non-weather, there may be local weather phenomena detected at time t0, escaping detection at time t1, but still affecting weather at time t2.
  • Imanari 3y ago
    Interesting indeed, only one lagged feature for time series forecasting? I’d imagine that including more lagged inputs would increase performance. Rolling the forecasts forward to get n-step-ahead forecasts is a common approach. I’d be interested in how they mitigated the problem of the errors accumulating/compounding.
  • Al-Khwarizmi 3y ago
    I don't know much about weather prediction, but if a model can improve the state of the art only with that data as input, my conclusion is that previous models were crap... or am I missing something?
    • 3y ago
• lispisok 3y ago
  I've been following these global ML weather models. The fact they make good forecasts at all was very impressive. What is blowing my mind is how fast they run. It takes hours on giant super computers for numerical weather prediction models to forecast the entire globe. These ML models are taking minutes or seconds. This is potentially huge for operational forecasting. Weather forecasting has been moving focus towards ensembles to account for uncertainty in forecasts. I see a future of large ensembles of ML models being ran hourly incorporating the latest measurements
  • wenc 3y ago
    Not to take away from the excitement but ML weather prediction builds upon the years of data produced by numerical models on supercomputers. It cannot do anything without that computation and its forecasts are dependent on the quality of that computation. Ensemble models are already used to quantify uncertainty (it’s referenced in their paper). But it is exciting that they are able to recognize patterns in multi year and produce medium term forecasts. Some comments here suggest this replaces supercomputers models. This would a wrong conclusion.It does not (the paper explicitly states this). It uses their output as input data.
    • boxed 3y ago
      I don't get this. Surely past and real weather should be the input training data, not the output of numerical modeling?
      • counters 3y ago
        Well, what is "real weather data?" We have dozens of complementary and contradictory sources of weather information. Different types of satellites measuring EM radiation in different bands, weather stations, terrestrial weather radars, buoys, weather balloons... it's a massive hodge-podge of different systems measuring different things in an uncoordinated fashion. Today, it's not really practical to assemble that data and directly feed it into an AI system. So the state-of-the-art in AI weather forecasting involves using an intermediate representation - "reanalysis" datasets which apply a sophisticated physics based weather model to assimilate all of these data sets into a single, self-consistent 3D and time-varying record of the state of the atmosphere. This data is the unsung hero of the weather revolution - just as the WMO's coordinated synoptic time observations for weather balloons catalyzed effective early numerical weather prediction in the 50's and 60's, accessible re-analysis data - and the computational tools and platforms to actually work with these peta-scale datasets - has catalyzed the advent of "pure AI" weather forecasting systems.
• freedomben 3y ago
  weather prediction seems to me like a terrific use of machine learning aka statistics. The challenge I suppose is in the data. To get perfect predictions you'd need to have a mapping of what conditions were like 6 hours, 12 hours, etc before, and what the various outcomes were, which butterflies flapped their wings and where (this last one is a joke about how hard this data would be). Hard but not impossible. Maybe impossible. I know very little about weather data though. Is there already such a format?
  • tash9 3y ago
    It's been a while since I was a grad student but I think the raw station/radiosonde data is interpolated into a grid format before it's put into the standard models.
    • kridsdale3 3y ago
      This was also in the article. It splits the sphere surface in to 1M grids (not actually grids in the cartesian sense of a plane, these are radial units). Then there's 37 altitude layers. So there's radial-coordinate voxels that represent a low resolution of the physical state of the entire atmosphere.
• serjester 3y ago
  To call this impressive is an understatement. Using a single GPU, outperforms models that run on the world's largest super computers. Completely open sourced - not just model weights. And fairly simple training / input data. > ... with the current version being the largest we can practically fit under current engineering constraints, but which have potential to scale much further in the future with greater compute resources and higher resolution data. I can't wait to see how far other people take this.
  • thatguysaguy 3y ago
    They said single TPU machine to be fair, which means like 8 TPUs (still impressive)
  • wenc 3y ago
    It builds on top of supercomputer model output and does better at the specific task of medium term forecasts. It is a kind of iterative refinement on the data that supercomputers produce — it doesn’t supplant supercomputers. In fact the paper calls out that it has a hard dependency on the output produced by supercomputers.
    • carbocation 3y ago
      I don't understand why this is downvoted. This is a classic thing to do with deep learning: take something that has a solution that is expensive to compute, and then train a deep learning model from that. And along the way, your model might yield improvements, too, and you can layer in additional features, interpolate at finer-grained resolution, etc. If nothing else, the forward pass in a deep learning model is almost certainly way faster than simulating the next step in a numerical simulation, but there is room for improvement as they show here. Doesn't invalidate the input data!
      • danielmarkbruce 3y ago
        Because "iterative refinement" is sort of wrong. It's not a refinement and it's not iterative. It's an entirely different model to physical simulation which works entirely differently and the speed up is order of magnitude. Building a statistical model to approximate a physical process isn't a new idea for sure.. there are literally dozens of them for weather.. the idea itself isn't really even iterative, it's the same idea... but it's all in the execution. If you built a model to predict stock prices tomorrow and it generated 1000% pa, it wouldn't be reasonable for me to call it iterative.
• meteo-jeff 3y ago
  In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-meteo.com https://open-meteo.com
  • boxed 3y ago
    Open-Meteo has a great API too. I used it to build my iOS weather app Frej (open source and free: https://github.com/boxed/frej https://github.com/boxed/frej) It was super easy and the responses are very fast.
  • mdbmdb 3y ago
    Is it able to provide data on extreme events. Say, the current and potential path of a hurricane? similar to .kml that NOAA provides
    • meteo-jeff 3y ago
      Extreme weather is predicted by numerical weather models. Correctly representing hurricanes has driven development on the NOAA GFS model for centuries. Open-Meteo focuses on providing access to weather data for single locations or small areas. If you look at data for coastal areas, forecast and past weather data will show severe winds. Storm tracks or maps are not available, but might be implemented in the future.
      • mdbmdb 3y ago
        Appreciate the response. Do you know of any services that provide what I described in the previous comments? I'm specifically interested in extreme weather conditions and their visual representation (hurricanes, tornados, hails etc.) with API capabilities
        • 3y ago
• amluto 3y ago
  I've never studied weather forecasting, but I can't say I'm surprised. All of these models, AFAICT, are based on the "state" of the weather, but "state" deserves massive scare quotes: it's a bunch of 2D fields (wind speed, pressure, etc) -- note the 2D. Actual weather dynamics happen in three dimensions, and three dimensional land features, buildings, etc as well as gnarly 2D surface phenomena (ocean surface temperature, ground surface temperature, etc) surely have strong effects. On top of this, surely the actual observations that feed into the model are terrible -- they come from weather stations, sounding rockets, balloons, radar, etc, none of which seem likely to be especially accurate in all locations. Except that, where a weather station exists, the output of that station is the observation that people care about -- unless you're in an airplane, you don't personally care about the geopotential, but you do care about how windy it is, what the temperature and humidity are, and how much precipitation there is. ISTM these dynamics ought to be better captured by learning them from actual observations than from trying to map physics both ways onto the rather limited datasets that are available. And a trained model could also learn about the idiosyncrasies of the observation and the extra bits of forcing (buildings, etc) that simply are not captured by the inputs. (Heck, my personal in-my-head neural network can learn a mapping from NWS forecasts to NWS observations later in the same day that seems better than what the NWS itself produces. Surely someone could train a very simple model that takes NWS forecasts as inputs and produces its estimates of NWS observations during the forecast period as outputs, thus handling things like "the NWS consistently underestimates the daily high temperature at such-and-such location during a summer heat wave.")
  • Difwif 3y ago
    I'm not sure why you're emphasizing that weather forecasting is just 2D fields. Even in the article they mention GraphCast predicts multiple data points at each global location across a variety of altitudes. All existing global computational forecast models work the same way. They're all 3d spherical coordinate systems.
    • amluto 3y ago
      See page three, table 1 of the paper. The model has 48 2D fields, on a grid, where the grid is a spherical thing wrapped around the surface of the Earth. There is not what I would call a 3D spherical coordinate system. There’s no field f defined as f(theta, phi, r) — ther are 48 fields that are functions of theta and phi.
• cryptoz 3y ago
  Again haha! Still no mention of using barometers in phones. Maybe some day.
  • EricLeer 3y ago
    The weather company claims to do this (they are also the main provider of weather data for apple).
• user_7832 3y ago
  (If someone with knowledge or experience can chime in, please feel free.) To the best of my knowledge, poor weather (especially wind shear/microbursts) are one of the most dangerous things possible in aviation. Is there any chance, or plans, to implement this in the current weather radars in planes?
  • tash9 3y ago
    If you're talking about small scale phenomena (less than 1km), then this wouldn't help other than to be able to signal when the conditions are such that these phenomena are more likely to happen.
• jauntywundrkind 3y ago
  From what I can tell from reading & based off https://colab.research.google.com/github/deepmind/graphcast/blob/master/graphcast_demo.ipynb#scrollTo=rQWk0RRuCjDN https://colab.research.google.com/github/deepmind/graphcast/... , one needs access to ECMWF Era5 or HRES data-sets or something similar to be able to run and use this model. Unknown what licensing options ECMWF offers for Era5, but to use this model in any live fashion, I think one is probably going to need a small fortune. Maybe some other dataset can be adapted (likely at great pain)...
  • sunshinesnacks 3y ago
    ERA5 is free. The API is a bit slow. I think that only some variables from the HRES are free, but not 100% sure.
    • hokkos 3y ago
      The API is unusably slow, the only way is to use the AWS, GCP or Azure mirrors, but they miss a lot of variables and are updated sparingly or with a delay.
    • jauntywundrkind 3y ago
      I created an account on ECMWF but I still dont have access to the ERA5 page, just a big permissions denied message. :/ Any pointers?
  • _visgean 3y ago
    You can get some of the historical data also from here: https://cloud.google.com/storage/docs/public-datasets/era5 https://cloud.google.com/storage/docs/public-datasets/era5 (if the official API is too slow. ) To use the data in live fashion I think you would need to get license from ECMWF...
• deleted 3y ago
  [deleted]
• sagarpatil 3y ago
  How does one go about hosting this and using this as an API?
• syntaxing 3y ago
  Maybe I missed it but does anyone know what it will take to run this model? Seems something fun to try out but not sure if 24GB of VRAM is suffice.
  • kridsdale3 3y ago
    It says in the article that it runs on Google's tensor units. So, go down to your nearest Google data center, dodge security, and grab one. Then escape the cops.
    • azeirah 3y ago
      You could also just buy a very large amount of their coral consumer TPUs :D
• comment_ran 3y ago
  So for a daily user, to make it a practical usage, let's say if I have a local measurement of X, I can predict, let's say, 10 days later, or even just tomorrow, or the day after tomorrow, let's say the wind direction, is it possible to do that? If it is possible, then I will try using the sensor to measure my velocity at some place where I live, and I can run the model and see how the results look like. I don't know if it's going to accurately predict the future or within a 10% error bar range.
  • dist-epoch 3y ago
    No, this model uses as input the current state of the weather across the whole planet.
• carabiner 3y ago
  > GraphCast makes forecasts at the high resolution of 0.25 degrees longitude/latitude (28km x 28km at the equator). Any way to run this at even higher resolution, like 1 km? Could this resolve terrain forced effects like lenticular clouds on mountain tops?
  • dist-epoch 3y ago
    One big problem is input weather data. It's resolution is poor.
    • carabiner 3y ago
      Yeah, not to mention trying to validate results. Unless we grid install weather stations every 200 m on a mountain top...
• max_ 3y ago
  I have far more respect for the AI team at DeepMind even thou they may be less popular than say OpenAI or "Grok". Why? Other AI studios seem to work on gimmicks while DeepMind seems to work on genuinely useful AI applications [0]. Thanks for the good work! [0] Not to say that Chat GPT & Midjourney are not useful, I just find DeepMinds quality of research more interesting.
  • deleted 3y ago
    [deleted]
• max_ 3y ago
  Has anyone here heard of "Numerical Forecasting" models for weather? I heard they "work so well". Does GraphCast come close to them?
• max_ 3y ago
  What's the difference between a "Graph Neural Network" and a deep neural network?
  • dil8 3y ago
    Graph neural networks are deep learning models that trained on graph data.
    • RandomWorker 3y ago
      Do you have any resources where I could learn more about these networks?
      • EricLeer 3y ago
        See for instance the pytorch geometric [1] package, which is the main implementation in pytorch. They also link to some papers there that might explain you more. [1] https://pytorch-geometric.readthedocs.io/en/latest/ https://pytorch-geometric.readthedocs.io/en/latest/
• deleted 3y ago
  [deleted]
• pyb 3y ago
  Curious. How can AI/ML perform on a problem that is, as far as I understand, inherently chaotic / unpredictable ? It sounds like a fundamental contradiction to me.
  • kouru225 3y ago
    Yes. Very accurate as long as you don’t need to predict the unpredictable. So it’s useless. Edit: I do see a benefit to the idea if you compare it to the Chaos Theorists “gaining intuition” about systems.
    • pyb 3y ago
      IDK if it's useless, but it's counter-intuitive to me.
  • vosper 3y ago
    Weather isn’t fundamentally unpredictable. We predict weather with a fairly high degree of accuracy (for most practical uses), and the accuracy getting better all the time. https://scijinks.gov/forecast-reliability https://scijinks.gov/forecast-reliability
    • sosodev 3y ago
      I'm kinda surprised that this government science website doesn't seem to link sources. I'd like to read the research to understand how they're measuring the accuracy.
  • keule 3y ago
    IMO a chaotic system will not allow for long-term forecast, but if there is any type of pattern to recognize (and I would assume there are plenty), an AI/ML model should be able to create short-term prediction with high accuracy.
    • pyb 3y ago
      Not an expert, but "Up to 10 days in advance" sounds like long-term to me ?
• simonebrunozzi 3y ago
  Amazing. Is there an easy way to run this on a local laptop?
• dnlkwk 3y ago
  Curious how this factors in long-range shifts or patterns eg el nino. Most accurate is a bold claim
• stabbles 3y ago
  If you live in a country where local, short-term rain / shower forecast is essential (like [1] [2]), it's funny to see how incredibly bad radar forecast is. There are really convenient apps that show an animated map with radar data of rain, historical data + prediction (typically). The prediction is always completely bonkers. You can eyeball it better. No wonder "AI" can improve that. Even linear extrapolation is better. Yes, local rain prediction is a different thing from global forecasting. [1] https://www.buienradar.nl https://www.buienradar.nl [2] https://www.meteoschweiz.admin.ch/service-und-publikationen/applikationen/niederschlag.html https://www.meteoschweiz.admin.ch/service-und-publikationen/...
  • bberenberg 3y ago
    Interesting that you say this. I spent in month in AMS 7-8 years ago and buienradar was accurate down to the minute when I used it. Has something changed?
    • bobviolier 3y ago
      I don't know how or why, but yes, it has become less accurate over at least the last year or so.
  • supdudesupdude 3y ago
    Funny to mention. None of the AI forecasts can actually predict precip. None of them mention this and i assume everyone thinks this means the rain forecasts are better. Nope just temperature and humidity and wind. Important but come on, it's a bunch of shite
  • je42 3y ago
    However, tools like buienrader seem to have trouble in the recent months/years to accurately predict local weather.
• brap 3y ago
  Beyond the difficulty of running calculations (or even accurately measuring the current state), is there a reason to believe weather is unpredictable? I would imagine we probably have a solid mathematical model of how weather behaves, so given enough resources to measure and calculate, could you, in theory, predict the daily weather going 10 years into the future? Or is there something inherently “random” there?
  • ethanbond 3y ago
    AFAIK there's nothing random anywhere except near atomic/subatomic scale. Everything else is just highly chaotic/hard-to-forecast deterministic causal chains.
    • mesoman 3y ago
      Cloud formation is affected by cosmic ray flux. It's effectively random. But the real problem is chaos - which says that even with perfect data, unless you also have computations with infinite precision and time/spatial/temperature/pressure/etc resolution, eventually you wind up far from reality. The use of ensembles reduces the effect of chaos a bit, although they tend to smooth it out - so your broad pattern 12 days out might be more accurately forecast than without them, but the weather at your house may not be. Iterative DL models tend to smooth it faster, according to a recent paper.
  • danbrooks 3y ago
    Small changes in initial state can lead to huge changes down the line. See: the butterfly effect or chaos theory. https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory
  • deleted 3y ago
    [deleted]
  • counters 3y ago
    What you're describing is effectively how climate models work; we run a physical model which solves the equations that govern how the atmosphere works out forward in time for very long time integrations. You get "daily weather" out as far as you choose to run the model. But this isn't a "weather forecast." Weather forecasting is an initial value problem - you care a great deal about how the weather will evolve from the current atmospheric conditions. Precisely because weather is a result of what happens in this complex, 3D fluid atmosphere surrounding the Earth, it happens that small changes in those initial conditions can have a very big impact on the forecast on relatively short time-periods - as little as 6-12 hours. Small perturbations grow into larger ones and feedback across spatial scales. Ultimately, by day ~3-7, you wind up with a very different atmospheric state than what you'd have if you undid those small changes in the initial conditions. This is the essence of what "chaos" means in the context of weather prediction; we can't perfectly know the initial conditions we feed into the model, so over some relatively short time, the "model world" will start to look very different than the "real world." Even if we had perfect models - capable of representing all the physics in the atmosphere - we'd still have this issue as long as we had to imperfectly sample the atmosphere for our initial conditions. So weather isn't inherently "unpredictable." And in fact, by running lots of weather models simultaneously with slightly perturbed initial conditions, we can suss out this uncertainty and improve our estimate of the forecast weather. In fact, this is what's so exciting to meteorologists about the new AI models - they're so much cheaper to run that we can much more effectively explore this uncertainty in initial conditions, which will indirectly lead to improved forecasts.
• Vagantem 3y ago
  Related to this, I built a service that shows what day it has rained the least on in the last 10 years - for any location and month! Perfect to find your perfect wedding date. Feel free to check out :) https://dropory.com https://dropory.com
  • helloplanets 3y ago
    Was interested to check this out for Helsinki, but site loads blank on Safari :(
    • Vagantem 3y ago
      Oh, yea spotted now - I’ll have a look as soon as I’m at my computer, will fix. Until then, I think you’ll have to use it on a desktop - thanks for spotting!
• supdudesupdude 3y ago
  I'll be impressed when it can predict rainfall better than GFS / HRRR / EURO etc
• knicholes 3y ago
  What are the similarities between weather forecasting and financial market forecasting?
  • KRAKRISMOTT 3y ago
    Both are complex systems traditionally modeled with differential equations and statistics.
  • sonya-ai 3y ago
    Well it's a start, but weather forecasting is far more predictable imo
• csours 3y ago
  Makes me wonder how much it would take to do this for a city at something like 100 meter resolution.
• layoric 3y ago
  I can't see any citation to accuracy comparisons, or maybe I just missed them? Given the amount of data, and complexity of the domain, it would be good to see a much more detailed breakdown of their performance vs other models. My experience in this space is that I was first employee at Solcast building a live 'nowcast' system for 4+ years (left ~2021) targeting solar radiation and cloud opacity initially, but expanding into all aspects of weather, focusing on the use of the newer generation of satellites, but also heavily using NWP models like ECMWF. Last I knew,nowcasts were made in minutes on a decent size cluster of systems, and has been shown in various studies and comparisons to produce extremely accurate data (This article claims 'the best' without links which is weird..), be interesting on how many TPUsv4 were used to produce these forecasts and how quickly? Solcast used ML as a part of their systems, but when it comes down to it, there is a lot more operationally to producing accurate and reliable forecasts, eg it would be arrogant to say the least to switch from something like ECMWF to this black box anytime soon. Something I said as just before I left Solcast was that their biggest competition would come from Amazon/Google/Microsoft and not other incumbent weather companies. They have some really smart modelers, but its hard to compete with big tech resources. I believe Amazon has been acquiring power usage IoT related companies over the past few years, I can see AI heavily moving into that space as well.. for better or worse.
  • shmageggy 3y ago
    I think the paper has what you are looking for. Several figures comparing performance to HRES, and "GraphCast... took roughly four weeks on 32 Cloud TPU v4 devices using batch parallelism. See supplementary materials section 4 for further training details."
  • alxmrs 3y ago
    I’m so happy you asked about this! Check out https://sites.research.google/weatherbench/ https://sites.research.google/weatherbench/
• crazygringo 3y ago
  Making progress on weather forecasting is amazing, and it's been interesting to see the big tech companies get into this space. Apple moved from using The Weather Channel to their own forecasting a year ago [1]. Using AI to produce better weather forecasts is exactly the kind of thing that is right up Google's alley -- I'm very happy to see this, and can't wait for this to get built into our weather apps. [1] https://en.wikipedia.org/wiki/Weather_(Apple) https://en.wikipedia.org/wiki/Weather_(Apple)
  • blacksmith_tb 3y ago
    Well, Apple acquired Dark Sky and then shut it down for Android users[1], and then eventually for iOS users as well (but rolled it into the built in weather app, I think). 1: https://www.theverge.com/2020/3/31/21201666/apple-acquires-weather-app-dark-sky-shut-down-android-wear-os-ios https://www.theverge.com/2020/3/31/21201666/apple-acquires-w...
  • _visgean 3y ago
    > Apple moved from using The Weather Channel to their own forecasting a year ago [1]. AFAIK they don't have their own forecasting models, they use same data sources as everyone else: https://support.apple.com/en-us/HT211777 https://support.apple.com/en-us/HT211777
    • crazygringo 3y ago
      Your linked article says they use their own, if you're on a version later than iOS 15.2.
      • _visgean 3y ago
        No it does not. Read the secion "Data sources", they list all the usual regional providers.
• joegibbs 3y ago
  When will we have enough data that we will be able to apply this to everything? Imagine a model that can predict all kinds of trends - what new consumer good will be the most likely to succeed, where the next war is most likely to break out, who will win the next election, which stocks are going to break out. One gigantic black box with a massive state, with input from everything - planning approvals, social media posts, solar activity, air travel numbers, seismic readings, TV feeds.
  • drakenot 3y ago
    Sounds a bit like the premise for the Asimov series, "The Foundation"
• hammad93 3y ago
  I think it's irresponsible to call first on this because it will hinder scientific collaboration. I appreciate this contribution but the journalism was sloppy.
• whoislewys_1 3y ago
  Predicting weather and stock prices don't seem too far apart. Is it inevitable that all market alpha gets mined by AI?
  • HereBePandas 3y ago
    I'd be shocked - given the incentives - if it hasn't already happened to a great extent. Many of the types of people Google DeepMind hires are also the types of people hedge funds hire.
• rottc0dd 3y ago
  How long does this forecasting hold, given butterfly effect et al?
• EricLeer 3y ago
  I am in the power forecasting domain, where weather forecasts are one of the most important inputs. What I find surprising is that with all the papers and publications from google in the past years, there seems to be no way to get access to these forecasts! We've now evaluated numerous of the ai weather forecasting startups that are popping up everywhere and so far for all of them their claims fall flat on their face when you actually start comparing their quality in a production setting next to the HRES model from ECMWF.
  • scellus 3y ago
    GraphCast, Pangu-Weather from Huawei, FourCastNet and EC's own AIFS are available on the ECMWF chart website https://charts.ecmwf.int https://charts.ecmwf.int, click "Machine learning models" on the left tab. (Clicking anything makes the URL very long.) Some of these forecasts are also downloadable as data, but I don't know whether GraphCast is. Alternatively, if forecasts have a big economic value to you, loading latest ERA5 and the model code, and running it yourself should be relatively trivial? (I'm no expert on this, but I think that is ECMWF's aim, to distribute some of the models and initial states as easily runnable.)
• isaacfrond 3y ago
  I find this quite surprising actually. You'd think predicting the weather is mostly a matter of fast computation. The physical rules are well understood, so to get a better estimate use a finer mesh in your finite element computation and use a smaller time scale in estimating your differential equations. Neural networks are notoriously bad at exact approximation. I mean you can never beat a calculator when the issue is doing calculations. So apparently the AI found some shortcut for doing the actual computational work. That is also surprising as weather is a chaotic system. Shortcuts should not exist. Long story short, I don't get what's going on here.
  • uoaei 3y ago
    > The physical rules are well understood Nope. They're constantly updating these models with really finnicky things like cloud nucleation rates that differ depending on which tree species's pollen is in the air. They've gotten a lot better (~2 day to ~7 day hi-res forecasts) but they're still wrong a lot of the time. The reason is the chaos as you say, however, chaos is deterministic, so, that a deterministic method can approximate a deterministic system is really not the surprising part. You don't get what's going on here because your baseline understanding is a lot worse than you think it is. What they're doing is skipping literal numerical simulation in favor of graph- (attention-) based approaches. Typical weather models simulate pretty fine resolution and return hourly forecasts. Google's new approach is learning an approximate Markov model at 6 hours resolution directly so they don't need to run on massive supercomputers.
    • flir 3y ago
      It's a model of a model? And it turns out to be better? That's so counter-intuitive I'm kinda amazed anyone even bothered to research it, let alone that it worked. Uh..... now do horse racing.
      • uoaei 3y ago
        "All models are wrong, some models are useful." Some are more wrong and more useful simultaneously ;) This is actually the typical state of things in numerical simulation: we have infinite-resolution differential equations modeling such physical systems, but to implement them in silico we need to discretize and approximate various aspects of those models to achieve usefulness re: time and accuracy. Google has merely gone one level further in the tradeoff. For more info on Google's approach, look into surrogate models. It's becoming more common especially in things like weather and geology.
• greatpostman 3y ago
  OpenAI is releasing legitimate AGI, google puts out a weather prediction model lol.
  • dnlkwk 3y ago
    I do love Google Maps more than any other product they have lol
  • lambda_garden 3y ago
    OpenAI has not released AGI.
  • deleted 3y ago
    [deleted]
  • postexitus 3y ago
    You assuming OpenAI's models are AGI tells more about you than anything else.
    • rational_indian 3y ago
      If Alan Turing says ChatGPT is an AGI, it's good enough for me.
      • GaggiX 3y ago
        The Turing test is there only to test a machine's ability to exhibit intelligent behaviour, not to test if it's an AGI or not.
      • piyh 3y ago
        Not quite https://arxiv.org/abs/2310.20216 https://arxiv.org/abs/2310.20216
      • debugnik 3y ago
        I doubt he would've, half the point of Turing's paper was to stop people from debating what is or isn't "thinking" and to focus on the actual capabilities instead (like passing the test). He specifically wrote: > "Can machines think?" I believe to be too meaningless to deserve discussion. So I don't think he would've appreciated such a fuzzy concept as AGI.
• mg 3y ago
  It's interesting, that Google keeps publishing AI research papers. Is there a business rationale behind it? OpenAI has become one of the fastest growing companies of all time. And much of it is based on Google's "Attention is all you need" and other papers. Since Microsoft added the Dall-E 3 image creator to Bing, Bing saw a huge inflow of new users. Dall-E is also a technology rooted in Google papers. I wonder how Google thinks about this internally.
  • ArnoVW 3y ago
    It’s difficult to retain top talent if you do not allow them to publish.
    • DaiPlusPlus 3y ago
      How does Apple do it, if anyone knows? Apple is so loathe to keep their potential product plans hidden that AAPL employees aren’t even allowed to have GitHub accounts without mgr approval… but they have to be employing serious researchers, but they’ll never get to publish on volition.
      • rational_indian 3y ago
        How does Apple do what? AFAICT Apple does not do research, at least at the same level or on the same topics as Google or Microsoft.
      • Someone 3y ago
        > they have to be employing serious researchers, but they’ll never get to publish on volition. That’s not true. I wouldn’t know how free they are to publish but they do publish stuf. See https://machinelearning.apple.com/ https://machinelearning.apple.com/
      • famouswaffles 3y ago
        Apple does publish some stuff. But anyway it's a balance between publishing and shipping products. The researcher wants to get some credit for his/her work. If you ship a lot of products he can put his/her name on then publishing research isn't quite as important and vice versa.
• matsemann 3y ago
  How's the distribution of the errors? For instance I don't care if it's better on average by 1 Celsius each day for normal weather, if it once every month is off by 10 Celsius when there is a drastic weather event, for instance. I'm all for better weather data, it's quite critical up in the mountains, so that's why my question about how reliable it is in life&death situations.
  • CorrectHorseBat 3y ago
    https://www.science.org/doi/10.1126/science.adi2336 https://www.science.org/doi/10.1126/science.adi2336 Seems like it's better at predicting extreme weather events
  • deleted 3y ago
    [deleted]
• tony_cannistra 3y ago
  Similar methodologies are being applied to climate modeling, too. The Allen Institute has worked on it for a while, and has hired quite a few PhDs (https://allenai.org/climate-modeling https://allenai.org/climate-modeling).
  • lainga 3y ago
    How long? The cloud microparameterisation looks really exciting, but 10-year stability for a GCM (and "nearly conserving" water) is not great
    • tony_cannistra 3y ago
      I'm not sure. NVIDIA is also working on it (with, interestingly, some of the original AI2 folks). Similar to the DeepMind effort, the ACE ML model that AI2+others developed is really just looking for parity with physical models at this stage. It looks like they've almost achieved this, with similar massive improvements in compute time + resource needs.
  • deleted 3y ago
    [deleted]
  • hackernewds 3y ago
    why is hiring phds a measure?
    • Lacerda69 3y ago
      if you don't hire phds you're not serious about it
    • tony_cannistra 3y ago
      in this particular case, most of the important/needle-moving work being done in climate modeling is done with a hell of a lot of context about prior work. PhDs have that, by necessity. They're also good at prioritizing outcomes, rather than other stuff.
  • 3y ago
• tokai 3y ago
  Just like their flu modelling outperformed conventional models right?
  • Anon84 3y ago
    Humm... are you referring to Google Flu? [1] That was a very different beast. It relied on using Google searches to infer the prevalence of various Influenza Like Illnesses in real time, while the CDC reports data with a 2-week lag. Notably, some of the queries they found to be correlated were... strange... like NBA results. Not unsurprisingly (in hindsight, at least) [2], this eventually broke down when epidemics and flu symptoms got in the news and completely changed what people were searching for. [1] https://www.nature.com/articles/nature07634 https://www.nature.com/articles/nature07634 [2] https://www.science.org/doi/10.1126/science.1248506 https://www.science.org/doi/10.1126/science.1248506
    • tokai 3y ago
      Yeah I know its way different methods. Sorry for being disingenuous. The point of my snarking was that google made a lot of noise about Google Flu but then quietly got rid of it when it didn't work. To me Googles research has a tendency to be more about headlines than actually solving problems.
      • Anon84 3y ago
        No worries, Google does tend to do a good job of monopolizing attention in whatever they do and Epidemic Modeling is... complicated. Probably much more complicated than pretty much any other kind of modeling since people have the bad habit of thinking and acting in whatever way they want (sometimes with the explicit purpose of breaking your model :). Now, if you want to see the real-world state-of-the-art epidemic modeling on a global scale, checkout GLEaM/GLEaMViz https://www.gleamviz.org/ https://www.gleamviz.org/ (full disclaimer, in a previous life I was the lead developer). And if you're interested in a basic intro, you can also checkout my (somewhat neglected) series of blog posts from the pandemic days: https://github.com/DataForScience/Epidemiology101 https://github.com/DataForScience/Epidemiology101 </ShamelessSelfPromotion>
• acolderentity 3y ago
  How could an ai, programmed with the bias of people that already suck at predicting the weather, even get close to being accurate?
  • david-gpu 3y ago
    You don't train the AI with the forecasts made by other systems. You train the AI with the actual weather that was measured hours/days later.
  • jvalencia 3y ago
    Weather is a complex mix of many systems. The traditional approach is to understand all the systems and add them together. Since we don't understand them all fully, we get a lot of chaos. The ML algorithm doesn't care about the science, the agendas, the theories, nothing. It just looks for patterns in the data. Instead of an exact calculation it's more akin to numerical analysis. Turns out that looking at the whole in this case, is better than the sum of the parts.
  • deleted 3y ago
    [deleted]
  • mesoman 3y ago
    The people who predict the weather are often damned smart and very experienced. It's the problem that's hard.
• tchvil 3y ago
  windguru which is in part or fully based on crowd-sourced weather stations is already surprisingly accurate few days in advance, in many regions I tried. For a few hours forecast nothing beats the rain radar. I wonder if they have already or will put some AI in their models.
  • hackernewds 3y ago
    anecdata does not equal data?
• hexo 3y ago
  "AI" aka machine learning
  • deleted 3y ago
    [deleted]
• drcongo 3y ago
  Is there a chance that it just made something up and got lucky like ChatGPT?
  • Kuinox 3y ago
    Is there a chance you made something up and got lucky ?
    • drcongo 3y ago
      A huge chance!
• alberth 3y ago
  Is this really an "AI" story? Aren't existing weather forecasting models, already a form of "AI"? I'm no AI/ML expert, but isn't the real story here is that a new model (like GPT-4.0) is better than the previous/existing model (GPT-3.5). It's just grabs way more attention calling the new model "AI" (vs not referring to the old as such).
  • mdpye 3y ago
    It's an ML story. The article specifies that the current (now previous?) state of the art models are numerical, crunching vast equations representing atmospheric physics.
  • surfmike 3y ago
    No, existing models use more numerical methods. This is using a completely different approach. > GraphCast utilizes what researchers call a "graph neural network" machine-learning architecture, trained on over four decades of ECMWF's historical weather data. It processes the current and six-hour-old global atmospheric states, generating a 10-day forecast in about a minute on a Google TPU v4 cloud computer. Google's machine learning method contrasts with conventional numerical weather prediction methods that rely on supercomputers to process equations based on atmospheric physics, consuming significantly more time and energy.
    • ramkumarkoppu 3y ago
      How to translate the graphcast model output to usual weather variables like temperature, rain, wind, etc if I have to build a weather dashboard?
  • deleted 3y ago
    [deleted]
• dauertewigkeit 3y ago
  The multimesh is interesting. Still, I bet the Fourier Neural Operator approach will prove superior. Members of the same team (Sanchez-Gonzales, Battaglia) have already published multiple variations of this model, applied to other physical scenarios and lots of them proved to be dead ends. My money is on the FNO approach, anyway, which for some reason is only given a brief reference. To their credit DeepMind usually publishes extensive comparisons with previously published models. This time such a comparison is conspicuously missing. Full disclosure: I think DeepMind often publish these bombastic headlines about their models which often don't live up to their hype, or at least that was my personal experience. They have a good PR team, anyway.
  • kleiba 3y ago
    How is that a disclosure?
  • counters 3y ago
    Pragmatically speaking, it doesn't really matter if one is better than the other, at least until there is a massive jump in forecast quality (e.g. advancing the Day 5 accuracy up to Day 3). In the real world, we would never take raw model guidance from _any_ source - the best forecasts invariably come from consensus systems that look across many different models. So it's good to have a diverse lineage of forecasting systems, as uncorrelated errors boost the performance of these consensus systems.
• thriftwy 3y ago
  Yandex claims to be using AI-based weather forecasting for a good part of a decade and claims it as a success. It is quite good. https://meteum.ai/ https://meteum.ai/
  • counters 3y ago
    My understanding is that they just use an AI-based precipitation nowcast (see [1]). Very different forecast/modeling problem than GraphCast. [1]: https://arxiv.org/abs/1905.09932 https://arxiv.org/abs/1905.09932
• devit 3y ago
  Seems like it would be much better to do conventional weather forecasting and then feed the predictions along with input data and other relevant information to a machine learning system.
• rldjbpin 3y ago
  > GraphCast makes forecasts at the high resolution of 0.25 degrees longitude/latitude (28km x 28km at the equator). the resolution, while seemingly impressive, is very imprecise compared to the SOTA in the theoretical modelling side. this discredits the computational claims made by the paper for me. i understand that the current simulations can go down to meter scale, but i wonder what the compuational requirements are when you calculate for this resolution.