5 ms·

Towards a Physics Foundation Model

117 points by NeoInHacker 1y ago

measurablefunc 1y ago
How do they prove their model preserves conservation principles? I looked in the paper & didn't find any evidence of how they verify that whatever their "trained" model is doing is actually physically plausible & maintains the relevant invariants like mass, energy, momentum, etc.
- esafak 1y ago
  From a quick scan, I do not think they explicitly encode that. They want "the model to predict the evolution of diverse physical systems governed by partial differential equations". It looks like a more sophisticated sibling of time series forecasting models rather than a physics-informed nonparametric symbolic regression model.
  - NeoInHacker 1y ago
    Yeah, It’s true that PDEs are the "top-tier tool" for describing physical phenomena—from the laws of motion in classical mechanics and electromagnetic waves in electromagnetism to the evolution of wave functions in quantum mechanics, they accurately model most macroscopic, classical scenarios. However, when it comes to covering all physical phenomena, they really "fall short": in quantum gravity, spacetime may be discontinuous, making the concept of differentiation meaningless; for complex systems like turbulence, PDEs cannot be solved nor can they capture macroscopic laws; even for the randomness of quantum measurements, PDEs can only predict probability distributions and fail to explain the underlying nature. In short, they are a "top-tier auxiliary," but by no means a "one-size-fits-all key."
    - codethief 1y ago
      > in quantum gravity GP was asking about conservation laws but in gravity you don't even have energy-momentum conservation.
- bobmarleybiceps 1y ago
  I think very few of these "replace numerical solver with ML model" papers do anything to verify invariants are satisfied (they often are not well preserved). They basically all just check that the model approximately reproduces some dynamics on a test data of PDEs, that's often sampled from the same distribution as the training dataset...
- flwi 1y ago
  Author here: we do NOT do conservation of energy/momentum. We are currently trying to incorporate that in a follow up paper, but for now, the models that try that (e.g. PINNs (soft constraint) or hard constraint models, all perform bad when modeling multiple systems. Perhaps, we will encounter the bitter lesson again and a well trained model will solve this. But as I said, we are also looking at hybrid models
- flwi 1y ago
  Author here: we do NOT do conservation of energy/momentum. We are currently trying to incorporate that in a follow up paper, but for now, the models that try that (e.g. PINNs (soft constraint) or hard constraint models, all perform badly when modeling multiple systems. Perhaps, we will encounter the bitter lesson again and a well trained model will solve this. But as I said, we are also looking at hybrid models
- ogogmad 1y ago
  Why? Is this important as a sanity check in the absence of any independent verifications?
  - bobmarleybiceps 1y ago
    I'm not an expert on this, so take this with a grain of salt. Chaotic PDEs are extremely sensitive to initial conditions. This essentially makes it so that any numerical solution will (quickly) diverge from the true solution over time. (Just due to floating point error, discretization error, etc.) This is why for a lot of turbulent navier-stokes stuff, people don't necessarily care about the specific phenomena that occur, but look at statistical properties. I think one of the reasons it is important to preserve conservation laws is that, at the very least, you can be confident that your solution satisfies whatever physical laws your PDE relies on, even if it's almost certainly not the "actual" solution to the PDE. You actually can ensure that a numerical solver will approximately satisfy conservation laws. Then at the very least, even if your solution diverges from the "actual" PDEs solution, you can have some confidence that it's still a useful exploration of possible states. If conservation laws are not preserved AND your solution diverges from the "actual" PDE solution, then you probably cannot be confident about the model's utility.
    - bobmarleybiceps 1y ago
      Actually I just happened to see this: https://www.stochasticlifestyle.com/how-chaotic-is-chaos-how-some-ai-for-science-sciml-papers-are-overstating-accuracy-claims/ https://www.stochasticlifestyle.com/how-chaotic-is-chaos-how.... It's basically explaining the same thing, but much better than me :-)
- woctordho 1y ago
  I guess it can be implemented in the 'sampler' part. When solving an actual PDE, project the output of the AI onto a space that preserves the invariants.
ISL 1y ago
ITAR will be quite a surprise to some when it suddenly makes an appearance.
flwi 1y ago
Author here, Wow, I didn't think this would HN. I actually planned to do the advertisement rounds only after the final ICLR submission. This is our attempt at creating a model which understands multiple physics, which is in contrast to PINNs and Neural Operators, which focus on much more narrow systems. Obviously, the biggest issue is still data (3D and real-world problems), but I think we and a few other groups make significant progress here.
- IronyMan100 1y ago
  Some month ago i stumbled upon two arcticles discussing PINNs and their failiures in more complex settings. are there similar challenges?
  - flwi 1y ago
    Can you point me to the papers? In general, faster dynamics and chaotic systems are probably the hardest. Of course combined with long-term stability
- lindboe 1y ago
  Great paper! Off the top of your head, are you aware of any similar general-multiphysics NN work that's been applied to electromagnetics problems? In particular, some colleagues in my lab are investigating imaging via acoustic waves which are induced by microwave absorptive heating (in liquids, biological tissues, etc.); this approach is most commonly known as RF-induced thermoacoustic imaging [1]. It's very tricky to model this phenomenon in simulation, doubly so to measure it experimentally. Most in my lab (myself included) are leery of throwing NNs at problems and seeing what sticks, but sometimes I wonder whether a model like yours might help us skip past the boring details to get at the novel technical stuff, or else extend simulations to more complicated boundary conditions. [1] https://ieeexplore.ieee.org/abstract/document/6248685 https://ieeexplore.ieee.org/abstract/document/6248685
  - flwi 1y ago
    I haven't seen electromagnetic systems included yet, probably since they are less training data for it. In your chase, with such specific systems, a model trained only on your data might make more sense, though
danpalmer 1y ago
Not the "foundational model" of physics I was expecting, but this is still great to see!
petargyurov 1y ago
Anyone remember that one time, a year or so ago, when some company teased a physics based generative model which showcased a drop of water sliding down a beer bottle and the model could display the forces acting on it? Whatever happened to that? Vapourware?
- flwi 1y ago
  I think you mean this: https://genesis-embodied-ai.github.io/ https://genesis-embodied-ai.github.io/ It seems that this is much more focused on robotics, but interesting nonetheless.
  - xxprogamerxy 1y ago
    Genesis is also a traditional physics engine, no ML-based physics prediction going on here. To my understanding their performance gains mainly come from building the engine to be highly parallelizable.
witnessme 1y ago
For folks wondering whether to read or not, here is the conclusion from the paper verbatim > We have demonstrated that a single transformer-based model can effectively learn and predict the dynamics of diverse physical systems without explicit physics-specific features, marking a significant step toward true Physics Foundation Models. GPhyT not only outperforms specialized architectures on known physics by up to an order of magnitude but, more importantly, exhibits emergent in-context learning capabilities—inferring new boundary conditions and even entirely novel physical phenomena from input prompts alone.
- godelski 1y ago
  Thanks, I haven't been able to give the paper a proper read, but are they're basing claims via results or the ability to recover physics equations? Because those two things are very different. You can have models that make accurate predictions without having accurate models of "the world" (your environment, not necessarily the actual world)[0]. We can't meaningful call something a physics model (or a world model) without that counterfactual recovery (you don't need the exact laws of physics but you need something reasonable). After all, our physics equations are the most compressed forms or representing the information we're after. I ask because this is a weird thing that happens in a lot of ML papers when approaching world models. But just looking at results isn't enough to conclude if a world is being modeled. Doesn't even tell you if that's self consistent, let alone counterfactual. [0] classic example is the geocentric model. They made accurate predictions, which is why it stayed around for so long. It's not like the heliocentric model didn't present new problems. There was reason for legitimate scientific debate at the time but that context is easily lost to history.
  - flwi 1y ago
    Hey author here. Your argument is completely valid, we only model physics implicitly and thus have no prove that the model "actually knows the physics". Practically, this might not matter much: If the model can predict the evolution of the system to a certain accuracy, the user won't care about the underlying knowledge. And even for modern physics (quantum / GR), we know we miss something and yet, the models we have are incredibly useful. On a tangent, we cannot prove that LLMs actually know language, yet they can be incredibly useful. Of course, a true world model would be much nicer to have, I agree with that!
outsideoftime 1y ago
[dead]

117 points by NeoInHacker 1y ago

How do they prove their model preserves conservation principles? I looked in the paper & didn't find any evidence of how they verify that whatever their "trained" model is doing is actually physically plausible & maintains the relevant invariants like mass, energy, momentum, etc.

esafak 1y ago
From a quick scan, I do not think they explicitly encode that. They want "the model to predict the evolution of diverse physical systems governed by partial differential equations". It looks like a more sophisticated sibling of time series forecasting models rather than a physics-informed nonparametric symbolic regression model.
- NeoInHacker 1y ago
  Yeah, It’s true that PDEs are the "top-tier tool" for describing physical phenomena—from the laws of motion in classical mechanics and electromagnetic waves in electromagnetism to the evolution of wave functions in quantum mechanics, they accurately model most macroscopic, classical scenarios. However, when it comes to covering all physical phenomena, they really "fall short": in quantum gravity, spacetime may be discontinuous, making the concept of differentiation meaningless; for complex systems like turbulence, PDEs cannot be solved nor can they capture macroscopic laws; even for the randomness of quantum measurements, PDEs can only predict probability distributions and fail to explain the underlying nature. In short, they are a "top-tier auxiliary," but by no means a "one-size-fits-all key."
  - codethief 1y ago
    > in quantum gravity GP was asking about conservation laws but in gravity you don't even have energy-momentum conservation.
bobmarleybiceps 1y ago
I think very few of these "replace numerical solver with ML model" papers do anything to verify invariants are satisfied (they often are not well preserved). They basically all just check that the model approximately reproduces some dynamics on a test data of PDEs, that's often sampled from the same distribution as the training dataset...
flwi 1y ago
Author here: we do NOT do conservation of energy/momentum. We are currently trying to incorporate that in a follow up paper, but for now, the models that try that (e.g. PINNs (soft constraint) or hard constraint models, all perform bad when modeling multiple systems. Perhaps, we will encounter the bitter lesson again and a well trained model will solve this. But as I said, we are also looking at hybrid models
flwi 1y ago
Author here: we do NOT do conservation of energy/momentum. We are currently trying to incorporate that in a follow up paper, but for now, the models that try that (e.g. PINNs (soft constraint) or hard constraint models, all perform badly when modeling multiple systems. Perhaps, we will encounter the bitter lesson again and a well trained model will solve this. But as I said, we are also looking at hybrid models
ogogmad 1y ago
Why? Is this important as a sanity check in the absence of any independent verifications?
- bobmarleybiceps 1y ago
  I'm not an expert on this, so take this with a grain of salt. Chaotic PDEs are extremely sensitive to initial conditions. This essentially makes it so that any numerical solution will (quickly) diverge from the true solution over time. (Just due to floating point error, discretization error, etc.) This is why for a lot of turbulent navier-stokes stuff, people don't necessarily care about the specific phenomena that occur, but look at statistical properties. I think one of the reasons it is important to preserve conservation laws is that, at the very least, you can be confident that your solution satisfies whatever physical laws your PDE relies on, even if it's almost certainly not the "actual" solution to the PDE. You actually can ensure that a numerical solver will approximately satisfy conservation laws. Then at the very least, even if your solution diverges from the "actual" PDEs solution, you can have some confidence that it's still a useful exploration of possible states. If conservation laws are not preserved AND your solution diverges from the "actual" PDE solution, then you probably cannot be confident about the model's utility.
  - bobmarleybiceps 1y ago
    Actually I just happened to see this: https://www.stochasticlifestyle.com/how-chaotic-is-chaos-how-some-ai-for-science-sciml-papers-are-overstating-accuracy-claims/ https://www.stochasticlifestyle.com/how-chaotic-is-chaos-how.... It's basically explaining the same thing, but much better than me :-)
woctordho 1y ago
I guess it can be implemented in the 'sampler' part. When solving an actual PDE, project the output of the AI onto a space that preserves the invariants.

ISL 1y ago

ITAR will be quite a surprise to some when it suddenly makes an appearance.

flwi 1y ago

Author here, Wow, I didn't think this would HN. I actually planned to do the advertisement rounds only after the final ICLR submission. This is our attempt at creating a model which understands multiple physics, which is in contrast to PINNs and Neural Operators, which focus on much more narrow systems. Obviously, the biggest issue is still data (3D and real-world problems), but I think we and a few other groups make significant progress here.

IronyMan100 1y ago
Some month ago i stumbled upon two arcticles discussing PINNs and their failiures in more complex settings. are there similar challenges?
- flwi 1y ago
  Can you point me to the papers? In general, faster dynamics and chaotic systems are probably the hardest. Of course combined with long-term stability
lindboe 1y ago
Great paper! Off the top of your head, are you aware of any similar general-multiphysics NN work that's been applied to electromagnetics problems? In particular, some colleagues in my lab are investigating imaging via acoustic waves which are induced by microwave absorptive heating (in liquids, biological tissues, etc.); this approach is most commonly known as RF-induced thermoacoustic imaging [1]. It's very tricky to model this phenomenon in simulation, doubly so to measure it experimentally. Most in my lab (myself included) are leery of throwing NNs at problems and seeing what sticks, but sometimes I wonder whether a model like yours might help us skip past the boring details to get at the novel technical stuff, or else extend simulations to more complicated boundary conditions. [1] https://ieeexplore.ieee.org/abstract/document/6248685 https://ieeexplore.ieee.org/abstract/document/6248685
- flwi 1y ago
  I haven't seen electromagnetic systems included yet, probably since they are less training data for it. In your chase, with such specific systems, a model trained only on your data might make more sense, though

danpalmer 1y ago

Not the "foundational model" of physics I was expecting, but this is still great to see!

petargyurov 1y ago

Anyone remember that one time, a year or so ago, when some company teased a physics based generative model which showcased a drop of water sliding down a beer bottle and the model could display the forces acting on it? Whatever happened to that? Vapourware?

flwi 1y ago
I think you mean this: https://genesis-embodied-ai.github.io/ https://genesis-embodied-ai.github.io/ It seems that this is much more focused on robotics, but interesting nonetheless.
- xxprogamerxy 1y ago
  Genesis is also a traditional physics engine, no ML-based physics prediction going on here. To my understanding their performance gains mainly come from building the engine to be highly parallelizable.

witnessme 1y ago

For folks wondering whether to read or not, here is the conclusion from the paper verbatim > We have demonstrated that a single transformer-based model can effectively learn and predict the dynamics of diverse physical systems without explicit physics-specific features, marking a significant step toward true Physics Foundation Models. GPhyT not only outperforms specialized architectures on known physics by up to an order of magnitude but, more importantly, exhibits emergent in-context learning capabilities—inferring new boundary conditions and even entirely novel physical phenomena from input prompts alone.

godelski 1y ago
Thanks, I haven't been able to give the paper a proper read, but are they're basing claims via results or the ability to recover physics equations? Because those two things are very different. You can have models that make accurate predictions without having accurate models of "the world" (your environment, not necessarily the actual world)[0]. We can't meaningful call something a physics model (or a world model) without that counterfactual recovery (you don't need the exact laws of physics but you need something reasonable). After all, our physics equations are the most compressed forms or representing the information we're after. I ask because this is a weird thing that happens in a lot of ML papers when approaching world models. But just looking at results isn't enough to conclude if a world is being modeled. Doesn't even tell you if that's self consistent, let alone counterfactual. [0] classic example is the geocentric model. They made accurate predictions, which is why it stayed around for so long. It's not like the heliocentric model didn't present new problems. There was reason for legitimate scientific debate at the time but that context is easily lost to history.
- flwi 1y ago
  Hey author here. Your argument is completely valid, we only model physics implicitly and thus have no prove that the model "actually knows the physics". Practically, this might not matter much: If the model can predict the evolution of the system to a certain accuracy, the user won't care about the underlying knowledge. And even for modern physics (quantum / GR), we know we miss something and yet, the models we have are incredibly useful. On a tangent, we cannot prove that LLMs actually know language, yet they can be incredibly useful. Of course, a true world model would be much nicer to have, I agree with that!

outsideoftime 1y ago

[dead]