5 ms·

I truly believe that Bayesian inference is the statistics of the 21st century. Recent advances in MCMC (e.g., NUTS, Stan [1]) and variational inference (e.g., A

I truly believe that Bayesian inference is the statistics of the 21st century. Recent advances in MCMC (e.g., NUTS, Stan [1]) and variational inference (e.g., ADVI [2], VAE [3], etc.) + more computing power than ever promise a near future in which Bayesian inference is the default inference engine. Prior distribution is a beautiful and logical mechanism for adding regularization, domain-specific knowledge to our model. [1] Stan, a platform for statistical modeling http://mc-stan.org/ http://mc-stan.org/ [2] Automatic Differentiation Variational Inference https://arxiv.org/abs/1603.00788 https://arxiv.org/abs/1603.00788 [3] Auto-Encoding Variational Bayes https://arxiv.org/abs/1312.6114 https://arxiv.org/abs/1312.6114

riku_iki 8y ago
Curious if there is any work to build ecosystem to run complex models on ML accelerators, like some layers for TF for example..
- xcodevn 8y ago
  There are several libraries. PyMC4 [1], the next version of PyMC3, will introduce TF as a backend. Tensorflow probability [2] from Google. PYRO [3], from Uber, uses Pytorch backend. [1]: https://github.com/pymc-devs/pymc4 https://github.com/pymc-devs/pymc4 [2]: https://www.tensorflow.org/probability/ https://www.tensorflow.org/probability/ [3]: http://pyro.ai/ http://pyro.ai/
andbberger 8y ago
NUTS and Stan are quite old at this point! Here's a more recent advance https://arxiv.org/pdf/1711.09268.pdf https://arxiv.org/pdf/1711.09268.pdf
- panarky 8y ago
  Stan supports Hamiltonian Monte Carlo. https://arxiv.org/pdf/1701.02434.pdf https://arxiv.org/pdf/1701.02434.pdf
  - andbberger 8y ago
    Many samplers are based on HMC, it's a general class of samplers, not a specific algorithm. NUTS is a variation of HMC, as well as the paper I linked above. 'vanilla' HMC uses detailed balance to guarantee that the stationary distribution of the chain is the one you want, causing the process to behave like a random walk. So although the Hamiltonian bit of HMC lets you take these great big steps through state space, you end up retracing your steps quite a lot. Hence NUTS (No-U-turn sampler) et. al
bo1024 8y ago
It's fine for a company or individual trying to optimize an objective, but not as a way to do good science (which this article is about).
- xcodevn 8y ago
  I strongly disagree. Bayesian inference is the only known self-consistent formal system for doing science, i.e. updating our belief system about the world based on the current evidence.
  - bo1024 8y ago
    The problem with that reasoning is: whose belief system? Where do you come up with a prior that everyone agrees with?
    - KirinDave 8y ago
      Actually, it's the strength of Bayesian inference that these assumptions are made apparent. Coming to consensus on priors is the same process for arriving at consensus that all scientific inquiry must engage in. Anyone who says frequentist methods somehow more accurately represent an underlying reality are pulling a fast one.
      - bo1024 8y ago
        Hmm, I think part of the question is where this debate and consensus should occur. I believe in firmly separating rigorous science from opinion and belief. To me, it follows that scientific research should focus on presenting evidence and leave it to Bayesian individuals to update their beliefs based on this evidence. Similarly I think argument or discussion about priors is not in scope for scientific research (except maybe a bit in the "motivation" subsection). (edit: I see now the top post here on this article also discusses this point.)
        KirinDave 8y ago
mturmon 8y ago
But note, the last paragraph of TFA cautions against use of a Bayesian prior in cases where it is not well supported by actual hard prior information. It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right. If the prior does not wash out due to lots of data, this indicates a serious and fundamental problem.
- xcodevn 8y ago
  > It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right. Both prior and likelihood are our model's assumptions. So, the prior validation problem is similar to the likelihood validation problem. To check a Bayesian model or any model, we need to bring the model out of the formal world, to the real world for validation. Prior predictive simulation method, which generates random data points from the prior, is a good heuristic to check if the prior is NOT plausible.
  - mturmon 8y ago
    > ...the prior validation problem is similar to the likelihood validation problem... But priors can be much harder. Say I’m trying to estimate a wind speed from the blade velocity of a windmill. I can bring a more accurate wind speed sensor to calibrate the windmill against the wind speed, perhaps aided by basic physics. This is the likelihood portion. But what should the prior be? The typical speed at that time of day? The speed in January? The speed on cloudy days? I have to have a crisp number — a full distribution actually, accurate out to the tails. I really have very little grounding for choosing that distribution. I started out just wanting to relate the wind speed to some data in a rather concrete way, and now I’ve been roped in to choosing a crisp distribution for a rather amorphous state of nature. This is a deep problem. We can sharpen the problem. Say my number and yours are different. How do we tell who is right? One can try a different tack: I’m being stubborn. The prior will mostly wash out in any well-posed problem, or else why try to solve it? But now we’re back to frequentism, just looking at the likelihood. HN tends to invoke the Bayesian framework as a complete solution to inference — I’m just trying to demonstrate that there are problems with that approach.
platz 8y ago
I feel like variational inference has never been described very well to an intro audience even having statistical basics. Is it a graduate level topic or is there an intuitive course that teaches it to beginners?
- klipt 8y ago
  "variational inference" is perhaps an uninformative name. You can just think of it as - approximating the posterior using a nice parametric distribution, then - minimizing some error (typically KL Divergence) between your approximate posterior and the true posterior
  - ced 8y ago
    Do you know _why_ KL divergence is minimized? I get that it gives a lower bound on the marginal likelihood, which is cool, but is that it? What are the alternatives?
    - ssivark 8y ago
      KL divergence is motivated nicely from an information/coding theory viewpoint. It's very closely related to Shannon-von Neumann entropy [1], and KL(P||Q) characterizes the efficiency of a code designed for a model distribution P, when applied to reality which is actually represented by Q. A lot of recent work focuses on the Wasserstein divergence [1] as an alternative. One advantage of Wasserstein over KL is that the Wasserstein metric provides better fit over the whole distribution instead of localizing on some specific regions, thereby preventing "mode collapse". This makes it a popular metric for training Generative Adversarial Networks (GANs). For recent work on applying Wasserstein distance to variational inference, see: https://arxiv.org/abs/1805.11284 https://arxiv.org/abs/1805.11284 [1]: https://physics.stackexchange.com/questions/64574/definition-of-the-entropy/64597#64597 https://physics.stackexchange.com/questions/64574/definition... [2]: https://en.wikipedia.org/wiki/Wasserstein_metric https://en.wikipedia.org/wiki/Wasserstein_metric
- meatmanek 8y ago
  I found this talk to be useful, despite the technical difficulties https://www.youtube.com/watch?v=Dv86zdWjJKQ https://www.youtube.com/watch?v=Dv86zdWjJKQ (I haven't watched https://www.youtube.com/watch?v=ogdv_6dbvVQ https://www.youtube.com/watch?v=ogdv_6dbvVQ but it seems like a longer version of the same talk)

riku_iki 8y ago

Curious if there is any work to build ecosystem to run complex models on ML accelerators, like some layers for TF for example..

xcodevn 8y ago
There are several libraries. PyMC4 [1], the next version of PyMC3, will introduce TF as a backend. Tensorflow probability [2] from Google. PYRO [3], from Uber, uses Pytorch backend. [1]: https://github.com/pymc-devs/pymc4 https://github.com/pymc-devs/pymc4 [2]: https://www.tensorflow.org/probability/ https://www.tensorflow.org/probability/ [3]: http://pyro.ai/ http://pyro.ai/

andbberger 8y ago

NUTS and Stan are quite old at this point! Here's a more recent advance https://arxiv.org/pdf/1711.09268.pdf https://arxiv.org/pdf/1711.09268.pdf

panarky 8y ago
Stan supports Hamiltonian Monte Carlo. https://arxiv.org/pdf/1701.02434.pdf https://arxiv.org/pdf/1701.02434.pdf
- andbberger 8y ago
  Many samplers are based on HMC, it's a general class of samplers, not a specific algorithm. NUTS is a variation of HMC, as well as the paper I linked above. 'vanilla' HMC uses detailed balance to guarantee that the stationary distribution of the chain is the one you want, causing the process to behave like a random walk. So although the Hamiltonian bit of HMC lets you take these great big steps through state space, you end up retracing your steps quite a lot. Hence NUTS (No-U-turn sampler) et. al

bo1024 8y ago

It's fine for a company or individual trying to optimize an objective, but not as a way to do good science (which this article is about).

xcodevn 8y ago
I strongly disagree. Bayesian inference is the only known self-consistent formal system for doing science, i.e. updating our belief system about the world based on the current evidence.
- bo1024 8y ago
  The problem with that reasoning is: whose belief system? Where do you come up with a prior that everyone agrees with?
  - KirinDave 8y ago
    Actually, it's the strength of Bayesian inference that these assumptions are made apparent. Coming to consensus on priors is the same process for arriving at consensus that all scientific inquiry must engage in. Anyone who says frequentist methods somehow more accurately represent an underlying reality are pulling a fast one.
    - bo1024 8y ago
      Hmm, I think part of the question is where this debate and consensus should occur. I believe in firmly separating rigorous science from opinion and belief. To me, it follows that scientific research should focus on presenting evidence and leave it to Bayesian individuals to update their beliefs based on this evidence. Similarly I think argument or discussion about priors is not in scope for scientific research (except maybe a bit in the "motivation" subsection). (edit: I see now the top post here on this article also discusses this point.)
      - KirinDave 8y ago

mturmon 8y ago

But note, the last paragraph of TFA cautions against use of a Bayesian prior in cases where it is not well supported by actual hard prior information. It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right. If the prior does not wash out due to lots of data, this indicates a serious and fundamental problem.

xcodevn 8y ago
> It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right. Both prior and likelihood are our model's assumptions. So, the prior validation problem is similar to the likelihood validation problem. To check a Bayesian model or any model, we need to bring the model out of the formal world, to the real world for validation. Prior predictive simulation method, which generates random data points from the prior, is a good heuristic to check if the prior is NOT plausible.
- mturmon 8y ago
  > ...the prior validation problem is similar to the likelihood validation problem... But priors can be much harder. Say I’m trying to estimate a wind speed from the blade velocity of a windmill. I can bring a more accurate wind speed sensor to calibrate the windmill against the wind speed, perhaps aided by basic physics. This is the likelihood portion. But what should the prior be? The typical speed at that time of day? The speed in January? The speed on cloudy days? I have to have a crisp number — a full distribution actually, accurate out to the tails. I really have very little grounding for choosing that distribution. I started out just wanting to relate the wind speed to some data in a rather concrete way, and now I’ve been roped in to choosing a crisp distribution for a rather amorphous state of nature. This is a deep problem. We can sharpen the problem. Say my number and yours are different. How do we tell who is right? One can try a different tack: I’m being stubborn. The prior will mostly wash out in any well-posed problem, or else why try to solve it? But now we’re back to frequentism, just looking at the likelihood. HN tends to invoke the Bayesian framework as a complete solution to inference — I’m just trying to demonstrate that there are problems with that approach.

platz 8y ago

I feel like variational inference has never been described very well to an intro audience even having statistical basics. Is it a graduate level topic or is there an intuitive course that teaches it to beginners?

klipt 8y ago
"variational inference" is perhaps an uninformative name. You can just think of it as - approximating the posterior using a nice parametric distribution, then - minimizing some error (typically KL Divergence) between your approximate posterior and the true posterior
- ced 8y ago
  Do you know _why_ KL divergence is minimized? I get that it gives a lower bound on the marginal likelihood, which is cool, but is that it? What are the alternatives?
  - ssivark 8y ago
    KL divergence is motivated nicely from an information/coding theory viewpoint. It's very closely related to Shannon-von Neumann entropy [1], and KL(P||Q) characterizes the efficiency of a code designed for a model distribution P, when applied to reality which is actually represented by Q. A lot of recent work focuses on the Wasserstein divergence [1] as an alternative. One advantage of Wasserstein over KL is that the Wasserstein metric provides better fit over the whole distribution instead of localizing on some specific regions, thereby preventing "mode collapse". This makes it a popular metric for training Generative Adversarial Networks (GANs). For recent work on applying Wasserstein distance to variational inference, see: https://arxiv.org/abs/1805.11284 https://arxiv.org/abs/1805.11284 [1]: https://physics.stackexchange.com/questions/64574/definition-of-the-entropy/64597#64597 https://physics.stackexchange.com/questions/64574/definition... [2]: https://en.wikipedia.org/wiki/Wasserstein_metric https://en.wikipedia.org/wiki/Wasserstein_metric
meatmanek 8y ago
I found this talk to be useful, despite the technical difficulties https://www.youtube.com/watch?v=Dv86zdWjJKQ https://www.youtube.com/watch?v=Dv86zdWjJKQ (I haven't watched https://www.youtube.com/watch?v=ogdv_6dbvVQ https://www.youtube.com/watch?v=ogdv_6dbvVQ but it seems like a longer version of the same talk)