6 ms·

For me, I really appreciate the Bayesian approach because it makes it very explicit that you pick a prior. Perhaps my experience is limited, but every (suppose

For me, I really appreciate the Bayesian approach because it makes it very explicit that you pick a prior. Perhaps my experience is limited, but every (supposedly non-Bayesian) model I've used in practice has been possible to re-express using Bayesian terms, priors and beliefs and so on. Then I get to look at the intitial assumptions (model/prior) and use suitable human hand-wavey judgement about whether they make sense. Bayes is a good way to _update_ models, but if you lose sight of the fact that the bottom of your chain of deduction was a hand-wavey guess, you're in trouble.

eanzenberg 7y ago
Yeah, no thanks though. I don't want every rando adding "priors" that "feel" right to their analysis. Frequentist is straight forward. Both can (and are) abused to prove bias.
- nightski 7y ago
  Frequentist has a prior also though. The uniform distribution. In a sense this might be more biased as it doesn't always accurately describe the situation. Either way I believe the effect of a prior diminishes greatly pretty quickly as you acquire more data. It's only a factor if you have extremely small set of data.
  - zozbot234 7y ago
    The uniform distribution ("flat") prior lets you interpret a maximum likelihood result as a maximum-a-posteriori (MAP) Bayesian point-estimate (implying a 0-or-1 loss function). One could argue that if you refrain from doing this and just stick to a literal application of the likelihood principle, you're not really depending on a flat prior. For that matter, what is a "flat" prior over the parameters also depends on what parameterization you're using. Results that are 'intuitive' under one parameterization may not be under a different one.
  - btilly 7y ago
    Frequentist has a prior also though. The uniform distribution. No. Experimental design affect frequentist conclusions in a way that is inconsistent with _ANY_ prior. Here is a real life example. My aunt and uncle had 7 children. 6 boys and one girl. Were they biased towards having one gender over another? If the null hypothesis is that they aren't, the p-value that you get is easily calculated as 16/2^7 = 1/8 = 0.125. (There is 1 arrangement of 7 girls, 7 of 6 girls and a boy, 7 of 6 boys and a girl, and 1 of 7 boys for 16 equally likely arrangements.) If I add the fact that they planned to have children until they had a boy and a girl, then that changes the p-value. In fact there are only 4 ways that their first 7 children can come out to give evidence this strong against the prior. So the p-value is now 4/2^7 = 1/32 = 0.03125. However a Bayesian looks at this and says that no matter what prior you pick, the knowledge that they planned to have children until they had both a boy and a girl does not affect your posterior conclusion. It literally has nowhere to go in the formula and can't make a difference. Therefore the frequentist's differing conclusions are not consistent with ANY prior, implicit or not.
    - pdonis 7y ago
      > a Bayesian looks at this and says that no matter what prior you pick, the knowledge that they planned to have children until they had both a boy and a girl does not affect your posterior conclusion A Bayesian would say no such thing. A Bayesian would agree that the knowledge that they planned to have children until they had both a boy and a girl doesn't affect your prior: you still are picking how much probability mass you allocate to all of the possible odds of having a boy vs. a girl, and the couple's plans don't affect that. However, a Bayesian would also say that the knowledge that they planned to have children until they had both a boy and a girl significantly changes the likelihood ratio (or p-value, if you prefer to use that) associated with the observed data. And one of the advantages of Bayesianism is that it forces you to make that explicit as well. Notice, for example, that when you calculated the first p-value of 1/8, you implicitly assumed that the couple's plan was "have 7 children, no matter what gender each of them is". The sample space is therefore all possible arrangements of 7 children by gender, and the p-value is 1/8, as you say. But when you calculated the second p-value of 1/32, while you did change the count of arrangements, you failed to recognize that the sample space changed! Now the possibilities are not just all possible arrangements of 7 children (which is what you used), but all possible arrangements of up to 7 children (because the "stop condition" now is not when there are 7 children total, but when there is at least one child of each gender, and that could have happened at a number of children less than 7). So the correct p-value is not 4/2^7, but 4/(2^7 + 2^6 + 2^5 + 2^4 + 2^3 + 2^2) = 4/(2^8 - 2) = 2/127. A Bayesian, who has to calculate the p-value starting from the hypothesis, not the data, would not make that mistake. And Bayesianism does something else too: it forces you to recognize that the p-value is not actually the answer to the question you were asking! By the p-value criterion, at least with the typical threshold of 0.05, the null hypothesis (that your aunt and uncle are not biased towards having one gender) is rejected. But a Bayesian recognizes that the prior probability of the gender ratio, based on abundant previous evidence, is strongly peaked around 50-50, much more strongly peaked than data with a bias equivalent to a p-value of 2/127 can overcome. So the Bayesian is quite ready to accept that your aunt and uncle had no actual bias towards having boys, they just happened to be one of the statistical outliers that are to be expected given the huge number of humans who have children.
- jules 7y ago
  The difference between a frequentist and a Bayesian is that the latter admits that he picks a prior. A frequentist smushes together (1) the statistical assumptions (2) the approximations that make the problem computationally tractable and (3) the mathematical derivations, into one big mess. Just because you're not stating your assumptions doesn't mean there are none. Consider maximum likelihood estimation. It is not invariant under coordinate transformations. So which coordinates you pick is an assumption. In fact, with Bayesian estimation you can do the same thing: picking a prior is equivalent to picking the uniform prior in a different coordinate system. So frequentist estimation does involve picking a prior by picking a coordinate system, even if the frequentist does not admit this. Frequentist methods are conceptually anything but straightforward. The advantage of frequentist methods is that they are computationally tractable. Usually they are best understood as approximations to Bayesian methods. For instance, MLE can be viewed as the variational approximation to Bayes where the family of probability distributions is the family of point masses, and the prior is uniform.
  - Akababa 7y ago
    What do you mean by coordinate transformation? MLE is invariant under parameter transformations because it's just the argmax of the likelihood.
    - jules 7y ago
      Indeed, it is the argmax of the likelihood, but the likelihood is not invariant under coordinate transformations. The quantity p(x)dx is invariant, not p(x). By picking a suitable coordinate transformation you can put the MLE on any value where the likelihood is not zero.
      - deleted 7y ago
        [deleted]
    - knzhou 7y ago
      Adding to the other comments, you still have prior-dependence on a more subtle level, because it depends on what hypotheses are allowed. Here's an extreme example. Consider flipping an apparently fair coin and getting "THHT". The hypothesis that the coin is fair gives this result with likelihood 1/16. The hypothesis that a worldwide government conspiracy has been formed with the sole purpose of ensuring this result... has a likelihood of 1. But nobody would ever declare this the MLE, because "government conspiracy" isn't one of the allowed options. But it isn't precisely because it's unlikely, i.e. because of your prior. Of course this is an extreme example, but there are more innocuous prior-based assumptions baked in too.
- jefft255 7y ago
  See my above comment for an example of a prior that doesn't just "feel" right.
- Turing_Machine 7y ago
  > I don't want every rando adding "priors" that "feel" right to their analysis. Conversely, maybe others see value in a technique that doesn't start from square one every single time. I mean, knowledge is a thing, yes?
  - perl4ever 7y ago
    That's not the point. The point is there is no choice between having priors and not. There is only the choice between acknowledging priors versus doublethink, confusion, and deception. With how fashionable it is to talk about implicit bias, I wonder how those concerns intersect with the people attacking Bayesian approaches here.
- wirrbel 7y ago
  Let's say you perform a maximum-likelihood estimate, you still have an assumption baked in, that maximizing the likelihood given the data is the right way to make your estimate. In fact, it's very interesting to reconstruct a Bayesian prior for a maximum likelihood estimate. For example when you calculate probabilities for a binary event, 10 head flips, 8 tail flips. The ML estimate is 8/18 = 4/9. A Beta-Binomial bayesian model leads to a posterior distribution of Beta(a=8, b=10) with a mean of (8+a0)/(10 + 8 + a0 + b0), with a0 and b0 representing the prior distribution Beta(a0, b0). Now you can see that the maximum likelihood estimate is identical in this case to assuming a bayesian Prior of Beta(0, 0). I am not saying by this that frequentism is Bayesian inference in disguise, rather, you cannot escape the assumptions. Also, frequentism is not that straightforward, there are many kinds of frequentist estimators and it can be complicated to choose among them.
  - zozbot234 7y ago
    The ML estimate is a posterior mode, assuming a flat prior. It's not immediately clear that it will always be possible to find a corresponding posterior mean. (From a Bayesian point of view, this is a difference in loss functions as opposed to priors over the parameters. With a posterior mean, you're making the optimal inference assuming a quadratic loss; a posterior mode is appropriate for a 0-or-1 loss.)
- FeepingCreature 7y ago
  That's why your analysis shouldn't report posterior probability but degree of shift in confidence.
madhadron 7y ago
> it makes it very explicit that you pick a prior But you don't, in general, pick a prior. You pick a procedure that has an expected loss under various conditions. It's one player game theory. If you happen to have a prior, then you can use it to choose a unique procedure that has minimal expected risk for that prior given the loss function, but even so that may not be what you want. For example, you may want a minimax procedure, which may be quite different from the Bayes procedure.
- jbay808 7y ago
  Minimax still requires a probability distribution, which means you need a prior. Edit: Based on the downvotes, I see my audience is not convinced. I'll repeat an explanation I posted a while ago. Probably should make this a blog post because I see this claim quite often. I'd love to know what book you read it in. -- In minimax regret, you have a set of available decisions D, and a set of possible states of nature N, and a utility U(D,N). Each state of nature also has a probability P(N) (which can be influenced by the decision too in some problems). States of nature include "interest rates rise 1%", "interest rates fall 1%", and "interest rates stay the same". Decisions include "invest in stocks" and "invest in bonds". Minimax regret proposes to ignore the probabilities P(N), instead suggesting a way to make a decision purely based on the utilities of the outcomes. But that is actually an illusion. Outside of math class word problems, we don't have N or U(D,N) handed to us on a silver platter. There is always an infinite range of possible states of nature, many of which have a probability approaching but never reaching zero, including states such as "win the lottery", "communist revolution", and "unexpected intergalactic nuclear war". In commonsense decision-making we don't include those states of nature in our decision matrix, because our common sense rules them out as being implausible before we even think about our options. You wouldn't choose to invest in bonds just because stocks have the most regret in the event of a communist takeover. So what actually happens is we intuitively apply some probability threshold that rules out states of nature falling below it from our consideration. Then we minimize max regret on the remaining "plausibly realistic" states of nature. Humans are so good at doing probability mentally that this step happens before we even realize it. But if you are writing code that makes decisions, you'll need to do it, and so you'll need to have at least a rough stab at the probability distributions.
mikorym 7y ago
Are all priors an application of Bayes's theorem? It is confusing to me that there is talk of Bayesian statistics vs. frequentist statistics when both are often used in conjunction. The classic example of a medical test with false positives and false negatives and the prior being incidence in the general population comes to mind. To me that is not just an example of Bayes, but a combination of frequentist statistics with Bayes's theorem. I also seem to recall that Bayes's theorem appears in a standard first year probability and statistics course.
- eli_gottlieb 7y ago
  Bayesian statistics: the parameters you want to infer are modeled as random variables with a non-empirical prior, and Bayes' rule is used to draw inferences. Frequentist statistics: you construct estimators for the parameters you care about, subject to appropriate loss/risk criteria, but without any explicit "prior knowledge". Frequentist statistics with Bayes' theorem: you use available empirical data, plus some exponential-family distribution, to construct an informed prior, then use Bayes' rule to update the prior on evidence. You use this Bayesian approach only for unobservable hypotheses, rather than for parameters which can be estimated. Machine learning: you stack lots and lots of polynomial regressors onto each-other and train them with a loss function until they predict well on the validation set.
  - jbay808 7y ago
    What do you mean by a "non-empirical prior"?
    - chongli 7y ago
      Bayesian statistics is sometimes called subjectivist statistics. Probability in Bayesian statistics reflects your degree of belief in some potential outcome. If you conduct an experiment, you use Bayes’ theorem to update your degree of belief, which is now conditional on the outcome of your experiment. By quantifying your degree of belief in a prior, you give yourself some starting point (rather than just assuming 0 probability), even if that prior is only an educated guess and not some well researched position. This can be good because you might not have done the research yet.

eanzenberg 7y ago

Yeah, no thanks though. I don't want every rando adding "priors" that "feel" right to their analysis. Frequentist is straight forward. Both can (and are) abused to prove bias.

nightski 7y ago
Frequentist has a prior also though. The uniform distribution. In a sense this might be more biased as it doesn't always accurately describe the situation. Either way I believe the effect of a prior diminishes greatly pretty quickly as you acquire more data. It's only a factor if you have extremely small set of data.
- zozbot234 7y ago
  The uniform distribution ("flat") prior lets you interpret a maximum likelihood result as a maximum-a-posteriori (MAP) Bayesian point-estimate (implying a 0-or-1 loss function). One could argue that if you refrain from doing this and just stick to a literal application of the likelihood principle, you're not really depending on a flat prior. For that matter, what is a "flat" prior over the parameters also depends on what parameterization you're using. Results that are 'intuitive' under one parameterization may not be under a different one.
- btilly 7y ago
  Frequentist has a prior also though. The uniform distribution. No. Experimental design affect frequentist conclusions in a way that is inconsistent with _ANY_ prior. Here is a real life example. My aunt and uncle had 7 children. 6 boys and one girl. Were they biased towards having one gender over another? If the null hypothesis is that they aren't, the p-value that you get is easily calculated as 16/2^7 = 1/8 = 0.125. (There is 1 arrangement of 7 girls, 7 of 6 girls and a boy, 7 of 6 boys and a girl, and 1 of 7 boys for 16 equally likely arrangements.) If I add the fact that they planned to have children until they had a boy and a girl, then that changes the p-value. In fact there are only 4 ways that their first 7 children can come out to give evidence this strong against the prior. So the p-value is now 4/2^7 = 1/32 = 0.03125. However a Bayesian looks at this and says that no matter what prior you pick, the knowledge that they planned to have children until they had both a boy and a girl does not affect your posterior conclusion. It literally has nowhere to go in the formula and can't make a difference. Therefore the frequentist's differing conclusions are not consistent with ANY prior, implicit or not.
  - pdonis 7y ago
    > a Bayesian looks at this and says that no matter what prior you pick, the knowledge that they planned to have children until they had both a boy and a girl does not affect your posterior conclusion A Bayesian would say no such thing. A Bayesian would agree that the knowledge that they planned to have children until they had both a boy and a girl doesn't affect your prior: you still are picking how much probability mass you allocate to all of the possible odds of having a boy vs. a girl, and the couple's plans don't affect that. However, a Bayesian would also say that the knowledge that they planned to have children until they had both a boy and a girl significantly changes the likelihood ratio (or p-value, if you prefer to use that) associated with the observed data. And one of the advantages of Bayesianism is that it forces you to make that explicit as well. Notice, for example, that when you calculated the first p-value of 1/8, you implicitly assumed that the couple's plan was "have 7 children, no matter what gender each of them is". The sample space is therefore all possible arrangements of 7 children by gender, and the p-value is 1/8, as you say. But when you calculated the second p-value of 1/32, while you did change the count of arrangements, you failed to recognize that the sample space changed! Now the possibilities are not just all possible arrangements of 7 children (which is what you used), but all possible arrangements of up to 7 children (because the "stop condition" now is not when there are 7 children total, but when there is at least one child of each gender, and that could have happened at a number of children less than 7). So the correct p-value is not 4/2^7, but 4/(2^7 + 2^6 + 2^5 + 2^4 + 2^3 + 2^2) = 4/(2^8 - 2) = 2/127. A Bayesian, who has to calculate the p-value starting from the hypothesis, not the data, would not make that mistake. And Bayesianism does something else too: it forces you to recognize that the p-value is not actually the answer to the question you were asking! By the p-value criterion, at least with the typical threshold of 0.05, the null hypothesis (that your aunt and uncle are not biased towards having one gender) is rejected. But a Bayesian recognizes that the prior probability of the gender ratio, based on abundant previous evidence, is strongly peaked around 50-50, much more strongly peaked than data with a bias equivalent to a p-value of 2/127 can overcome. So the Bayesian is quite ready to accept that your aunt and uncle had no actual bias towards having boys, they just happened to be one of the statistical outliers that are to be expected given the huge number of humans who have children.
jules 7y ago
The difference between a frequentist and a Bayesian is that the latter admits that he picks a prior. A frequentist smushes together (1) the statistical assumptions (2) the approximations that make the problem computationally tractable and (3) the mathematical derivations, into one big mess. Just because you're not stating your assumptions doesn't mean there are none. Consider maximum likelihood estimation. It is not invariant under coordinate transformations. So which coordinates you pick is an assumption. In fact, with Bayesian estimation you can do the same thing: picking a prior is equivalent to picking the uniform prior in a different coordinate system. So frequentist estimation does involve picking a prior by picking a coordinate system, even if the frequentist does not admit this. Frequentist methods are conceptually anything but straightforward. The advantage of frequentist methods is that they are computationally tractable. Usually they are best understood as approximations to Bayesian methods. For instance, MLE can be viewed as the variational approximation to Bayes where the family of probability distributions is the family of point masses, and the prior is uniform.
- Akababa 7y ago
  What do you mean by coordinate transformation? MLE is invariant under parameter transformations because it's just the argmax of the likelihood.
  - jules 7y ago
    Indeed, it is the argmax of the likelihood, but the likelihood is not invariant under coordinate transformations. The quantity p(x)dx is invariant, not p(x). By picking a suitable coordinate transformation you can put the MLE on any value where the likelihood is not zero.
    - deleted 7y ago
      [deleted]
  - knzhou 7y ago
    Adding to the other comments, you still have prior-dependence on a more subtle level, because it depends on what hypotheses are allowed. Here's an extreme example. Consider flipping an apparently fair coin and getting "THHT". The hypothesis that the coin is fair gives this result with likelihood 1/16. The hypothesis that a worldwide government conspiracy has been formed with the sole purpose of ensuring this result... has a likelihood of 1. But nobody would ever declare this the MLE, because "government conspiracy" isn't one of the allowed options. But it isn't precisely because it's unlikely, i.e. because of your prior. Of course this is an extreme example, but there are more innocuous prior-based assumptions baked in too.
jefft255 7y ago
See my above comment for an example of a prior that doesn't just "feel" right.
Turing_Machine 7y ago
> I don't want every rando adding "priors" that "feel" right to their analysis. Conversely, maybe others see value in a technique that doesn't start from square one every single time. I mean, knowledge is a thing, yes?
- perl4ever 7y ago
  That's not the point. The point is there is no choice between having priors and not. There is only the choice between acknowledging priors versus doublethink, confusion, and deception. With how fashionable it is to talk about implicit bias, I wonder how those concerns intersect with the people attacking Bayesian approaches here.
wirrbel 7y ago
Let's say you perform a maximum-likelihood estimate, you still have an assumption baked in, that maximizing the likelihood given the data is the right way to make your estimate. In fact, it's very interesting to reconstruct a Bayesian prior for a maximum likelihood estimate. For example when you calculate probabilities for a binary event, 10 head flips, 8 tail flips. The ML estimate is 8/18 = 4/9. A Beta-Binomial bayesian model leads to a posterior distribution of Beta(a=8, b=10) with a mean of (8+a0)/(10 + 8 + a0 + b0), with a0 and b0 representing the prior distribution Beta(a0, b0). Now you can see that the maximum likelihood estimate is identical in this case to assuming a bayesian Prior of Beta(0, 0). I am not saying by this that frequentism is Bayesian inference in disguise, rather, you cannot escape the assumptions. Also, frequentism is not that straightforward, there are many kinds of frequentist estimators and it can be complicated to choose among them.
- zozbot234 7y ago
  The ML estimate is a posterior mode, assuming a flat prior. It's not immediately clear that it will always be possible to find a corresponding posterior mean. (From a Bayesian point of view, this is a difference in loss functions as opposed to priors over the parameters. With a posterior mean, you're making the optimal inference assuming a quadratic loss; a posterior mode is appropriate for a 0-or-1 loss.)
FeepingCreature 7y ago
That's why your analysis shouldn't report posterior probability but degree of shift in confidence.

madhadron 7y ago

> it makes it very explicit that you pick a prior But you don't, in general, pick a prior. You pick a procedure that has an expected loss under various conditions. It's one player game theory. If you happen to have a prior, then you can use it to choose a unique procedure that has minimal expected risk for that prior given the loss function, but even so that may not be what you want. For example, you may want a minimax procedure, which may be quite different from the Bayes procedure.

jbay808 7y ago
Minimax still requires a probability distribution, which means you need a prior. Edit: Based on the downvotes, I see my audience is not convinced. I'll repeat an explanation I posted a while ago. Probably should make this a blog post because I see this claim quite often. I'd love to know what book you read it in. -- In minimax regret, you have a set of available decisions D, and a set of possible states of nature N, and a utility U(D,N). Each state of nature also has a probability P(N) (which can be influenced by the decision too in some problems). States of nature include "interest rates rise 1%", "interest rates fall 1%", and "interest rates stay the same". Decisions include "invest in stocks" and "invest in bonds". Minimax regret proposes to ignore the probabilities P(N), instead suggesting a way to make a decision purely based on the utilities of the outcomes. But that is actually an illusion. Outside of math class word problems, we don't have N or U(D,N) handed to us on a silver platter. There is always an infinite range of possible states of nature, many of which have a probability approaching but never reaching zero, including states such as "win the lottery", "communist revolution", and "unexpected intergalactic nuclear war". In commonsense decision-making we don't include those states of nature in our decision matrix, because our common sense rules them out as being implausible before we even think about our options. You wouldn't choose to invest in bonds just because stocks have the most regret in the event of a communist takeover. So what actually happens is we intuitively apply some probability threshold that rules out states of nature falling below it from our consideration. Then we minimize max regret on the remaining "plausibly realistic" states of nature. Humans are so good at doing probability mentally that this step happens before we even realize it. But if you are writing code that makes decisions, you'll need to do it, and so you'll need to have at least a rough stab at the probability distributions.

mikorym 7y ago

Are all priors an application of Bayes's theorem? It is confusing to me that there is talk of Bayesian statistics vs. frequentist statistics when both are often used in conjunction. The classic example of a medical test with false positives and false negatives and the prior being incidence in the general population comes to mind. To me that is not just an example of Bayes, but a combination of frequentist statistics with Bayes's theorem. I also seem to recall that Bayes's theorem appears in a standard first year probability and statistics course.

eli_gottlieb 7y ago
Bayesian statistics: the parameters you want to infer are modeled as random variables with a non-empirical prior, and Bayes' rule is used to draw inferences. Frequentist statistics: you construct estimators for the parameters you care about, subject to appropriate loss/risk criteria, but without any explicit "prior knowledge". Frequentist statistics with Bayes' theorem: you use available empirical data, plus some exponential-family distribution, to construct an informed prior, then use Bayes' rule to update the prior on evidence. You use this Bayesian approach only for unobservable hypotheses, rather than for parameters which can be estimated. Machine learning: you stack lots and lots of polynomial regressors onto each-other and train them with a loss function until they predict well on the validation set.
- jbay808 7y ago
  What do you mean by a "non-empirical prior"?
  - chongli 7y ago
    Bayesian statistics is sometimes called subjectivist statistics. Probability in Bayesian statistics reflects your degree of belief in some potential outcome. If you conduct an experiment, you use Bayes’ theorem to update your degree of belief, which is now conditional on the outcome of your experiment. By quantifying your degree of belief in a prior, you give yourself some starting point (rather than just assuming 0 probability), even if that prior is only an educated guess and not some well researched position. This can be good because you might not have done the research yet.