8 ms·

It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert). A calibrated forecast means that if you say ther

by wosk 2mo ago

It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert). A calibrated forecast means that if you say there is a 20% chance of rain, then it actually rains 20% of the time. It’s a desired feature, but not the only one (e.g. you could be calibrated by stating: Chick-fil-A is open every day except Monday, but your forecast will always be wrong on Sunday and Monday). So if 1. you are Bayesian (you state your beliefs) 2. and coherent (the laws of probability apply, so e.g. if P(A) = 0.4, then P(not A) cannot be anything other than 0.6): and you are predicting something (e.g rain tomorrow), then if you believe it will rain with probability 0.7 but you are 80% sure of your belief, you won’t say 0.7; you will say something else: 0.8 × 0.7 + 0.2 × something_you_believe = 0.58. Coherence forces you to collapse your uncertainty into your probability at each forecast. This theorem shows that, over many forecasts, in your belief system you are certain to be producing a calibrated forecast: your current beliefs assign probability 1 to the proposition that your future forecasts will be calibrated. But that can’t be, which is the paradox. So Bayesianism is too strong compared to how scientists reason, because scientists always think their model can have an error.

PaulHoule 2mo ago
I didn't see a real Bayesian point of view in that article. A Bayesian does not give you a probability estimate they give you a probability distribution for the probability! Like in Star Trek Spock is always saying something like "Captain, we have a 15.31% chance of surviving this mission" which is a ridiculous example of precision without accuracy. [1] If you observe a coin flipped 100 times and it came up heads 65 times it is not a crazy point estimate to say it has a 65% chance of coming up heads but this is just one sample and if you did it another time maybe it comes up 61 or 68 times. You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36) since that has the "error bars" built in, can be updated if you get more samples, etc. [1] ... and you know he underestimates survival probabilities the same way Scotty overestimates how long it will take to fix the engines
- clickety_clack 2mo ago
  Was Spock poorly calibrated?
  - PaulHoule 2mo ago
    If he was well calibrated there is no way they would have made it through 79 episodes!
    - bryanrasmussen 2mo ago
      Spock had not realized that James Kirk emitted a psionic reality distortion field through higher dimensional "luck", if he had he would have been a bit more relaxed.
      - oliculipolicula 2mo ago
        To his defence. in the trekkie timeline, neither Vulcans nor humans had discovered the Metropolis algorithm, so Spock had (for sure) nothing with which to calibrate that
- wosk 2mo ago
  The \pi_i in the paper is not the estimate of a latent parameter. It is the predictive probability of the event, which is a single number by necessity in a binary challenge. It's the integration of a distribution function which can contains very complex distributions: in my example something_you_believe can be a probability distribution. So everything in the paper is distribution and when you forecast for a binary event, you give a number which is the expectation of that distribution. This is a probabilistic forecast. If you were to give a probabilistic forecast for a continuous quantity, then yes you would give in a distribution, as in section 4.2
  - edbaskerville 2mo ago
    Emphasizing this response. Bayesian models can always produce simple probabilities if you ask them to. E.g., given this data, what is the probability that the next flip is heads? The fact that the model is represented as a distribution over Bernoulli parameter p doesn't contradict this: you just integrate over the posterior.
mitthrowaway2 2mo ago
> if you believe it will rain with probability 0.7 but you are 80% sure of your belief What does this even mean...? If I believe it will rain with probability 0.7, that 0.7 figure should already be taking into account the sum total of all of my uncertainty over all of my beliefs: my trust in the weather forecast, my past experience with the local area in this season, my certainty that the earth will continue to exist tomorrow. Bayesians of course accept that their models can have errors, and if they're doing a good job they'll factor all of the most influential ones into the probability calculation itself.
- wosk 2mo ago
  > What does this even mean...? You are making the same point that I did, if you read the rest. 0.7 at 80% is an intermediate step in formulating a forecast, and the point is to show that you cannot stop there and have to include everything in your number, like you have said. In your words, the intermediate steps are what you list (trust in weather forecasts, past experience,…).
  - mitthrowaway2 2mo ago
    No, what I'm saying is that 0.7 is the ending point of the calculation, and there is no 80% associated with it. The 0.7 number already factors in my uncertainty in all of my knowledge about whether it will rain tomorrow. Saying that you're 80% sure of your 0.7 probability forecast throws a TypeErrorException.
    - wosk 2mo ago
      yes and in my example the ending point of the calculation is 0.58. I make an intermediate step explicit in this derivation because it is important for the understanding of the problem. This kind of intermediate step happens a lot in forecasting competitions, where participants are asked for their forecasts and their confidence in their forecasts. I want to show here that you need to include that in your forecast (all you belief) and not keep it separated.

by wosk 2mo ago

PaulHoule 2mo ago

I didn't see a real Bayesian point of view in that article. A Bayesian does not give you a probability estimate they give you a probability distribution for the probability! Like in Star Trek Spock is always saying something like "Captain, we have a 15.31% chance of surviving this mission" which is a ridiculous example of precision without accuracy. [1] If you observe a coin flipped 100 times and it came up heads 65 times it is not a crazy point estimate to say it has a 65% chance of coming up heads but this is just one sample and if you did it another time maybe it comes up 61 or 68 times. You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36) since that has the "error bars" built in, can be updated if you get more samples, etc. [1] ... and you know he underestimates survival probabilities the same way Scotty overestimates how long it will take to fix the engines

clickety_clack 2mo ago
Was Spock poorly calibrated?
- PaulHoule 2mo ago
  If he was well calibrated there is no way they would have made it through 79 episodes!
  - bryanrasmussen 2mo ago
    Spock had not realized that James Kirk emitted a psionic reality distortion field through higher dimensional "luck", if he had he would have been a bit more relaxed.
    - oliculipolicula 2mo ago
      To his defence. in the trekkie timeline, neither Vulcans nor humans had discovered the Metropolis algorithm, so Spock had (for sure) nothing with which to calibrate that
wosk 2mo ago
The \pi_i in the paper is not the estimate of a latent parameter. It is the predictive probability of the event, which is a single number by necessity in a binary challenge. It's the integration of a distribution function which can contains very complex distributions: in my example something_you_believe can be a probability distribution. So everything in the paper is distribution and when you forecast for a binary event, you give a number which is the expectation of that distribution. This is a probabilistic forecast. If you were to give a probabilistic forecast for a continuous quantity, then yes you would give in a distribution, as in section 4.2
- edbaskerville 2mo ago
  Emphasizing this response. Bayesian models can always produce simple probabilities if you ask them to. E.g., given this data, what is the probability that the next flip is heads? The fact that the model is represented as a distribution over Bernoulli parameter p doesn't contradict this: you just integrate over the posterior.

mitthrowaway2 2mo ago

> if you believe it will rain with probability 0.7 but you are 80% sure of your belief What does this even mean...? If I believe it will rain with probability 0.7, that 0.7 figure should already be taking into account the sum total of all of my uncertainty over all of my beliefs: my trust in the weather forecast, my past experience with the local area in this season, my certainty that the earth will continue to exist tomorrow. Bayesians of course accept that their models can have errors, and if they're doing a good job they'll factor all of the most influential ones into the probability calculation itself.

wosk 2mo ago
> What does this even mean...? You are making the same point that I did, if you read the rest. 0.7 at 80% is an intermediate step in formulating a forecast, and the point is to show that you cannot stop there and have to include everything in your number, like you have said. In your words, the intermediate steps are what you list (trust in weather forecasts, past experience,…).
- mitthrowaway2 2mo ago
  No, what I'm saying is that 0.7 is the ending point of the calculation, and there is no 80% associated with it. The 0.7 number already factors in my uncertainty in all of my knowledge about whether it will rain tomorrow. Saying that you're 80% sure of your 0.7 probability forecast throws a TypeErrorException.
  - wosk 2mo ago
    yes and in my example the ending point of the calculation is 0.58. I make an intermediate step explicit in this derivation because it is important for the understanding of the problem. This kind of intermediate step happens a lot in forecasting competitions, where participants are asked for their forecasts and their confidence in their forecasts. I want to show here that you need to include that in your forecast (all you belief) and not keep it separated.