4 ms·

> The difference between bias and variance. But when you're bad at the skill and can't underestimate, they look the same. > That's the hypothesis that's being

> The difference between bias and variance. But when you're bad at the skill and can't underestimate, they look the same. > That's the hypothesis that's being tested And evidence from DK supports it.

omnicognate 4y ago
> But when you're bad at the skill and can't underestimate, they look the same. The (definitional) difference between bias and variance isn't related to do with whether you're bad at the skill or not. It's just mean vs variance of a probability distribution. If there's a good faith acknowledgement on your part that there's something here you're not getting then I'm very happy to try and help you understand it, and in the spirit of hn I'm assuming that is the case as you've claimed. I'm definitely not interested in any sort of motivated argument, though. If you're attached to the ideas you're putting forward here in some way I have no desire to try and dissuade you. Operating on the former assumption, I'm not really clear where the misunderstanding lies at this stage, but perhaps it would help if you were to expand on in what sense you think being "bad at the skill" would make bias and variance "look the same"?
- haberman 4y ago
  I'm watching this thread with interest and will try to restate my understanding of GP's argument, by means of an example. If a person's true skill is 5 on a 1-100 point scale, but the person is completely unaware of their true skill and will guess randomly, then their estimate will bias heavily in the direction of overestimating their skill, even if they were not intrinsically motivated to overestimate their skill, simply because far more of the available guesses are higher than their true ability. In other words, the available probability space itself biases in the direction of overestimating their ability, for those people. Is that right andersource? I don't know the statistical right answer here, but curious to know.
  - omnicognate 4y ago
    It's worth reading the discussion between author and Nicolas Bonneel that starts with the first comment below the article. The author's explanation is very helpful regarding this point. The main point is that in the paper's randomly generated numbers example, the DK effect disappears if you measure the actual "skill" and the "prediction error" in separate, independent experiments. In the example if you take a "person" and conduct the test you get a totally random result, and you get another, independent totally random result if you test them again. If you perform your "actual skill" measurement using one of those test runs and your "skill estimation error" measurement using another, the DK effect disappears completely. So, to the extent the result of your skills test has any "noisiness" to it, if you analyse it the way Dunning & Kruger did, the autocorrelation resulting from using the same sample of that noise in the two things you're trying to assess the relationship between will show up as a powerful DK effect, and can easily swamp any actual correlations in the underlying distribution. Edit: Also worth mentioning footnote 3 on the article, which points out that the use of quantiles introduces a separate bias for the same reason you mention (about there being a minimum and maximum score).
- andersource 4y ago
  Here's my main point of confusion - what does the random data experiment have to do with the DK results? As stated elsewhere DK has 2 claims: 1. Low-skilled people overestimate their performance and skilled people underestimate their performance 2. Skill correlates with self-assessment accuracy My first issue with the article is that it implies that since we get effect #1 with random data, that invalidates the respective DK conclusion. This IMO is misleading because random data represents a null model that is very different from my intuitive null model, that of people generally capable of assessing their skills (which I truly believe). My second issue is that there's no relationship between effect 2 and the random data experiment, which doesn't exhibit anything of the sort. We can have a discussion about the cited papers and effect 2 as the reproduced plot doesn't show density and density plots from the paper do seem to support DK, but that's not my main gripe with the article.
  - omnicognate 4y ago
    As far as I can see (having checked wiki and the abstract of the original paper - I'm no expert on this) the DK effect is only the first of those claims. However it sounds like claim 2 is less significant here anyway. Re claim 1 the random numbers example is "all noise, no signal" and I can see the objection that a more convincing example might be to demonstrate the "false" DK effect in an example that does have some signal (i.e. a positive relationship between actual and estimated skill), but that is easy to do and I hope you'll be able to see why if you see my reply at https://news.ycombinator.com/item?id=31042619 https://news.ycombinator.com/item?id=31042619 and read the comments under the article I mentioned there. The point is that the DK analysis involves comparing two things which both contain the same single sample from a noise source. Pure noise like the random numbers in the example displays a powerful DK effect due to autocorrelation that says nothing interesting (just that a single random sample of noise is correlated with itself), and that powerful effect can swamp any actual relationships in the distributions. To avoid that effect appearing, you have to make sure that if the two things you are comparing contain samples of a single noise source they are separate, independent samples of it. The experiment with the education level groups achieves this because the education level is "measured" as a separate event from the "actual" skill measurement so they have separate noise sources (and even if they didn't the noise source would have been sampled separately and independently). I have to say, during the discussion above I hadn't thought through it deeply enough to grok this level of it, and while pondering your last comment I went through a phase of "hang on, am I actually understanding this myself?", so I apologise and retract any suggestion of bad faith.

> The difference between bias and variance. But when you're bad at the skill and can't underestimate, they look the same. > That's the hypothesis that's being tested And evidence from DK supports it.

omnicognate 4y ago

> But when you're bad at the skill and can't underestimate, they look the same. The (definitional) difference between bias and variance isn't related to do with whether you're bad at the skill or not. It's just mean vs variance of a probability distribution. If there's a good faith acknowledgement on your part that there's something here you're not getting then I'm very happy to try and help you understand it, and in the spirit of hn I'm assuming that is the case as you've claimed. I'm definitely not interested in any sort of motivated argument, though. If you're attached to the ideas you're putting forward here in some way I have no desire to try and dissuade you. Operating on the former assumption, I'm not really clear where the misunderstanding lies at this stage, but perhaps it would help if you were to expand on in what sense you think being "bad at the skill" would make bias and variance "look the same"?

haberman 4y ago
I'm watching this thread with interest and will try to restate my understanding of GP's argument, by means of an example. If a person's true skill is 5 on a 1-100 point scale, but the person is completely unaware of their true skill and will guess randomly, then their estimate will bias heavily in the direction of overestimating their skill, even if they were not intrinsically motivated to overestimate their skill, simply because far more of the available guesses are higher than their true ability. In other words, the available probability space itself biases in the direction of overestimating their ability, for those people. Is that right andersource? I don't know the statistical right answer here, but curious to know.
- omnicognate 4y ago
  It's worth reading the discussion between author and Nicolas Bonneel that starts with the first comment below the article. The author's explanation is very helpful regarding this point. The main point is that in the paper's randomly generated numbers example, the DK effect disappears if you measure the actual "skill" and the "prediction error" in separate, independent experiments. In the example if you take a "person" and conduct the test you get a totally random result, and you get another, independent totally random result if you test them again. If you perform your "actual skill" measurement using one of those test runs and your "skill estimation error" measurement using another, the DK effect disappears completely. So, to the extent the result of your skills test has any "noisiness" to it, if you analyse it the way Dunning & Kruger did, the autocorrelation resulting from using the same sample of that noise in the two things you're trying to assess the relationship between will show up as a powerful DK effect, and can easily swamp any actual correlations in the underlying distribution. Edit: Also worth mentioning footnote 3 on the article, which points out that the use of quantiles introduces a separate bias for the same reason you mention (about there being a minimum and maximum score).
andersource 4y ago
Here's my main point of confusion - what does the random data experiment have to do with the DK results? As stated elsewhere DK has 2 claims: 1. Low-skilled people overestimate their performance and skilled people underestimate their performance 2. Skill correlates with self-assessment accuracy My first issue with the article is that it implies that since we get effect #1 with random data, that invalidates the respective DK conclusion. This IMO is misleading because random data represents a null model that is very different from my intuitive null model, that of people generally capable of assessing their skills (which I truly believe). My second issue is that there's no relationship between effect 2 and the random data experiment, which doesn't exhibit anything of the sort. We can have a discussion about the cited papers and effect 2 as the reproduced plot doesn't show density and density plots from the paper do seem to support DK, but that's not my main gripe with the article.
- omnicognate 4y ago
  As far as I can see (having checked wiki and the abstract of the original paper - I'm no expert on this) the DK effect is only the first of those claims. However it sounds like claim 2 is less significant here anyway. Re claim 1 the random numbers example is "all noise, no signal" and I can see the objection that a more convincing example might be to demonstrate the "false" DK effect in an example that does have some signal (i.e. a positive relationship between actual and estimated skill), but that is easy to do and I hope you'll be able to see why if you see my reply at https://news.ycombinator.com/item?id=31042619 https://news.ycombinator.com/item?id=31042619 and read the comments under the article I mentioned there. The point is that the DK analysis involves comparing two things which both contain the same single sample from a noise source. Pure noise like the random numbers in the example displays a powerful DK effect due to autocorrelation that says nothing interesting (just that a single random sample of noise is correlated with itself), and that powerful effect can swamp any actual relationships in the distributions. To avoid that effect appearing, you have to make sure that if the two things you are comparing contain samples of a single noise source they are separate, independent samples of it. The experiment with the education level groups achieves this because the education level is "measured" as a separate event from the "actual" skill measurement so they have separate noise sources (and even if they didn't the noise source would have been sampled separately and independently). I have to say, during the discussion above I hadn't thought through it deeply enough to grok this level of it, and while pondering your last comment I went through a phase of "hang on, am I actually understanding this myself?", so I apologise and retract any suggestion of bad faith.