john_pryan

6 ms·

1. 1.▲
   by john_pryan 4y ago

   I agree, the article seems to imply the plot in the original paper will always be an incorrect thing to do, instead of something which can have some issues in cases when we have inaccurate tests. I've gone back and updated the colab no
2. 2.▲
   by john_pryan 4y ago

   That's a nice spot about the decreasing CC as we increase accuracy! My hypothesis would be that some of the DK effect in the original paper may be down to an effect like this (as suggested in the original article) but that asserting it
3. 3.▲
   by john_pryan 4y ago

   For anyone who is interested in playing around with these charts, the various assumptions that under pin them etc. I've thrown together a colab notebook as a starting point. Observation: if you rank via true "skill" and assum