8 ms·

Funny you bring this up. Back in the day when I was like 15 and DC++ was still a thing, I used to browse people's shared folders. One day I came across a file

Funny you bring this up. Back in the day when I was like 15 and DC++ was still a thing, I used to browse people's shared folders. One day I came across a file called "the paradox of false positive". It was a 1 pager that described how a machine which is 99.9% accurate at identifying terrorists would be completely useless due to this false positive base rate fallacy you're describing. It really stuck with me throughout the years. It's kind o remarkable how even a 99.9% accurate heuristic is insufficient at scale. Which begs the question: lets assume the intentions are pure (which we know they're not but lets be generous), what other options are there when 99.9% heuristic is not good enough? how do you design systems when they're guaranteed to fail as they scale up? edit: and what do you know, I just saw this as I scrolled down on HN https://news.ycombinator.com/item?id=48816959 https://news.ycombinator.com/item?id=48816959

deleted 2mo ago
[deleted]
nanis 2mo ago
Because no one around me understood how to calculate false positive/false negative probabilities, I put together this calculator[1] in 2020. [1]: https://www.covid2020.icu/false-positive-false-negative-simulator/ https://www.covid2020.icu/false-positive-false-negative-simu...
m12k 2mo ago
The intuition I've built is that you can't talk about a false positive rate being high or low on its own - it's always relative to the actual occurrence rate of positives in the tested population. E.g. if there's a 1 in 10000 risk of a false positive, but real positives also are only 1 out of 10000 tested cases, then a positive case will have a 50/50 chance of being a false positive (because for every 10000 tests, you'll have on average one false positive and one real positive). So a false positive rate can only be said to be low if it's significantly lower than the real occurrence rate of positives.
- ablob 2mo ago
  The mentioned accuracy in the comment you are replying to already encapsulates the relation of true positives to false positives.
  - cortesoft 2mo ago
    It really doesn’t, and it is easy to demonstrate by using an extreme example. Suppose I invent a device that can detect whether there is a giant invisible dragon living in your house, and it has an accuracy of 99.999% Now, I use it in your house and it tells me there is an invisible dragon… so what are the chances that there is a dragon in your house? Based on your statement, it would be 99.999% likely that there is an invisible dragon in your house. However, we actually know that there is a 0% chance there is an invisible dragon, so even with the positive test result we still know there is a 0% chance a dragon is there.
  - fc417fc802 2mo ago
    No I don't believe it does. I interpret 99.9% accurate to mean 1 in 1000 false positives. If 0.1% of your population are terrorists that means each alert has a 50% chance of being correct. That's nowhere near good enough to fully automate things but it is quite reasonable assuming this is merely information provided to a human agent. Whereas if only 0.001% of your population are terrorists then 99 out of 100 alerts are false positives at which point the system is well on its way to being useless. There is an important difference between scenarios where we care about the relative versus absolute frequency of errors.
    - ablob 2mo ago
      You're right it doesn't. At least not completely. I was thinking about precision (i.e.: if the test is positive, what are the odds that its prediction is true). It turns out, that accuracy is not defined as "true positive / (true pos. + true neg.)", but "correct predictions / all predictions". The whole point of OP's statement: "It's kind o remarkable how even a 99.9% accurate heuristic is insufficient at scale.", which you actually support with your example. > There is an important difference between scenarios where we care about the relative versus absolute frequency of errors. The context is chat control without probable cause over the whole population of Europe with a low prevalence. My point, and presumably that of OP, is that even a small relative frequency of errors will yield an unsustainably high absolute frequncy of errors. > This is merely information provided to a human agent. It will be in theory. In practice the human agent will just forward the decision. A human agent is not sufficient; you need to test only with probable cause for the kind of scenario we're talking about. The exact opposite of "Chat Control 1.0 and 2.0". P.S.: The comment I originally replied to choose a very convoluted way of saying that the false discovery rate of the test matters for a proper evaluation. Both you and they explain this by throwing numbers without context in combination with slightly inaccurate definitions. I got the definitions mixed up differently, which led to this follow-up.
Asmod4n 2mo ago
The system we got for this is called parenting. And there is a saying where I grew up: you need a village to raise a kid, I feel like we lost track of that and feel the issues of that now. Btw, von der leyen is trying to get stuff like this written down as laws since 2009, it got her the nickname Zensursula.
- joe_mamba 2mo ago
  >Btw, von der leyen is trying to get stuff like this written down as laws since 2009, it got her the nickname Zensursula. And Germans and Europeans looked at that and thought the best place for her is leading the EU?! Remind me again how she got elected in that position? Because it seems like the entire EU population knew her being infamous for that, except for the few elites who appointed her there via "democratic process" to the head of the EU.
  - arrowsmith 2mo ago
    The president of the European Commission is “elected” through a thin pretence of democracy that the people of Europe have effectively no control over, and mostly pay no attention to. If you think she’s there because the greater public decided she’s the best person for the job then you don’t know how the EU works. Also most of the EU population don’t know her for anything at all. I’d be surprised if more than 50% of Europeans could name her.
  - kvgr 2mo ago
    Not a single person that is not attached to EU voted for he. She is second hand vote. These roles should all be result of direct vote. This way you only get votes by people who are sucking the money of Eu parliament or. The only position people vote for is EP. And that % is so small, that if they ask the people who didn't vote if they want it, they would have to tear it down. I am not against EU cooperation, mainly in external security and free market economy. But the system we have is not very democratic, and def not very representative of people. They act like demigods, elected by parliament with no real consequences of their actions.
    - 2mo ago

deleted 2mo ago

[deleted]

nanis 2mo ago

Because no one around me understood how to calculate false positive/false negative probabilities, I put together this calculator[1] in 2020. [1]: https://www.covid2020.icu/false-positive-false-negative-simulator/ https://www.covid2020.icu/false-positive-false-negative-simu...

m12k 2mo ago

The intuition I've built is that you can't talk about a false positive rate being high or low on its own - it's always relative to the actual occurrence rate of positives in the tested population. E.g. if there's a 1 in 10000 risk of a false positive, but real positives also are only 1 out of 10000 tested cases, then a positive case will have a 50/50 chance of being a false positive (because for every 10000 tests, you'll have on average one false positive and one real positive). So a false positive rate can only be said to be low if it's significantly lower than the real occurrence rate of positives.

ablob 2mo ago
The mentioned accuracy in the comment you are replying to already encapsulates the relation of true positives to false positives.
- cortesoft 2mo ago
  It really doesn’t, and it is easy to demonstrate by using an extreme example. Suppose I invent a device that can detect whether there is a giant invisible dragon living in your house, and it has an accuracy of 99.999% Now, I use it in your house and it tells me there is an invisible dragon… so what are the chances that there is a dragon in your house? Based on your statement, it would be 99.999% likely that there is an invisible dragon in your house. However, we actually know that there is a 0% chance there is an invisible dragon, so even with the positive test result we still know there is a 0% chance a dragon is there.
- fc417fc802 2mo ago
  No I don't believe it does. I interpret 99.9% accurate to mean 1 in 1000 false positives. If 0.1% of your population are terrorists that means each alert has a 50% chance of being correct. That's nowhere near good enough to fully automate things but it is quite reasonable assuming this is merely information provided to a human agent. Whereas if only 0.001% of your population are terrorists then 99 out of 100 alerts are false positives at which point the system is well on its way to being useless. There is an important difference between scenarios where we care about the relative versus absolute frequency of errors.
  - ablob 2mo ago
    You're right it doesn't. At least not completely. I was thinking about precision (i.e.: if the test is positive, what are the odds that its prediction is true). It turns out, that accuracy is not defined as "true positive / (true pos. + true neg.)", but "correct predictions / all predictions". The whole point of OP's statement: "It's kind o remarkable how even a 99.9% accurate heuristic is insufficient at scale.", which you actually support with your example. > There is an important difference between scenarios where we care about the relative versus absolute frequency of errors. The context is chat control without probable cause over the whole population of Europe with a low prevalence. My point, and presumably that of OP, is that even a small relative frequency of errors will yield an unsustainably high absolute frequncy of errors. > This is merely information provided to a human agent. It will be in theory. In practice the human agent will just forward the decision. A human agent is not sufficient; you need to test only with probable cause for the kind of scenario we're talking about. The exact opposite of "Chat Control 1.0 and 2.0". P.S.: The comment I originally replied to choose a very convoluted way of saying that the false discovery rate of the test matters for a proper evaluation. Both you and they explain this by throwing numbers without context in combination with slightly inaccurate definitions. I got the definitions mixed up differently, which led to this follow-up.

Asmod4n 2mo ago

The system we got for this is called parenting. And there is a saying where I grew up: you need a village to raise a kid, I feel like we lost track of that and feel the issues of that now. Btw, von der leyen is trying to get stuff like this written down as laws since 2009, it got her the nickname Zensursula.

joe_mamba 2mo ago
>Btw, von der leyen is trying to get stuff like this written down as laws since 2009, it got her the nickname Zensursula. And Germans and Europeans looked at that and thought the best place for her is leading the EU?! Remind me again how she got elected in that position? Because it seems like the entire EU population knew her being infamous for that, except for the few elites who appointed her there via "democratic process" to the head of the EU.
- arrowsmith 2mo ago
  The president of the European Commission is “elected” through a thin pretence of democracy that the people of Europe have effectively no control over, and mostly pay no attention to. If you think she’s there because the greater public decided she’s the best person for the job then you don’t know how the EU works. Also most of the EU population don’t know her for anything at all. I’d be surprised if more than 50% of Europeans could name her.
- kvgr 2mo ago
  Not a single person that is not attached to EU voted for he. She is second hand vote. These roles should all be result of direct vote. This way you only get votes by people who are sucking the money of Eu parliament or. The only position people vote for is EP. And that % is so small, that if they ask the people who didn't vote if they want it, they would have to tear it down. I am not against EU cooperation, mainly in external security and free market economy. But the system we have is not very democratic, and def not very representative of people. They act like demigods, elected by parliament with no real consequences of their actions.
  - 2mo ago