6 ms·

It is criticized frequently, and study pre-registration has become the norm already in some fields.

CoastalCoder 3y ago
Thanks. Then maybe it's something else I'm having trouble with: I'd think that (the extent to which data support a particular model) is unrelated to (subjective details such as what hypothesis the researchers intended to test). IIUC, the criticism of P-hacking hinges on such subjective details. If the main goal is to find useful models, why would we care about the researchers' intent when originally designing the experiment?
- jncfhnb 3y ago
  It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias. If we are testing everything we can think of, we need to be more conservative about our statistical tests to reflect that. That is, helping to determine if what you found is actually useful or not. Further a lot of science finds things pass the statistical test but are still not useful, which is a different kind of problem. If you do an experiment and see something interesting you should probably follow up on it. But it’s the difference between saying “there’s a 5% chance that this is just sample bias” using basic methods and “there’s a 90% chance this is sample bias using methods that account for how many things we tried”. Who knows? Maybe your finding is novel, important, statistically strong, and useful, without you having planned to explore it. Maybe you accidentally found penicillin. The two big problems is that people don’t know how to do these calcs (a lot of which don’t really make any sense because they try to assert independence into things where independence is impossible) and they don’t mention the things they tried which didn’t work which makes evaluating their claims impossible.
  - CoastalCoder 3y ago
    > It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias. So is the fundamental issue that an experiment is probably designed to avoid bias for the data related to the hypothesis being tested, but probably isn't designed to avoid bias for other hypotheses you might generate once the results are in?
    - jncfhnb 3y ago
      The math required to interpret 20 results is different than the math required to interpret 1 result. It’s not a weakness of the experiment itself. On some level the math required to interpret 20 results may not actually exist. Think of sample bias as a dice roll. Every time you check a new hypothesis, you roll the die. If it rolls a 1, you get a false positive. Pre registering a hundred different things you want to test doesn’t make that better, but it does create confidence that you didn’t continually test new ideas one at a time until number 100 was something that looked good.

Thanks. Then maybe it's something else I'm having trouble with: I'd think that (the extent to which data support a particular model) is unrelated to (subjective details such as what hypothesis the researchers intended to test). IIUC, the criticism of P-hacking hinges on such subjective details. If the main goal is to find useful models, why would we care about the researchers' intent when originally designing the experiment?

jncfhnb 3y ago
It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias. If we are testing everything we can think of, we need to be more conservative about our statistical tests to reflect that. That is, helping to determine if what you found is actually useful or not. Further a lot of science finds things pass the statistical test but are still not useful, which is a different kind of problem. If you do an experiment and see something interesting you should probably follow up on it. But it’s the difference between saying “there’s a 5% chance that this is just sample bias” using basic methods and “there’s a 90% chance this is sample bias using methods that account for how many things we tried”. Who knows? Maybe your finding is novel, important, statistically strong, and useful, without you having planned to explore it. Maybe you accidentally found penicillin. The two big problems is that people don’t know how to do these calcs (a lot of which don’t really make any sense because they try to assert independence into things where independence is impossible) and they don’t mention the things they tried which didn’t work which makes evaluating their claims impossible.
- CoastalCoder 3y ago
  > It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias. So is the fundamental issue that an experiment is probably designed to avoid bias for the data related to the hypothesis being tested, but probably isn't designed to avoid bias for other hypotheses you might generate once the results are in?
  - jncfhnb 3y ago
    The math required to interpret 20 results is different than the math required to interpret 1 result. It’s not a weakness of the experiment itself. On some level the math required to interpret 20 results may not actually exist. Think of sample bias as a dice roll. Every time you check a new hypothesis, you roll the die. If it rolls a 1, you get a false positive. Pre registering a hundred different things you want to test doesn’t make that better, but it does create confidence that you didn’t continually test new ideas one at a time until number 100 was something that looked good.