6 ms·

hm. upon reading the paper, this is sounding very suspicious. > The study was truly blind. Although the observers were acquainted with our previous studies on

hm. upon reading the paper, this is sounding very suspicious. > The study was truly blind. Although the observers were acquainted with our previous studies on magnetic alignment in animals and could have consciously or unconsciously biased the results, no one, not even the coordinators of the study, hypothesized that expression of alignment could have been affected by the geomagnetic situation, and particularly by such subtle changes of the magnetic declination. The idea leading to the discovery of the correlation emerged after sampling was closed and the first statistical analyses (with rather negative results, cf.Figure 1) had been performed. Like, am I reading this wrong, or are they straight-up saying "we couldn't achieve statistical significance on our original hypothesis, so we just went fishing for correlations until one of them came up significant, and it turned out to be magnetic declination"?

lonelappde 7y ago
Yes, that's a straight up admission of p-hacking. Also, the weasel word "truly" signifies deceit.
- eganist 7y ago
  I don't think that's necessarily a bad thing, though my opinion might've been influenced somewhat by 538. https://fivethirtyeight.com/features/science-isnt-broken/ https://fivethirtyeight.com/features/science-isnt-broken/
ncmncm 7y ago
Why not? Science that insists on hypotheses written down beforehand is cargo-cult science. Observation is the first and most productive science. Double-blind experiments are to cement gains.
- lonelappde 7y ago
  You are confusing hypothesis generation with hypothesis testing. Both are science, but only one is a reliable way to determine truth.
  - throwawayhhakdl 7y ago
    Probable claims. Not truth.
    - lupire 7y ago
      In the non-Platonic real world, truth is claims that we believe have high probability.
- BenoitEssiambre 7y ago
  Not if you want to claim statistical significance. The math behind this method is based on defining the hypothesis before seeing the data (and even then it's usually very weak evidence of a tiny signal within the noise).
- MrEldritch 7y ago
  basically, because once you start trying multiple hypotheses on the same dataset, the math used to determine "is this conclusion real, or am I just fooling myself" begins to break down. The statistical significance threshold usually used is p<0.05, meaning that something is (generally, this is beginning to change since the replication crisis) considered to be a real discovery if it has less than a 1/20 chance of being a false positive under the chosen model. As soon as you start trying multiple hypotheses, then that 1/20 chance of being a false positive begins to become meaningless. If you can just keep rolling d20s until one of them comes up with a critical hit, then you can easily generate false positives that still look very robust. This is exactly the sort of bad science - p-hacking, fishing expeditions, and the garden of forking paths - that led to the replication crisis. (And that makes sense, as this paper is from 2013, and predates the widespread discovery of the crisis)
  - ORioN63 7y ago
    Thanks! For someone that didn't understand why this was considered p-hacking, that made a whole lot of sense.
  - ncmncm 7y ago
    p<0.05 is also cargo-cult science, and is much more responsible for the replication crisis -- along with biased sampling (pop. 18-22 yo US psych students). It is also why we see repeated, spurious insistence that anti-depressants don't do anything. Experiment design is a subtle skill.
  - dodobirdlord 7y ago
    The math continues to work out as long as you use the right approach. You have to collect twice as much data, and then set half of it aside at random without examining it. Then you can do whatever perverse p-hacking multi-modeling curve-fitting whatever to the half you kept until you reach a hypothesis, then check it against the half you set aside to recover the statistical significance you lost by using techniques that may have overfit the first half. Unsurprisingly, the math works out because this approach is isomorphic to collecting the first half, studying it to form a hypothesis, then conducting a proper pre-hypothesized experiment to collect the second half. Validation via holdout sets is the same approach used in machine learning and elsewhere to prevent models from overfitting data.
- yodon 7y ago
  No. If you collect data and then hunt for "significant" results in it you are guaranteed to find spurious results. This is one of the most basic truths of statistics.
- shanemhansen 7y ago
  xkcd explains it better than I can. Basically if you pick p values that give 95% certainty 20 times you're probably going to "discover" at least one falsehood. https://xkcd.com/882/ https://xkcd.com/882/
- stygiansonic 7y ago
  You seem to be under the impression that a study like this gives a hard "yes/no" answer as to whether some hypothesis is true. That is not the case, nor is it ever the case with most studies like these. Instead, you need to do some sort of statistical hypothesis test. As other comments have pointed out, once you start testing multiple hypothesis on the same dataset, you cannot apply the same significance threshold that you would if you had just begun with a single hypothesis before observing the data. Instead, you need to apply some sort of correction that takes into account the number of hypothesis being tested: https://en.wikipedia.org/wiki/Family-wise_error_rate#Controlling_procedures https://en.wikipedia.org/wiki/Family-wise_error_rate#Control...
rossdavidh 7y ago
Well, it would be p-hacking if you tried 1,032 different hypotheses until you got one that passed your threshold. There's quite a lot of scientific history (e.g. Kepler's discovery that planets went in elliptical orbits) that would have to be thrown out if you decided you could never use data for anything other than the original hypothesis. Kepler didn't even collect the data, much less collect it with the idea that the planetary orbits were elliptical. Having said that, the results smells (pun intended) bad, just because I cannot think of any plausible reason for a non-migrating animal to align with the magnetic field, when defecating or at any other time.
- MrEldritch 7y ago
  This is true, and a valid point. The way they phrased it does make me feel more than a little suspicious, nonetheless. (Besides, there's some other oddity there, like that apparently the alignment only matters when the magnetic field is calm)
  - petschge 7y ago
    When the magnetic field is non-calm that is probably due to a space weather event that is geo-effective and inducing large currents in the ground. The local magnetic field an then be significantly distorted depending on local conductivity. So to me that is not an oddity.
  - austinjp 7y ago
    You're right to still feel suspicious. Who's to say they didn't try 1,000 different post-hoc ideas? They declare only one, the may have been others. I'd be looking for preceding research and any published protocols, if I wasn't on mobile and didn't think it would be fruitless.
- monocasa 7y ago
  I mean, humans can detect magnetic fields as well. There's even languages without relative egocentric positions like left/right, only north south east west. Given all that, I don't think it's out there that dogs sense it and like being aligned when they're trying to poop. https://www.sciencemag.org/news/2019/03/humans-other-animals-may-sense-earth-s-magnetic-field https://www.sciencemag.org/news/2019/03/humans-other-animals... https://en.wikipedia.org/wiki/Guugu_Yimithirr_language https://en.wikipedia.org/wiki/Guugu_Yimithirr_language
Reason077 7y ago
> "The study was truly blind." I'd argue the study would only be truly blind if the dogs were blind. If blind dogs also oriented themselves north-south, then that would prove that they weren't using visual cues for alignment, such as the position of the sun.
- philosopher1234 7y ago
  You can feel the heat of the sun
marcosdumay 7y ago
P-hacking only apply to proofs. This study does two things, it falsifies a previous hypothesis fair and square, with no p-hacking, and it postulates another hypothesis, an activity where the concept of p-hacking does not even apply. The only wrong party here is the one that reported the study found something.

lonelappde 7y ago

Yes, that's a straight up admission of p-hacking. Also, the weasel word "truly" signifies deceit.

eganist 7y ago
I don't think that's necessarily a bad thing, though my opinion might've been influenced somewhat by 538. https://fivethirtyeight.com/features/science-isnt-broken/ https://fivethirtyeight.com/features/science-isnt-broken/

ncmncm 7y ago

Why not? Science that insists on hypotheses written down beforehand is cargo-cult science. Observation is the first and most productive science. Double-blind experiments are to cement gains.

lonelappde 7y ago
You are confusing hypothesis generation with hypothesis testing. Both are science, but only one is a reliable way to determine truth.
- throwawayhhakdl 7y ago
  Probable claims. Not truth.
  - lupire 7y ago
    In the non-Platonic real world, truth is claims that we believe have high probability.
BenoitEssiambre 7y ago
Not if you want to claim statistical significance. The math behind this method is based on defining the hypothesis before seeing the data (and even then it's usually very weak evidence of a tiny signal within the noise).
MrEldritch 7y ago
basically, because once you start trying multiple hypotheses on the same dataset, the math used to determine "is this conclusion real, or am I just fooling myself" begins to break down. The statistical significance threshold usually used is p<0.05, meaning that something is (generally, this is beginning to change since the replication crisis) considered to be a real discovery if it has less than a 1/20 chance of being a false positive under the chosen model. As soon as you start trying multiple hypotheses, then that 1/20 chance of being a false positive begins to become meaningless. If you can just keep rolling d20s until one of them comes up with a critical hit, then you can easily generate false positives that still look very robust. This is exactly the sort of bad science - p-hacking, fishing expeditions, and the garden of forking paths - that led to the replication crisis. (And that makes sense, as this paper is from 2013, and predates the widespread discovery of the crisis)
- ORioN63 7y ago
  Thanks! For someone that didn't understand why this was considered p-hacking, that made a whole lot of sense.
- ncmncm 7y ago
  p<0.05 is also cargo-cult science, and is much more responsible for the replication crisis -- along with biased sampling (pop. 18-22 yo US psych students). It is also why we see repeated, spurious insistence that anti-depressants don't do anything. Experiment design is a subtle skill.
- dodobirdlord 7y ago
  The math continues to work out as long as you use the right approach. You have to collect twice as much data, and then set half of it aside at random without examining it. Then you can do whatever perverse p-hacking multi-modeling curve-fitting whatever to the half you kept until you reach a hypothesis, then check it against the half you set aside to recover the statistical significance you lost by using techniques that may have overfit the first half. Unsurprisingly, the math works out because this approach is isomorphic to collecting the first half, studying it to form a hypothesis, then conducting a proper pre-hypothesized experiment to collect the second half. Validation via holdout sets is the same approach used in machine learning and elsewhere to prevent models from overfitting data.
yodon 7y ago
No. If you collect data and then hunt for "significant" results in it you are guaranteed to find spurious results. This is one of the most basic truths of statistics.
shanemhansen 7y ago
xkcd explains it better than I can. Basically if you pick p values that give 95% certainty 20 times you're probably going to "discover" at least one falsehood. https://xkcd.com/882/ https://xkcd.com/882/
stygiansonic 7y ago
You seem to be under the impression that a study like this gives a hard "yes/no" answer as to whether some hypothesis is true. That is not the case, nor is it ever the case with most studies like these. Instead, you need to do some sort of statistical hypothesis test. As other comments have pointed out, once you start testing multiple hypothesis on the same dataset, you cannot apply the same significance threshold that you would if you had just begun with a single hypothesis before observing the data. Instead, you need to apply some sort of correction that takes into account the number of hypothesis being tested: https://en.wikipedia.org/wiki/Family-wise_error_rate#Controlling_procedures https://en.wikipedia.org/wiki/Family-wise_error_rate#Control...

rossdavidh 7y ago

Well, it would be p-hacking if you tried 1,032 different hypotheses until you got one that passed your threshold. There's quite a lot of scientific history (e.g. Kepler's discovery that planets went in elliptical orbits) that would have to be thrown out if you decided you could never use data for anything other than the original hypothesis. Kepler didn't even collect the data, much less collect it with the idea that the planetary orbits were elliptical. Having said that, the results smells (pun intended) bad, just because I cannot think of any plausible reason for a non-migrating animal to align with the magnetic field, when defecating or at any other time.

MrEldritch 7y ago
This is true, and a valid point. The way they phrased it does make me feel more than a little suspicious, nonetheless. (Besides, there's some other oddity there, like that apparently the alignment only matters when the magnetic field is calm)
- petschge 7y ago
  When the magnetic field is non-calm that is probably due to a space weather event that is geo-effective and inducing large currents in the ground. The local magnetic field an then be significantly distorted depending on local conductivity. So to me that is not an oddity.
- austinjp 7y ago
  You're right to still feel suspicious. Who's to say they didn't try 1,000 different post-hoc ideas? They declare only one, the may have been others. I'd be looking for preceding research and any published protocols, if I wasn't on mobile and didn't think it would be fruitless.
monocasa 7y ago
I mean, humans can detect magnetic fields as well. There's even languages without relative egocentric positions like left/right, only north south east west. Given all that, I don't think it's out there that dogs sense it and like being aligned when they're trying to poop. https://www.sciencemag.org/news/2019/03/humans-other-animals-may-sense-earth-s-magnetic-field https://www.sciencemag.org/news/2019/03/humans-other-animals... https://en.wikipedia.org/wiki/Guugu_Yimithirr_language https://en.wikipedia.org/wiki/Guugu_Yimithirr_language

Reason077 7y ago

> "The study was truly blind." I'd argue the study would only be truly blind if the dogs were blind. If blind dogs also oriented themselves north-south, then that would prove that they weren't using visual cues for alignment, such as the position of the sun.

philosopher1234 7y ago
You can feel the heat of the sun

marcosdumay 7y ago

P-hacking only apply to proofs. This study does two things, it falsifies a previous hypothesis fair and square, with no p-hacking, and it postulates another hypothesis, an activity where the concept of p-hacking does not even apply. The only wrong party here is the one that reported the study found something.