6 ms·

> There was a great article published in PLOS several years ago (ATM I don't have the link) showing mathematically that the odds were about 1 in a million that

by nmjohn 12y ago

> There was a great article published in PLOS several years ago (ATM I don't have the link) showing mathematically that the odds were about 1 in a million that an observational study like the above would turn out to be a "true" causal relationship, and the author concluded most published studies were junk. I very much doubt the second part of that, that most published studies are junk. The idea that not showing a causal link and only a correlational one makes a study junk is not held by anyone in the field who garners a whole lot of respect or notoriety. I'm not saying the quality is equal whatsoever, simply that correlation studies are not inherently junk - some are fantastic and some are quite the opposite. There is no question that journals in general are pumping out a significant amount of junk (in studies of all types), but I speculate the root cause of that has more to do with the rise of "publish or perish" and significant increases in grad school enrollment. And even worse the notion that for a grad student, the number of publications they have their name attributed to is more important than the content they publish in terms of employment after they graduate. So there is a situation where people have more pressure to publish than ever before [0], there is more competition for scarce funding so studies are vastly underfunded and studies are rushed so another can begin to add to the resume. That doesn't mean there is less good science being done either! There probably is more good science being published now than ever before, the problem is the signal:noise ratio has gone down making it harder for good studies to get media attention, and easier for the media to latch on to whatever story they think will get viewers. [0]: Sidenote, this also makes it increasingly challenging not only to have high quality research, but research that only is making a correlation as opposed to going through and providing evidence to claim you might have a causal link.

a_bonobo 12y ago
This, I think, is the study the first poster meant, Ioannidis 2005, 'Why Most Published Research Findings Are False': http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj... It's quite a famous paper. Many rebuttals, comments, blog-posts etc, have been published, have a look at the comments on the PLOS site, I especially like this blog post summarizing more research: http://simplystatistics.org/2013/09/25/is-most-science-false-the-titans-weigh-in/ http://simplystatistics.org/2013/09/25/is-most-science-false... From the same author, here's a small overview of the discussion at the end of 2013: http://simplystatistics.org/2013/12/16/a-summary-of-the-evidence-that-most-published-research-is-false/ http://simplystatistics.org/2013/12/16/a-summary-of-the-evid... Quote on the Ioannidis paper: >Under assumptions about the way people perform these tests and report them it is possible to construct a universe where most published findings are false positive results. Important drawback: The paper contains no real data, it is purely based on conjecture and simulation.

by nmjohn 12y ago

a_bonobo 12y ago

This, I think, is the study the first poster meant, Ioannidis 2005, 'Why Most Published Research Findings Are False': http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj... It's quite a famous paper. Many rebuttals, comments, blog-posts etc, have been published, have a look at the comments on the PLOS site, I especially like this blog post summarizing more research: http://simplystatistics.org/2013/09/25/is-most-science-false-the-titans-weigh-in/ http://simplystatistics.org/2013/09/25/is-most-science-false... From the same author, here's a small overview of the discussion at the end of 2013: http://simplystatistics.org/2013/12/16/a-summary-of-the-evidence-that-most-published-research-is-false/ http://simplystatistics.org/2013/12/16/a-summary-of-the-evid... Quote on the Ioannidis paper: >Under assumptions about the way people perform these tests and report them it is possible to construct a universe where most published findings are false positive results. Important drawback: The paper contains no real data, it is purely based on conjecture and simulation.