5 ms·

What percentage of papers even reproduce these days? Is it more than 50%?

by sickofparadox 1y ago

ktallett 1y ago
The bigger issue is what percentage of papers contain enough details to even attempt to reproduce.
- fuzzfactor 1y ago
  In over 80 volumes of ASTM publications, I would estimate they may amount to more kilos than that. Almost all of the actual lab work requires statistical determination of repeatability & reproducibility to be calculated between different labs, and the summary is included with each document. I would say there is way less than a kilo without this. And the amount of supporting raw data on file amounts to kilos which dwarf the pages published. Formally accessible so everything can be thoroughly reviewed at any time in the future, allowing for complete reconsideration if called for. Scietific instrumentation doesn't stand still. So it definitely can be done. Even if it's to the extreme not suitable for everybody else. The less-reproducible documents are there, they did the best they could, but have a smell not shared by the good stuff. You know "exactly" how good or bad the underlying science turned out to be in the real world. Paradoxically, or intuitively, as the case may be, if you're going to utilize the less-reliable techniques (most likely because they're the best there is), you may need to know how bad they are most of all. Maybe other publications should raise the bar on statistics as appropriate, I figure zero statistics is about as far as you can get from ASTM "standards". Some places probably have a lot further to go than others and it would be nice to have a whole lot sweeter smell all around.
  - ta20240528 1y ago
    Or you could, you know, email the authors for the original data, etc. when planning your reproduction study.
    - fuzzfactor 1y ago
      I thought it was understood that's what they do these days. People can probably appreciate that lot of the kilos are more figurative than ever, but there's still enough hard copies made to fill big trucks though. I'm not in academics professionally myself, how universal would you estimate it is among their journals these days to require a statistical study between an adequate number of different labs before final publication?
gus_massa 1y ago
It depends a lot on the area. I'd not be so pessimistic. The problem is how many of the papers that reach newspapers are reproducible? I guess less than the average. And also strange results that are misinterpreted to get a amazing but wrong layman explanation.
Fomite 1y ago
Everyone wants reproduction, nobody wants to fund reproduction studies.
- SJC_Hacker 1y ago
  Depending in the field and impact, reproduction becomes “free” because other scientists try to build on the results. Science is often about chasing the latest, hottest thing If they can’t reproduce the original, they should get called out eventually
  - Fomite 1y ago
    It often is, but that's much less formal and field dependent than a lot of people want.
    - SJC_Hacker 1y ago
      Yes, large clinical studies rarely get reproduced
nothercastle 1y ago
You would be lucky to get 1/10

What percentage of papers even reproduce these days? Is it more than 50%?

ktallett 1y ago

The bigger issue is what percentage of papers contain enough details to even attempt to reproduce.

fuzzfactor 1y ago
In over 80 volumes of ASTM publications, I would estimate they may amount to more kilos than that. Almost all of the actual lab work requires statistical determination of repeatability & reproducibility to be calculated between different labs, and the summary is included with each document. I would say there is way less than a kilo without this. And the amount of supporting raw data on file amounts to kilos which dwarf the pages published. Formally accessible so everything can be thoroughly reviewed at any time in the future, allowing for complete reconsideration if called for. Scietific instrumentation doesn't stand still. So it definitely can be done. Even if it's to the extreme not suitable for everybody else. The less-reproducible documents are there, they did the best they could, but have a smell not shared by the good stuff. You know "exactly" how good or bad the underlying science turned out to be in the real world. Paradoxically, or intuitively, as the case may be, if you're going to utilize the less-reliable techniques (most likely because they're the best there is), you may need to know how bad they are most of all. Maybe other publications should raise the bar on statistics as appropriate, I figure zero statistics is about as far as you can get from ASTM "standards". Some places probably have a lot further to go than others and it would be nice to have a whole lot sweeter smell all around.
- ta20240528 1y ago
  Or you could, you know, email the authors for the original data, etc. when planning your reproduction study.
  - fuzzfactor 1y ago
    I thought it was understood that's what they do these days. People can probably appreciate that lot of the kilos are more figurative than ever, but there's still enough hard copies made to fill big trucks though. I'm not in academics professionally myself, how universal would you estimate it is among their journals these days to require a statistical study between an adequate number of different labs before final publication?

gus_massa 1y ago

It depends a lot on the area. I'd not be so pessimistic. The problem is how many of the papers that reach newspapers are reproducible? I guess less than the average. And also strange results that are misinterpreted to get a amazing but wrong layman explanation.

Fomite 1y ago

Everyone wants reproduction, nobody wants to fund reproduction studies.

SJC_Hacker 1y ago
Depending in the field and impact, reproduction becomes “free” because other scientists try to build on the results. Science is often about chasing the latest, hottest thing If they can’t reproduce the original, they should get called out eventually
- Fomite 1y ago
  It often is, but that's much less formal and field dependent than a lot of people want.
  - SJC_Hacker 1y ago
    Yes, large clinical studies rarely get reproduced

nothercastle 1y ago

You would be lucky to get 1/10