6 ms·

Some (e.g. Marc Andreessen: https://richardhanania.substack.com/p/flying-x-wings-into-the-death-star https://richardhanania.substack.com/p/flying-x-wings-into-t

by oerpli 5y ago

Some (e.g. Marc Andreessen: https://richardhanania.substack.com/p/flying-x-wings-into-the-death-star https://richardhanania.substack.com/p/flying-x-wings-into-th... ) say, 90% are either wrong or useless. It's roughly what I would guess as well.

mirker 5y ago
Citations follow some power-law type of distribution, so the 10% (or whatever) of useful ones are basically the high impact papers, anyway. It would be more surprising if citations and usefulness were anti-correlated in some way. Back to the “gradient” analogy, in this case, papers have a feedback mechanism to suppress less useful papers. Science is not a random flurry of results mashed together.
- AussieWog93 5y ago
  >It would be more surprising if citations and usefulness were anti-correlated in some way. I have definitely seen some highly cited and influential papers overstay their welcome and hold the field back. That's not to say they didn't propel the field forward when they were initially published, just that their overbearing influence causes us to stick to a paradigm that is no longer producing results. "Science progresses one funeral at a time" and all that.
AussieWog93 5y ago
Based on my experience as a PhD student (Machine Learning Algorithms for Brain-Computer Interfaces), I would say that useless papers are far more of a problem than wrong ones. I didn't see a lot of evidence of outright fraud, but a lot of mundane and irrelevant research ("This algorithm happens to work well on this one particular test dataset well") presented as if it were a breakthrough that was important to the field. "Publish or perish" is a helluva drug.
- api 5y ago
  You get what you incentivize of course. The question is: what else could we incentivize that would be better and would still allow us to scale science? One of the hardest things about scaling human activity is figuring out what your goal functions should be. I don’t think raw publication count is a great metric. The one I personally always used was engineering usefulness, but that only works for those areas that are close to application. It’s an increasingly useless metric as you get far from application, but the stuff that is far from application is essential to our large scale understanding and to future applicable research.
  - AussieWog93 5y ago
    >One of the hardest things about scaling human activity is figuring out what your goal functions should be. Marvin Minsky suggested a fairly reasonable solution to this problem based on his experience in the DARPA program during the 60s and 70s: give smart/accomplished people funding and then stand back and let them work out the rest. A decent amount of the funding will be wasted, but it's still better than the current system of jumping through hoops and appealing to the biases of the old guard in order to get grants.

by oerpli 5y ago

mirker 5y ago

Citations follow some power-law type of distribution, so the 10% (or whatever) of useful ones are basically the high impact papers, anyway. It would be more surprising if citations and usefulness were anti-correlated in some way. Back to the “gradient” analogy, in this case, papers have a feedback mechanism to suppress less useful papers. Science is not a random flurry of results mashed together.

AussieWog93 5y ago
>It would be more surprising if citations and usefulness were anti-correlated in some way. I have definitely seen some highly cited and influential papers overstay their welcome and hold the field back. That's not to say they didn't propel the field forward when they were initially published, just that their overbearing influence causes us to stick to a paradigm that is no longer producing results. "Science progresses one funeral at a time" and all that.

AussieWog93 5y ago

Based on my experience as a PhD student (Machine Learning Algorithms for Brain-Computer Interfaces), I would say that useless papers are far more of a problem than wrong ones. I didn't see a lot of evidence of outright fraud, but a lot of mundane and irrelevant research ("This algorithm happens to work well on this one particular test dataset well") presented as if it were a breakthrough that was important to the field. "Publish or perish" is a helluva drug.

api 5y ago
You get what you incentivize of course. The question is: what else could we incentivize that would be better and would still allow us to scale science? One of the hardest things about scaling human activity is figuring out what your goal functions should be. I don’t think raw publication count is a great metric. The one I personally always used was engineering usefulness, but that only works for those areas that are close to application. It’s an increasingly useless metric as you get far from application, but the stuff that is far from application is essential to our large scale understanding and to future applicable research.
- AussieWog93 5y ago
  >One of the hardest things about scaling human activity is figuring out what your goal functions should be. Marvin Minsky suggested a fairly reasonable solution to this problem based on his experience in the DARPA program during the 60s and 70s: give smart/accomplished people funding and then stand back and let them work out the rest. A decent amount of the funding will be wasted, but it's still better than the current system of jumping through hoops and appealing to the biases of the old guard in order to get grants.