4 ms·

> It's not science until it passes peer review Peer reviewers don't generally reproduce the work in the publications they are asked to review, at least not du

> It's not science until it passes peer review Peer reviewers don't generally reproduce the work in the publications they are asked to review, at least not during the review process itself. Neither do journal editors. If you want a real vanity press, check vixra and its origin story. ArXiv does have moderators and endorsers in each area, although they are usually light-touch, weeding out literally unreadable submissions, ones that so obviously ignore formatting guidelines that it beggars belief they could comply with a typical journal's rules, and ones that are clearly submitted to the wrong area. If anything, I think sometimes the net is too fine. Will Kinney just this week had a version of <https://www.acsu.buffalo.edu/~whkinney/SpecialRelativityBootCamp/index.html https://www.acsu.buffalo.edu/~whkinney/SpecialRelativityBoot...> rejected by the arXiv, for example. The reasons for arXiv-rejection can be opaque. Having a stable document early (a pre-print) tends to widen the scrutiny of papers that may ultimately be published to a journal whose editor's expertise lies in a very different area from the submission; this can and does lead to author corrections being made before journal publication. There are plenty of peer-reviewed papers which are hot garbage that have found their way into prestigious high-impact journals like Nature and Science (see <https://retractionwatch.com/the-retraction-watch-leaderboard/top-10-most-highly-cited-retracted-papers/ https://retractionwatch.com/the-retraction-watch-leaderboard...> for examples, and note none of the top 10 are in areas covered by the arXiv). > being published by a famous professor from a prestigious university is also no guarantee Everyone in academia knows this, including the vast majority of famous professors from prestigious universities, because of online repositories like <https://retractionwatch.com/retractions-by-nobel-prize-winners/ https://retractionwatch.com/retractions-by-nobel-prize-winne...> (and much more sadly because of <https://en.wikipedia.org/wiki/Nobel_disease https://en.wikipedia.org/wiki/Nobel_disease>, lower-profile versions of which academic paper-writers -- and dissertation writers -- tend to encounter as they chase the history of the problem before them or read late citations to works they are relying upon). This particular part of your set of claims is is not a real problem in academia or with the arXiv in particular. Finally, what value does publication in a predatory journal bring? Do you believe that peer review and editing were actually even performed in the majority of MDPI's most predatory journals, for example? <https://www.predatoryjournals.org/news/list-of-all-mdpi-predatory-journals https://www.predatoryjournals.org/news/list-of-all-mdpi-pred...> Some papers published in some of their pay-for-publication open access journals don't even get submitted to the arXiv; one might hope this is out of embarrassment by the authors, although the (low) bar set by the arXiv itself is certainly a factor. The remedy for a reasonably argued but wrong academic paper isn't lack of publication, failed peer review, or editorial alteration, but rather reply papers. That's the academic dialogue.

rhelz 9d ago
Thanks for this reply, your points are well taken and it really got me to think about this stuff. I suppose you could view arxiv as being a way to let more people do a peer review on the papers, i.e. as one component of a more open and transparent peer review process. As such, it is certainly a welcome alternative to what the snootier journal reviews do, which is to put off reading your paper for a year, and then spend about 50 seconds looking for a reason to say "no". But I had a narrow point to make when I said that arxiv is a vanity press: just because something is on there doesn't guarantee that it is correct, or even useful. It is a vanity press in the sense that it will publish pretty much anything--modulo obscenity and porn laws, and apparently also some sanity checking by arxiv moderators--but so does a vanity press for print books. To your point of predatory journals, power-broker reviewers and such, yes, I have done my fair share of suffering from them. It slows down good research and passes through bad research. The system is badly in need of reform, and arxiv gives a much needed alternative. But really, the problem is (as one of my profs said) that science is totally run on volunteer, unpaid labor. Peer reviewers are very busy, get paid nothing, it is mostly a distraction. This leads them to look for other ways of being compensated--e.g. the can try to be a power broker, a feared authority, etc. The real solution would be to actually pay peer reviewers and fund them so they can actually reproduce the results. Here's how it could work: when a researcher submits a grant proposal, they budget for enough money to fund the research and enough money to peer review and reproduce it. If the research isn't good enough to warrant an attempt to validate it, then it isn't worth doing in the first place.
- raattgift 8d ago
  No paper is guaranteed to be completely accurate and correct at the time of publication; the remedy for honest errors is follow-on papers by the original authors or others in the field putting their names to formalized counter-arguments. This is how the academic dialogue works, and it dates back to early durable records in the ancient world -- about as far before Plato as Plato's Academy is before today's universities. Innovations like academic journals come from "merely" four centuries ago, and pre-publication evaluation by peers about three hundred years ago. Anonymized pre-publication review is more recent still, and publishing schedules (as well as the needs of the submitter to have the submitted work dealt with in good time) don't admit a dialogue over some subtle point a referee takes some issue with. Moreover, such a pre-publication dialogue is generally non-public, and often also filtered through an editor. It can get messy [1]. As I'm sure you've discovered in your own post-secondary education, academia is very conservative (in the sense of "retaining what mostly works, even in the face of mounting scaling problems", foremost among those being the ever-increasing volume of publishable works generated by researchers). Your own idea is quite conservative, and has been tried from time to time. Direct compensation by journals of referees for their time has been the subject of rigorous human experimentation: https://10.1097/CCM.0000000000006637 https://10.1097/CCM.0000000000006637 https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1 https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1 as recent examples, with mixed results. Direct compensation of referees by journals has a number of pitfalls ranging from the fiddly details of how to deal with income tax liability pay (and credits towards future publication fees, etc) incurs to whether it compromises the objectivity of pre-publication evaluation (which is frankly already not very good in highly specialized areas). Payment of reviewers by paper submitters is imho an even worse idea -- how do you preserve the anonymity of referees (especially with respect to the submitters)? Do submitters or reviewers have exposure to their (probably different) tax authorities for these transactions? (I can think of at least four national tax agencies that would view the received pay as taxable income, and also potentially subject to value-added tax on professional services. One already sees this problem with conference honoraria.) Another -- one that already exists in the real world -- might be for institutions and grant-funders to build in support for peer review of others' work by project members. To the extent that detailed pre-publication review benefits academia, and societies that fund academia, as a whole, maybe referees should be rewarded (or even obliged) to do paid reviewing duty. One might compare the requirement to do pro bono work imposed on many practising lawyers by their regulating authorities (especially interesting are jurisdictions in which a party losing a court contest to a party represented pro bono usually must pay into a fund that supports the overall scheme). Personally I think it would be more useful to everyone to have a solid reply paper published in due course than a blocked initial publication. My ideal is that referees aim to help submitters not accidentally professionally embarrass themselves. Then on your reproduction idée fixe, supporting the generation of papers which confirm reproduction is something that societies could certainly work on. Many incentives to write and submit papers calling out flaws in others' work already exist, and of course could be expanded. I don't think any of that should be done with masking of the participants. Additionally, reproduction of results is often less interesting than (sometimes much) later testing of published results in significantly different ways. (Indeed, foundational papers from more than ninety years ago are often re-proven by modern-day experiments, both as a side effect of (or opportunity arising from) pursuing the primary research goal, and as a way of testing e.g. new metrology technology, new computational tools, mathematical advances, and so on. Also, some of those foundational papers simply could not be tested with great accuracy with tools existing at the time of publication. Their publication, however, tended to drive the development of such tools, and in modern days such foundational papers are found to be good only within certain limits that could not have been explored closer to the time of publication.) - -- [1] https://theconversation.com/hate-the-peer-review-process-einstein-did-too-27405 https://theconversation.com/hate-the-peer-review-process-ein... (2014)

rhelz 9d ago

Thanks for this reply, your points are well taken and it really got me to think about this stuff. I suppose you could view arxiv as being a way to let more people do a peer review on the papers, i.e. as one component of a more open and transparent peer review process. As such, it is certainly a welcome alternative to what the snootier journal reviews do, which is to put off reading your paper for a year, and then spend about 50 seconds looking for a reason to say "no". But I had a narrow point to make when I said that arxiv is a vanity press: just because something is on there doesn't guarantee that it is correct, or even useful. It is a vanity press in the sense that it will publish pretty much anything--modulo obscenity and porn laws, and apparently also some sanity checking by arxiv moderators--but so does a vanity press for print books. To your point of predatory journals, power-broker reviewers and such, yes, I have done my fair share of suffering from them. It slows down good research and passes through bad research. The system is badly in need of reform, and arxiv gives a much needed alternative. But really, the problem is (as one of my profs said) that science is totally run on volunteer, unpaid labor. Peer reviewers are very busy, get paid nothing, it is mostly a distraction. This leads them to look for other ways of being compensated--e.g. the can try to be a power broker, a feared authority, etc. The real solution would be to actually pay peer reviewers and fund them so they can actually reproduce the results. Here's how it could work: when a researcher submits a grant proposal, they budget for enough money to fund the research and enough money to peer review and reproduce it. If the research isn't good enough to warrant an attempt to validate it, then it isn't worth doing in the first place.

raattgift 8d ago
No paper is guaranteed to be completely accurate and correct at the time of publication; the remedy for honest errors is follow-on papers by the original authors or others in the field putting their names to formalized counter-arguments. This is how the academic dialogue works, and it dates back to early durable records in the ancient world -- about as far before Plato as Plato's Academy is before today's universities. Innovations like academic journals come from "merely" four centuries ago, and pre-publication evaluation by peers about three hundred years ago. Anonymized pre-publication review is more recent still, and publishing schedules (as well as the needs of the submitter to have the submitted work dealt with in good time) don't admit a dialogue over some subtle point a referee takes some issue with. Moreover, such a pre-publication dialogue is generally non-public, and often also filtered through an editor. It can get messy [1]. As I'm sure you've discovered in your own post-secondary education, academia is very conservative (in the sense of "retaining what mostly works, even in the face of mounting scaling problems", foremost among those being the ever-increasing volume of publishable works generated by researchers). Your own idea is quite conservative, and has been tried from time to time. Direct compensation by journals of referees for their time has been the subject of rigorous human experimentation: https://10.1097/CCM.0000000000006637 https://10.1097/CCM.0000000000006637 https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1 https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1 as recent examples, with mixed results. Direct compensation of referees by journals has a number of pitfalls ranging from the fiddly details of how to deal with income tax liability pay (and credits towards future publication fees, etc) incurs to whether it compromises the objectivity of pre-publication evaluation (which is frankly already not very good in highly specialized areas). Payment of reviewers by paper submitters is imho an even worse idea -- how do you preserve the anonymity of referees (especially with respect to the submitters)? Do submitters or reviewers have exposure to their (probably different) tax authorities for these transactions? (I can think of at least four national tax agencies that would view the received pay as taxable income, and also potentially subject to value-added tax on professional services. One already sees this problem with conference honoraria.) Another -- one that already exists in the real world -- might be for institutions and grant-funders to build in support for peer review of others' work by project members. To the extent that detailed pre-publication review benefits academia, and societies that fund academia, as a whole, maybe referees should be rewarded (or even obliged) to do paid reviewing duty. One might compare the requirement to do pro bono work imposed on many practising lawyers by their regulating authorities (especially interesting are jurisdictions in which a party losing a court contest to a party represented pro bono usually must pay into a fund that supports the overall scheme). Personally I think it would be more useful to everyone to have a solid reply paper published in due course than a blocked initial publication. My ideal is that referees aim to help submitters not accidentally professionally embarrass themselves. Then on your reproduction idée fixe, supporting the generation of papers which confirm reproduction is something that societies could certainly work on. Many incentives to write and submit papers calling out flaws in others' work already exist, and of course could be expanded. I don't think any of that should be done with masking of the participants. Additionally, reproduction of results is often less interesting than (sometimes much) later testing of published results in significantly different ways. (Indeed, foundational papers from more than ninety years ago are often re-proven by modern-day experiments, both as a side effect of (or opportunity arising from) pursuing the primary research goal, and as a way of testing e.g. new metrology technology, new computational tools, mathematical advances, and so on. Also, some of those foundational papers simply could not be tested with great accuracy with tools existing at the time of publication. Their publication, however, tended to drive the development of such tools, and in modern days such foundational papers are found to be good only within certain limits that could not have been explored closer to the time of publication.) - -- [1] https://theconversation.com/hate-the-peer-review-process-einstein-did-too-27405 https://theconversation.com/hate-the-peer-review-process-ein... (2014)