7 ms·

I happen to be an (associate) editor of an academic journal. What the people who critique the publication process are missing: 90% of submissions are crap - un

by jpfr 2y ago

I happen to be an (associate) editor of an academic journal. What the people who critique the publication process are missing: 90% of submissions are crap - unfit for publication. We need some process to gate-keep. a) The venue of publication is a good signal, whether the time to read a paper is well-spent. b) The PhD students learning the craft need objective feedback. The supervising professior/university often has the incentive to "just submit" -- even if they know that a publication does not meet the quality standard. Before peer-review, somebody also needed to make a decision on what to publish. This typically fell to a single individual. The editor or some well-known member of the community who could recommend a paper for publication. On old journal issues they even mention the "recommender". So the question is not whether peer-review is bad, the question is which alternative gate-keeping process would be better. Otherwise we will drown in crap publications (even more) and the PhD students don't get a honest feedback signal upon which they can improve their craft.

Spacecosmonaut 2y ago
We are on the cusp of being able to plow through vast quantities of literature and data in an instant using multimodal machine learning models. Journal articles are written for people. The landscape is changing. We are headed toward a future where scientist upload data and thoughts en masse into the cloud to be consumed by interpreter models that in turn feed back into the scientific machine. Data quality and attribution (scientist performance rating) is automatically allocated by models.
selimthegrim 2y ago
PNAS still mentions the recommender.
cherryteastain 2y ago
As a (former) reviewer at 5 journals, I disagree first and foremost with the notion that > We need some process to gate-keep. Journals, when print was the medium through which academic research was disseminated, had to gatekeep because there were practical considerations regarding how many articles they could put in each issue. With online repositories like arxiv, this is hardly a concern anymore. Someone putting a crap article on arxiv does not hurt anyone else, and I'm saying this as a person who recommended tons of articles to be rejected because they had atrocious grammar/spelling issues. Worst case, it gets 0 attention and is ignored by the research community. Something not being published in a journal/conference proceedings clearly does not prevent it from drawing tons of research attention, as we saw in numerous cases like the Adam optimizer [1]. Which brings us to the second point: what even is the purpose of a journal now? The answer is that the sole function of a journal now is gatekeeping, with the presupposition that, as you observed > The venue of publication is a good signal, whether the time to read a paper is well-spent Except, well, top journals have tons of articles that get 0 citations too. Clearly the filter fails at this purpose as well. So, why gatekeep at all then? Because if we did not have some exclusive prestigious journal, the plebs would not be separated from the esteemed titans of academia with the biggest grants, most prestigious scholarships and diplomas from the most famous universities. The only reason we need to gatekeep today is to feed the academic prestige and politics machine. If you care about the science, upload the goddamn PDF to arxiv , tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting. [1] https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980
- Al-Khwarizmi 2y ago
  The Adam optimizer was published at ICLR, a top conference in machine learning. So, basically, analogous to a peer-reviewed journal for the purposes of this discussion. ML (and some other subfields of computer science) have the particularity that the really comptetitive gatekeeping happens even more in conferences than in journals. That said, there definitely are very relevant papers that are not published in any peer-reviewed venue. A good example is "Language Models are Unsupervised Multitask Learners" (the GPT-2 paper, which I would argue started the whole generative LLM revolution). But I think if you look for this kind of papers, you will find something in common to all of them: they are by very well known researchers, elite institutions or influential companies. That's why people went out of their way to read them even if they were posted somewhere without peer review. If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact. It's peer review that allows them to be able to submit to a top journal, where the editor and reviewers will read their paper, and they can get a somewhat fair chance. PS: I don't really like the peer review system that much either. It's just that every alternative that I have seen proposed so far is worse.
  - cherryteastain 2y ago
    > The Adam optimizer was published at ICLR, a top conference in machine learning. Fair, must have misremembered that one. > If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact. I disagree on this one. I did my PhD at an institution that ranks in the top 10 in the most well known university rankings, and I distinctly remember that one time when I was submitting a manuscript to a prominent journal in my field, got some reviews back which weren't positive yet were quite valid criticisms, and my professor told me not to worry because the editor is his buddy and my manuscript will get published for sure. When that sort of "scratch my back and I'll scratch yours" culture exists in journals I don't see how peer review can be an equalizer. It just means everyone who publishes at a journal, including the less well esteemed ones, can claim they went through the rigor of peer review. Of course, we all know peer review is just a vibe check and is actually not that rigorous at all, and besides no one cares unless you published in a prestigious journal anyway. The less revered journals exist to collect $5k in open access fees for the publisher in return for hosting a pdf at the marginal cost of maybe a cent a year.
naasking 2y ago
> What the people who critique the publication process are missing: 90% of submissions are crap - unfit for publication. Publication is antiquated. HN doesn't need reviewers to boost the best content or to provide commentary on how to improve a paper or fact check its contents. Join the 21st century.

by jpfr 2y ago

Spacecosmonaut 2y ago

We are on the cusp of being able to plow through vast quantities of literature and data in an instant using multimodal machine learning models. Journal articles are written for people. The landscape is changing. We are headed toward a future where scientist upload data and thoughts en masse into the cloud to be consumed by interpreter models that in turn feed back into the scientific machine. Data quality and attribution (scientist performance rating) is automatically allocated by models.

selimthegrim 2y ago

PNAS still mentions the recommender.

cherryteastain 2y ago

As a (former) reviewer at 5 journals, I disagree first and foremost with the notion that > We need some process to gate-keep. Journals, when print was the medium through which academic research was disseminated, had to gatekeep because there were practical considerations regarding how many articles they could put in each issue. With online repositories like arxiv, this is hardly a concern anymore. Someone putting a crap article on arxiv does not hurt anyone else, and I'm saying this as a person who recommended tons of articles to be rejected because they had atrocious grammar/spelling issues. Worst case, it gets 0 attention and is ignored by the research community. Something not being published in a journal/conference proceedings clearly does not prevent it from drawing tons of research attention, as we saw in numerous cases like the Adam optimizer [1]. Which brings us to the second point: what even is the purpose of a journal now? The answer is that the sole function of a journal now is gatekeeping, with the presupposition that, as you observed > The venue of publication is a good signal, whether the time to read a paper is well-spent Except, well, top journals have tons of articles that get 0 citations too. Clearly the filter fails at this purpose as well. So, why gatekeep at all then? Because if we did not have some exclusive prestigious journal, the plebs would not be separated from the esteemed titans of academia with the biggest grants, most prestigious scholarships and diplomas from the most famous universities. The only reason we need to gatekeep today is to feed the academic prestige and politics machine. If you care about the science, upload the goddamn PDF to arxiv , tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting. [1] https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980

Al-Khwarizmi 2y ago
The Adam optimizer was published at ICLR, a top conference in machine learning. So, basically, analogous to a peer-reviewed journal for the purposes of this discussion. ML (and some other subfields of computer science) have the particularity that the really comptetitive gatekeeping happens even more in conferences than in journals. That said, there definitely are very relevant papers that are not published in any peer-reviewed venue. A good example is "Language Models are Unsupervised Multitask Learners" (the GPT-2 paper, which I would argue started the whole generative LLM revolution). But I think if you look for this kind of papers, you will find something in common to all of them: they are by very well known researchers, elite institutions or influential companies. That's why people went out of their way to read them even if they were posted somewhere without peer review. If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact. It's peer review that allows them to be able to submit to a top journal, where the editor and reviewers will read their paper, and they can get a somewhat fair chance. PS: I don't really like the peer review system that much either. It's just that every alternative that I have seen proposed so far is worse.
- cherryteastain 2y ago
  > The Adam optimizer was published at ICLR, a top conference in machine learning. Fair, must have misremembered that one. > If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact. I disagree on this one. I did my PhD at an institution that ranks in the top 10 in the most well known university rankings, and I distinctly remember that one time when I was submitting a manuscript to a prominent journal in my field, got some reviews back which weren't positive yet were quite valid criticisms, and my professor told me not to worry because the editor is his buddy and my manuscript will get published for sure. When that sort of "scratch my back and I'll scratch yours" culture exists in journals I don't see how peer review can be an equalizer. It just means everyone who publishes at a journal, including the less well esteemed ones, can claim they went through the rigor of peer review. Of course, we all know peer review is just a vibe check and is actually not that rigorous at all, and besides no one cares unless you published in a prestigious journal anyway. The less revered journals exist to collect $5k in open access fees for the publisher in return for hosting a pdf at the marginal cost of maybe a cent a year.

naasking 2y ago

> What the people who critique the publication process are missing: 90% of submissions are crap - unfit for publication. Publication is antiquated. HN doesn't need reviewers to boost the best content or to provide commentary on how to improve a paper or fact check its contents. Join the 21st century.