6 ms·

The recent announcement to reject review articles and position papers already smelled like a shift towards a more "opinionated" stance, and this move smells wor

The recent announcement to reject review articles and position papers already smelled like a shift towards a more "opinionated" stance, and this move smells worse. The vacuum that arXiv originally filled was one of a glorified PDF hosting service with just enough of a reputation to allow some preprints to be cited in a formally published paper, and with just enough moderation to not devolve into spam and chaos. It has also been instrumental in pushing publishers towards open access (i.e., to finally give up). Unfortunately, over the years, arXiv has become something like a "venue" in its own right, particularly in ML, with some decently cited papers never formally published and "preprints" being cited left and right. Consider the impression you get when seeing a reference to an arXiv preprint vs. a link to an author's institutional website. In my view, arXiv fulfills its function better the less power it has as an institution, and I thus have exactly zero trust that the split from Cornell is driven by that function. We've seen the kind of appeasement prose from their statement and FAQ [1] countless times before, and it's now time for the usual routine of snapshotting the site to watch the inevitable amendments to the mission statement. "What positive changes should users expect to see?" - I guess the negative ones we'll have to see for ourselves. [1] https://tech.cornell.edu/arxiv/ https://tech.cornell.edu/arxiv/

hijodelsol 6mo ago
I came here to say something similar. As someone who works in a field that applies machine learning but is not purely focused on it, I interact with people who think that arXiv is the only relevant platform and that they don't need to submit their work to any journal, as well as people who still think that preprints don't count at all and that data isn't published until it's printed in an academic journal. It can feel like a clash of worlds. I think both sides could learn from the other. In the case of ML, I understand the desire to move fast and that average time to publication of 250-300 days in some of the top-tier journals can feel like an unnecessary burden. But having been on both sides of peer review, there is value to the system and it has made for better work. Not doing any of it follows the same spirit as not benchmarking your approach against more than maybe one alternative and that already as an after-thought. Or benchmaxxing but not exploring the actual real-world consequences, time and cost trade offs, etc. Now, is academic publishing perfect? Of course not, very very far from it. It desperately needs to be reformed to keep it economically accessible, time efficient for both authors, editors and peer reviewers and to prevent the "hot topic of the day" from dominating journals and making sure that peer review aligns with the needs of the community and actually improves the quality of the work, rather than having "malicious peer review" to get some citations or pet peeves in. Given the power that the ML field holds and the interesting experiments with open review, I would wish for the field to engage more with the scientific system at large and perhaps try to drive reforms and improve it, rather than completely abandoning it and treating a PDF hosting service as a journal (ofc, preprints would still be desirable and are important, but they can not carry the entire field alone).
- bonoboTP 6mo ago
  Simply anticipating basic push backs from reviewers makes sure that you do a somewhat thorough job. Not 100% thorough and the reviews are sometimes frivolous and lazy and stupid. But just knowing that what you put out there has to pass the admittedly noisily gatekept gate of peer review overall improves papers in my estimation. There is also a negative side because people try to hide limitations and honest assessments and cherry pick and curate their tables more in anticipation of knee jerk reviewers but overall I think without any peer review, author culture would become much more lax and bombastic and generally trend toward engagement bait and social media attention optimized stuff. The current balance where people wrote a paper with reviers in mind, upload it to Arxiv before the review concludes and keep it on Arxiv even if rejected is a nice balance. People get to form their own opinion on it but there is also enough self-imposed quality control on it just due to wanting it to pass peer review, that even if it doesn't pass peer review, it is still better than if people write it in a way that doesn't care or anticipate peer review. And this works because people are somewhat incentivized to get peer reviewed official publications too. But being rejected is not the end of the world either because people can already read it and build on it based on Arxiv.
  - bjourne 6mo ago
    I really am not sure about that: https://biologue.plos.org/wp-content/uploads/sites/7/2020/05/Peer-Review-Nick-Kim-cartoon3-resize.jpg https://biologue.plos.org/wp-content/uploads/sites/7/2020/05... The problem is that "optimizing for peer-review" is not the same thing as optimizing for quality. E.g., I like to add a few tongue-in-cheeks to entertain the reader. But then I have to worry endlessly about anal-retentive reviewers who refuse to see the big picture.
    - bonoboTP 6mo ago
      Currently a kind of rule of thumb is that a PhD student can graduate after approximately 3 papers published in a good peer reviewed venue. If peer review were to go away, this whole academic system would get into a crisis. It's dysfunctional and has many problems but it's kinda load bearing for the system to chug along.
ph4rsikal 6mo ago
My observation is that research, especially in AI has left universities, which are now focusing their research to a lesser degree on STEM. It appears research is now done by companies like Meta, OpenAI, Anthropic, Tencent, Alibaba, among many others.
- bonoboTP 6mo ago
  Universities (outside a few) just have much weaker PR machines so you never hear what they do. Also their work is not user facing products so regular people, even tech power users won't see them.
  - 0x3f 6mo ago
    Not sure about that. How would a university test scaling hypotheses in AI, for example? The level of funding required is just not there, as far as I know.
    - rsfern 6mo ago
      This issue of accessibility is widely acknowledged in the academic literature, but it doesn’t mean that only large companies are doing good research. Personally I think this resource mismatch can help drive creative choice of research problems that don’t require massive resources. To misquote Feynman, there’s plenty of room at the bottom
    - oscaracso 6mo ago
      Universities are also not suited to test which race car is the fastest, but that does not obviate the need for academic research in mechanical engineering.
      - 0x3f 6mo ago
        Perhaps but the fastest race car is not possibly marshalling in the end of human involvement in science, so you might consider these of considerably different levels of meriting the funding.
stared 6mo ago
> arXiv fulfills its function better the less power it has as an institution It is an interesting instance of the rule of least power, https://en.wikipedia.org/wiki/Rule_of_least_power https://en.wikipedia.org/wiki/Rule_of_least_power.
- fidotron 6mo ago
  The irony of the TBL quotes there being the entire problem with the semantic web is the ontological tarpit that results due to the excessive expressive power of a general triple store.
  - PaulHoule 6mo ago
    Well, I’d argue that many things in the semweb are not expressive enough and lead to the misunderstandings we have. People think, for instance, that RDFS and OWL are meant to SHACL people into bad an over engineered ontologies. The problem is these standards add facts and don’t subtract facts. At risk of sounding like ChatGPT: it’s a data transformation system not a validation system. That is, you’re supposed to use RDFS to say something like ?s :myTermForLength ?o -> ?s :yourTermForLength ?o . The point of the namespace system is not to harass you, it is to be able to suck in data from unlimited sources and transform it. Trouble is it can’t do the simple math required to do that for real, like ?s :lengthInFeet ?o -> ?s :lengthInInches 12*?o . Because if you were trying OWL-style reasoning over arithmetic you would run into Kurt Gödel kinds of problems. Meanwhile you can’t subtract facts that fail validation, you can’t subtract facts that you just don’t need in the next round of processing. It would have made sense to promote SHACL first instead of OWL because garbage-in-garbage out, you are not going to reason successfully unless you have clean data… but what the hell do I know, I’m just an applications programmer who models business processes enough to automate them. Similarly the problem of ordered collections has never been dealt with properly in that world. PostgreSQL, N1QL and other post-relational and document DB languages can write queries involving ordered collections easily. I can write rather unobvious queries by hand to handle a lot of cases (wrote a paper about it) but I can’t cover all the cases and I know back in the day I could write SPAQL queries much better than the average RDF postdoc or professor. As for underengineering, Dublin Core came out when I worked at a research library and it just doesn’t come close in capability to MARC from 1970. Larry Masinter over at Adobe had to hack the standard to handle ordered collections because… the authors of a paper sure as hell care what order you write their names in. And it is all like that: RDF standards neglect basic requirements that they need to be useful and then all the complex/complicated stuff really stands out. If you could get the basics done maybe people would use them but they don’t.
light_hue_1 6mo ago
> Unfortunately, over the years, arXiv has become something like a "venue" in its own right, particularly in ML, with some decently cited papers never formally published and "preprints" being cited left and right. Consider the impression you get when seeing a reference to an arXiv preprint vs. a link to an author's institutional website. This just isn't true. arXiv is not a venue. There's no place that gives you credit for arXiv papers. No one cares if you cite an arXiv paper or some random website. The vast vast majority of papers that have any kind of attention or citations are published in another venue.
- contubernio 6mo ago
  A Fields medal was awarded based mainly on this paper never published elsewhere: https://arxiv.org/abs/math/0211159 https://arxiv.org/abs/math/0211159
  - auggierose 6mo ago
    I think there is a misunderstanding here. Does arXiv count as a publication? Yes, pretty much anything that gives you a DOI does, for example Zenodo. Does it function as a reputable anything? No. The paper you link to counts as a publication, but its reputation stands on its own, it has nothing to do with arXiv as a venue. Ideally, that's how it is for all papers, but it isn't, just by publishing in certain venues your paper automatically gets a certain amount of reputation depending on the venue.
    - fc417fc802 6mo ago
      > Ideally, that's how it is for all papers, but it isn't We require a method of filtering such that a given researcher doesn't have to personally vet in excruciating detail every paper he comes across because there simply isn't enough time in the day for that. Ideally such a system would individually for each paper provide a multi-dimensional score that was reputable. How can those be calculated in a manner such that they're reputable? Who knows; that exercise is left for the reader. In practice "well it got published in Nature" makes for a pretty decent spam filter followed by metrics such as how many times it's been cited since publication, checking that the people citing it are independent authors who actually built directly on top of the work, and checking how many of such citing authors are from a different field.
queuebert 6mo ago
> Unfortunately, over the years, arXiv has become something like a "venue" in its own right, ... In my experience as a publishing scientist, this is partly because publishing with "reputable" journals is an increasingly onerous process, with exorbitant fees, enshittified UIs, and useless reviews. The alternative is to upload to arXiv and move on with your life.
- groundzeros2015 6mo ago
  That’s true. But that’s separate than the use in ML in Blockchain circles as a form of a marketing - using academic appearances.
  - jjk166 6mo ago
    That sounds more like an issue of certain fields having crappy standards because the people in those fields benefit from crappy standards than an issue with the site they happen to host papers on.
    - deleted 6mo ago
      [deleted]
    - groundzeros2015 6mo ago
      I don’t buy “some fields are just more honorable”. Everyone uses publishing for personal gain. But yes it’s a people problem, not an arxiv problem.
  - StableAlkyne 6mo ago
    Every field and every publisher has this issue though. I've read papers in the chemical literature that were clearly thinly veiled case studies for whatever instrument or software the authors were selling. Hell, I've read papers that had interesting results, only to dig into the math and find something fundamentally wrong. The worst was an incorrect CFD equation that I traced through a telephone game of 4 papers only to find something to the effect of "We speculate adding $term may improve accuracy, but we have not extensively tested this" Just because something passed peer review does not make it a good paper. It just means somebody* looked at it and didn't find any obvious problems. If you are engaged in research, or in a position where you're using the scientific literature, it is vital that you read every paper with a critical lens. Contrary to popular belief, the literature isn't a stone tablet sent from God. It's messy and filled with contradictory ideas. *Usually it's actually one of their grad students
Aurornis 6mo ago
> and with just enough moderation to not devolve into spam and chaos arXiv has become a target for grifters in other domains like health and supplements. I’ve seen several small scale health influencers who ChatGPT some “papers” and then upload them to arXiv, then cite arXiv as proof of their “published research”. It’s not fooling anyone who knows how research work but it’s very convincing to an average person who thinks that that they’re doing the right thing when they follow sources that have done academic research. I’ve been surprised as how bad and obviously grifty some of the documents I’ve seen on arXiv have become lately. Is there any moderation, or is it a free for all as long as you can get an invite?
- deleted 6mo ago
  [deleted]
aimarketintel 6mo ago
[flagged]
PaulHoule 6mo ago
Review papers are interesting. Bibliometrics reveal that they are highly cited. Internal data we had at arXiv 20 years ago show they are highly read. Reading review papers is a big part of the way you go from a civilian to an expert with a PhD. On the other hand, they fall through the cracks of the normal methods of academic evaluation. They create a lot of value for people but they are not likely to advance your career that much as an academic, certainly not in proportion to the value they create, or at least the value they used to create. One of the most fun things I did on the way to a PhD was writing a literature review on giant magnetoresistance for the experimentalist on my thesis committee. I went from knowing hardly anything about the topic to writing a summary that taught him a lot he didn't know. Given any random topic in any field you could task me with writing a review paper and I could go out and do a literature search and write up a summary. An expert would probably get some details right that I'd get wrong, might have some insights I'd miss, but it's actually a great job for a beginner, it will teach you the field much more effectively than reading a review paper! How you regulate review papers is pretty tricky. If it is original research the criterion of "is it original research" is an important limit. There might already be 25 review papers on a topic, but maybe I think they all suck (they might) and I can write the 26th and explain it to people the way I wish it was explained to me. Now you might say in the arXiv age there was not a limit on pages, but LLMs really do problematize things because they are pretty good at summarization. Send one off on the mission to write a review paper and in some ways they will do better than I do, in other ways will do worse. Plenty of people have no taste or sense of quality and they are going to miss the latter -- hypothetically people could do better as a centaur but I think usually they don't because of that. One could make the case that LLMs make review papers obsolete since you can always ask one to write a review for you or just have conversations about the literature with them. I know I could have spend a very long time studying the literature on Heart Rate Variability and eventually made up my mind about which of the 20 or so metrics I want to build into my application and I did look at some review papers and can highlight sentences that support my decisions but I made those decisions based on a few weekends of experiments and talking to LLMs. The funny thing is that if you went to a conference and met the guy who wrote the review paper and gave them the hard question of "I can only display one on my consumer-facing HRV app, which one do I show?" they would give you that clear answer that isn't in the review paper and maybe the odds are 70-80% that it will be my answer.
abdullahkhalids 6mo ago
> Unfortunately, over the years, arXiv has become something like a "venue" in its own right, particularly in ML, with some decently cited papers never formally published and "preprints" being cited left and right. This has been a common practice in physics, especially the more theoretical branches, since the inception of arXiv. Senior researchers write a paper draft, and then send copies to some of their peers, get and incorporate feedback, and just submit to arxiv.
- godelski 6mo ago
  And this is really how it should be. Honestly the only thing I want arxiv to do is become more like open review. Allow comments by peers and some better linking to data and project pages. It works for physics because physicists are very rigorous. So papers don't change very much. It also works for ML because everyone is moving very fast that it's closer to doing open research. Sloppier, but as long as the readers are other experts then it's generally fine. I think research should really just be open. It helps everyone. The AI slop and mass publishing is exploiting our laziness; evaluating people on quantity rather than quality. I'm not sure why people are so resistant to making this change. Yes, it's harder, but it has a lot of benefits. And at the end of the day it doesn't matter if a paper is generated if it's actually a quality paper (not in just how it reads, but the actual research). Slop is slop and we shouldn't want slop regardless. But if we evaluate on quality and everything is open it becomes much easier to figure out who is producing slop, collision rings, plagiarist rings, and all that. A little extra work for a lot of benefits. But we seem to be willing to put in a lot of work to avoid doing more work
  - abdullahkhalids 6mo ago
    I don't agree actually that is how it should or can work for everyone. Senior researchers produce good quality research, and they have a network of high quality peers built over decades. Both those are necessary for them to reach out and ask for feedback, and get genuine and high quality feedback. Junior researchers don't have these typically. They also benefit more from anonymous feedback, which enables the reviewers to bluntly identify wrong or close to wrong results. So I think open journals should continue to exist. They fill an essential role in the scientific ecosystem.
fsckboy 6mo ago
>We've seen the kind of appeasement prose from their statement and FAQ [1] countless times before what are you referring to, who is being appeased who shouldn't be? what are you worried about happening?
deleted 6mo ago
[deleted]
thienannguyencv 6mo ago
[dead]

hijodelsol 6mo ago

I came here to say something similar. As someone who works in a field that applies machine learning but is not purely focused on it, I interact with people who think that arXiv is the only relevant platform and that they don't need to submit their work to any journal, as well as people who still think that preprints don't count at all and that data isn't published until it's printed in an academic journal. It can feel like a clash of worlds. I think both sides could learn from the other. In the case of ML, I understand the desire to move fast and that average time to publication of 250-300 days in some of the top-tier journals can feel like an unnecessary burden. But having been on both sides of peer review, there is value to the system and it has made for better work. Not doing any of it follows the same spirit as not benchmarking your approach against more than maybe one alternative and that already as an after-thought. Or benchmaxxing but not exploring the actual real-world consequences, time and cost trade offs, etc. Now, is academic publishing perfect? Of course not, very very far from it. It desperately needs to be reformed to keep it economically accessible, time efficient for both authors, editors and peer reviewers and to prevent the "hot topic of the day" from dominating journals and making sure that peer review aligns with the needs of the community and actually improves the quality of the work, rather than having "malicious peer review" to get some citations or pet peeves in. Given the power that the ML field holds and the interesting experiments with open review, I would wish for the field to engage more with the scientific system at large and perhaps try to drive reforms and improve it, rather than completely abandoning it and treating a PDF hosting service as a journal (ofc, preprints would still be desirable and are important, but they can not carry the entire field alone).

bonoboTP 6mo ago
Simply anticipating basic push backs from reviewers makes sure that you do a somewhat thorough job. Not 100% thorough and the reviews are sometimes frivolous and lazy and stupid. But just knowing that what you put out there has to pass the admittedly noisily gatekept gate of peer review overall improves papers in my estimation. There is also a negative side because people try to hide limitations and honest assessments and cherry pick and curate their tables more in anticipation of knee jerk reviewers but overall I think without any peer review, author culture would become much more lax and bombastic and generally trend toward engagement bait and social media attention optimized stuff. The current balance where people wrote a paper with reviers in mind, upload it to Arxiv before the review concludes and keep it on Arxiv even if rejected is a nice balance. People get to form their own opinion on it but there is also enough self-imposed quality control on it just due to wanting it to pass peer review, that even if it doesn't pass peer review, it is still better than if people write it in a way that doesn't care or anticipate peer review. And this works because people are somewhat incentivized to get peer reviewed official publications too. But being rejected is not the end of the world either because people can already read it and build on it based on Arxiv.
- bjourne 6mo ago
  I really am not sure about that: https://biologue.plos.org/wp-content/uploads/sites/7/2020/05/Peer-Review-Nick-Kim-cartoon3-resize.jpg https://biologue.plos.org/wp-content/uploads/sites/7/2020/05... The problem is that "optimizing for peer-review" is not the same thing as optimizing for quality. E.g., I like to add a few tongue-in-cheeks to entertain the reader. But then I have to worry endlessly about anal-retentive reviewers who refuse to see the big picture.
  - bonoboTP 6mo ago
    Currently a kind of rule of thumb is that a PhD student can graduate after approximately 3 papers published in a good peer reviewed venue. If peer review were to go away, this whole academic system would get into a crisis. It's dysfunctional and has many problems but it's kinda load bearing for the system to chug along.

ph4rsikal 6mo ago

My observation is that research, especially in AI has left universities, which are now focusing their research to a lesser degree on STEM. It appears research is now done by companies like Meta, OpenAI, Anthropic, Tencent, Alibaba, among many others.

bonoboTP 6mo ago
Universities (outside a few) just have much weaker PR machines so you never hear what they do. Also their work is not user facing products so regular people, even tech power users won't see them.
- 0x3f 6mo ago
  Not sure about that. How would a university test scaling hypotheses in AI, for example? The level of funding required is just not there, as far as I know.
  - rsfern 6mo ago
    This issue of accessibility is widely acknowledged in the academic literature, but it doesn’t mean that only large companies are doing good research. Personally I think this resource mismatch can help drive creative choice of research problems that don’t require massive resources. To misquote Feynman, there’s plenty of room at the bottom
  - oscaracso 6mo ago
    Universities are also not suited to test which race car is the fastest, but that does not obviate the need for academic research in mechanical engineering.
    - 0x3f 6mo ago
      Perhaps but the fastest race car is not possibly marshalling in the end of human involvement in science, so you might consider these of considerably different levels of meriting the funding.

stared 6mo ago

> arXiv fulfills its function better the less power it has as an institution It is an interesting instance of the rule of least power, https://en.wikipedia.org/wiki/Rule_of_least_power https://en.wikipedia.org/wiki/Rule_of_least_power.

fidotron 6mo ago
The irony of the TBL quotes there being the entire problem with the semantic web is the ontological tarpit that results due to the excessive expressive power of a general triple store.
- PaulHoule 6mo ago
  Well, I’d argue that many things in the semweb are not expressive enough and lead to the misunderstandings we have. People think, for instance, that RDFS and OWL are meant to SHACL people into bad an over engineered ontologies. The problem is these standards add facts and don’t subtract facts. At risk of sounding like ChatGPT: it’s a data transformation system not a validation system. That is, you’re supposed to use RDFS to say something like ?s :myTermForLength ?o -> ?s :yourTermForLength ?o . The point of the namespace system is not to harass you, it is to be able to suck in data from unlimited sources and transform it. Trouble is it can’t do the simple math required to do that for real, like ?s :lengthInFeet ?o -> ?s :lengthInInches 12*?o . Because if you were trying OWL-style reasoning over arithmetic you would run into Kurt Gödel kinds of problems. Meanwhile you can’t subtract facts that fail validation, you can’t subtract facts that you just don’t need in the next round of processing. It would have made sense to promote SHACL first instead of OWL because garbage-in-garbage out, you are not going to reason successfully unless you have clean data… but what the hell do I know, I’m just an applications programmer who models business processes enough to automate them. Similarly the problem of ordered collections has never been dealt with properly in that world. PostgreSQL, N1QL and other post-relational and document DB languages can write queries involving ordered collections easily. I can write rather unobvious queries by hand to handle a lot of cases (wrote a paper about it) but I can’t cover all the cases and I know back in the day I could write SPAQL queries much better than the average RDF postdoc or professor. As for underengineering, Dublin Core came out when I worked at a research library and it just doesn’t come close in capability to MARC from 1970. Larry Masinter over at Adobe had to hack the standard to handle ordered collections because… the authors of a paper sure as hell care what order you write their names in. And it is all like that: RDF standards neglect basic requirements that they need to be useful and then all the complex/complicated stuff really stands out. If you could get the basics done maybe people would use them but they don’t.

light_hue_1 6mo ago

> Unfortunately, over the years, arXiv has become something like a "venue" in its own right, particularly in ML, with some decently cited papers never formally published and "preprints" being cited left and right. Consider the impression you get when seeing a reference to an arXiv preprint vs. a link to an author's institutional website. This just isn't true. arXiv is not a venue. There's no place that gives you credit for arXiv papers. No one cares if you cite an arXiv paper or some random website. The vast vast majority of papers that have any kind of attention or citations are published in another venue.

contubernio 6mo ago
A Fields medal was awarded based mainly on this paper never published elsewhere: https://arxiv.org/abs/math/0211159 https://arxiv.org/abs/math/0211159
- auggierose 6mo ago
  I think there is a misunderstanding here. Does arXiv count as a publication? Yes, pretty much anything that gives you a DOI does, for example Zenodo. Does it function as a reputable anything? No. The paper you link to counts as a publication, but its reputation stands on its own, it has nothing to do with arXiv as a venue. Ideally, that's how it is for all papers, but it isn't, just by publishing in certain venues your paper automatically gets a certain amount of reputation depending on the venue.
  - fc417fc802 6mo ago
    > Ideally, that's how it is for all papers, but it isn't We require a method of filtering such that a given researcher doesn't have to personally vet in excruciating detail every paper he comes across because there simply isn't enough time in the day for that. Ideally such a system would individually for each paper provide a multi-dimensional score that was reputable. How can those be calculated in a manner such that they're reputable? Who knows; that exercise is left for the reader. In practice "well it got published in Nature" makes for a pretty decent spam filter followed by metrics such as how many times it's been cited since publication, checking that the people citing it are independent authors who actually built directly on top of the work, and checking how many of such citing authors are from a different field.

queuebert 6mo ago

> Unfortunately, over the years, arXiv has become something like a "venue" in its own right, ... In my experience as a publishing scientist, this is partly because publishing with "reputable" journals is an increasingly onerous process, with exorbitant fees, enshittified UIs, and useless reviews. The alternative is to upload to arXiv and move on with your life.

groundzeros2015 6mo ago
That’s true. But that’s separate than the use in ML in Blockchain circles as a form of a marketing - using academic appearances.
- jjk166 6mo ago
  That sounds more like an issue of certain fields having crappy standards because the people in those fields benefit from crappy standards than an issue with the site they happen to host papers on.
  - deleted 6mo ago
    [deleted]
  - groundzeros2015 6mo ago
    I don’t buy “some fields are just more honorable”. Everyone uses publishing for personal gain. But yes it’s a people problem, not an arxiv problem.
- StableAlkyne 6mo ago
  Every field and every publisher has this issue though. I've read papers in the chemical literature that were clearly thinly veiled case studies for whatever instrument or software the authors were selling. Hell, I've read papers that had interesting results, only to dig into the math and find something fundamentally wrong. The worst was an incorrect CFD equation that I traced through a telephone game of 4 papers only to find something to the effect of "We speculate adding $term may improve accuracy, but we have not extensively tested this" Just because something passed peer review does not make it a good paper. It just means somebody* looked at it and didn't find any obvious problems. If you are engaged in research, or in a position where you're using the scientific literature, it is vital that you read every paper with a critical lens. Contrary to popular belief, the literature isn't a stone tablet sent from God. It's messy and filled with contradictory ideas. *Usually it's actually one of their grad students

Aurornis 6mo ago

> and with just enough moderation to not devolve into spam and chaos arXiv has become a target for grifters in other domains like health and supplements. I’ve seen several small scale health influencers who ChatGPT some “papers” and then upload them to arXiv, then cite arXiv as proof of their “published research”. It’s not fooling anyone who knows how research work but it’s very convincing to an average person who thinks that that they’re doing the right thing when they follow sources that have done academic research. I’ve been surprised as how bad and obviously grifty some of the documents I’ve seen on arXiv have become lately. Is there any moderation, or is it a free for all as long as you can get an invite?

deleted 6mo ago
[deleted]

aimarketintel 6mo ago

[flagged]

PaulHoule 6mo ago

Review papers are interesting. Bibliometrics reveal that they are highly cited. Internal data we had at arXiv 20 years ago show they are highly read. Reading review papers is a big part of the way you go from a civilian to an expert with a PhD. On the other hand, they fall through the cracks of the normal methods of academic evaluation. They create a lot of value for people but they are not likely to advance your career that much as an academic, certainly not in proportion to the value they create, or at least the value they used to create. One of the most fun things I did on the way to a PhD was writing a literature review on giant magnetoresistance for the experimentalist on my thesis committee. I went from knowing hardly anything about the topic to writing a summary that taught him a lot he didn't know. Given any random topic in any field you could task me with writing a review paper and I could go out and do a literature search and write up a summary. An expert would probably get some details right that I'd get wrong, might have some insights I'd miss, but it's actually a great job for a beginner, it will teach you the field much more effectively than reading a review paper! How you regulate review papers is pretty tricky. If it is original research the criterion of "is it original research" is an important limit. There might already be 25 review papers on a topic, but maybe I think they all suck (they might) and I can write the 26th and explain it to people the way I wish it was explained to me. Now you might say in the arXiv age there was not a limit on pages, but LLMs really do problematize things because they are pretty good at summarization. Send one off on the mission to write a review paper and in some ways they will do better than I do, in other ways will do worse. Plenty of people have no taste or sense of quality and they are going to miss the latter -- hypothetically people could do better as a centaur but I think usually they don't because of that. One could make the case that LLMs make review papers obsolete since you can always ask one to write a review for you or just have conversations about the literature with them. I know I could have spend a very long time studying the literature on Heart Rate Variability and eventually made up my mind about which of the 20 or so metrics I want to build into my application and I did look at some review papers and can highlight sentences that support my decisions but I made those decisions based on a few weekends of experiments and talking to LLMs. The funny thing is that if you went to a conference and met the guy who wrote the review paper and gave them the hard question of "I can only display one on my consumer-facing HRV app, which one do I show?" they would give you that clear answer that isn't in the review paper and maybe the odds are 70-80% that it will be my answer.

abdullahkhalids 6mo ago

> Unfortunately, over the years, arXiv has become something like a "venue" in its own right, particularly in ML, with some decently cited papers never formally published and "preprints" being cited left and right. This has been a common practice in physics, especially the more theoretical branches, since the inception of arXiv. Senior researchers write a paper draft, and then send copies to some of their peers, get and incorporate feedback, and just submit to arxiv.

godelski 6mo ago
And this is really how it should be. Honestly the only thing I want arxiv to do is become more like open review. Allow comments by peers and some better linking to data and project pages. It works for physics because physicists are very rigorous. So papers don't change very much. It also works for ML because everyone is moving very fast that it's closer to doing open research. Sloppier, but as long as the readers are other experts then it's generally fine. I think research should really just be open. It helps everyone. The AI slop and mass publishing is exploiting our laziness; evaluating people on quantity rather than quality. I'm not sure why people are so resistant to making this change. Yes, it's harder, but it has a lot of benefits. And at the end of the day it doesn't matter if a paper is generated if it's actually a quality paper (not in just how it reads, but the actual research). Slop is slop and we shouldn't want slop regardless. But if we evaluate on quality and everything is open it becomes much easier to figure out who is producing slop, collision rings, plagiarist rings, and all that. A little extra work for a lot of benefits. But we seem to be willing to put in a lot of work to avoid doing more work
- abdullahkhalids 6mo ago
  I don't agree actually that is how it should or can work for everyone. Senior researchers produce good quality research, and they have a network of high quality peers built over decades. Both those are necessary for them to reach out and ask for feedback, and get genuine and high quality feedback. Junior researchers don't have these typically. They also benefit more from anonymous feedback, which enables the reviewers to bluntly identify wrong or close to wrong results. So I think open journals should continue to exist. They fill an essential role in the scientific ecosystem.

fsckboy 6mo ago

>We've seen the kind of appeasement prose from their statement and FAQ [1] countless times before what are you referring to, who is being appeased who shouldn't be? what are you worried about happening?

deleted 6mo ago

[deleted]

thienannguyencv 6mo ago

[dead]