7 ms·
GPT-fabricated scientific papers on Google Scholar
- layer8 2y agoI appreciate that, appropriately, the article image is not AI-generated.
- judge2020 2y agoI was able to get pretty close with chatgpt: https://rr.judge.sh/Commabutterfly/76b34e/nmwguWGt8pIe.jpg https://rr.judge.sh/Commabutterfly/76b34e/nmwguWGt8pIe.jpg > create a picture of scrabble pieces strewn on a table, with a closeup of a line of scrabble letters spelling "CHATGPT" on top of them. photographic, realistic quality, maintain realism and believability
- Daub 2y agoI prefer yours. Better lighting.
- doesnt_know 2y agoThe points on all the tiles are messed up and there are tiles with random squiggles where there should be letters...
- GaggiX 2y agoYou can get much better results with Ideogram 2 (also free): https://ideogram.ai/assets/image/lossless/response/vF81gKjHSA6fYUSxyzzljQ https://ideogram.ai/assets/image/lossless/response/vF81gKjHS... https://ideogram.ai/assets/image/lossless/response/EcRpDLumSAOGdLsCSraBhw https://ideogram.ai/assets/image/lossless/response/EcRpDLumS... Almost the same prompt.
- renewiltord 2y agoGreat tip. The text handling here is far superior.
- vunderba 2y agoYou can also get reasonably close with an open model that you can run locally (flux dev). https://replicate.com/p/xm41nvz05drm00chsywb6am7f0 https://replicate.com/p/xm41nvz05drm00chsywb6am7f0 https://replicate.com/p/kdw8bnkj39rm40chsyzbyg5e04 https://replicate.com/p/kdw8bnkj39rm40chsyzbyg5e04 But of course anyone who has even a passing familiarity with scrabble is going to be able to tell that something's off.
- GaggiX 2y agoThe biggest problem with the default Flux model is that it generates images with that strong AI look, probably caused by the distillation of the CFG. You should try some LoRAs for this, and also prompt the model to generate the rack that holds the letters.
- vunderba 2y agoGood point. I have a comfyui setup for it but its super basic right now just the diffusion model / clip loader / vae. Another thing you've probably noticed is that 99% of images from Flux tend to have that classic narrow depth of field look. I've seen people occasionally be able to get around it with pretty amusing prompt tokens like "instagram photo, selfie, gopro, etc." though.
- layer8 2y agoThe number markings on the Scrabble pieces are nonsensical, the wooden ground looks like plastic, there are strange artifacts like the white smudge on the edge of the “E” tile in the front, and so on. AI-generated images are clearly identifiable as such, and it just gets annoying to continually see those desultory fabrications.
- snakeyjake 2y ago[flagged]
- EnigmaFlare 2y agoIt's silly that there's a stigma attached to AI generated images in cases where it's perfectly reasonable to do. People seem to appreciate things more for the fact that they were created by spending time out of another human's life more than what it actually is.
- __loam 2y agoIt's built on theft and it's a negative quality signal usually.
- layer8 2y agoIt would be silly if they were indistinguishable from human-created images, but they aren't, exhibiting the typical AI artifacts and weirdness, and thereby signal a lack of care/caring.
- EnigmaFlare 2y ago"lack of care" - that's the part about spending time out of another human's life. It's not the poor quality that's the problem but the lack of human effort. Oil paintings are full of visible brush strokes which are an artifact but people love them. For most applications of art - advertising, background decorations, news article pictures, etc. there really is no need to show that humans spent effort on it. The human effort idea is even a bit morally objectionable. You can feel that you're worth more than others because more of the lives of others were consumed to create your possessions. It's a zero sum game where poor people can never afford high-care art because their time is worth less than the artist's.
- nomilk 2y agoGPT might make fabricating scientific papers easier, but let's not forget how many humans fabricated scientific research in recent years - they did a great job without AI! For any who haven't seen/heard, this makes for some entertaining and eye-opening viewing! https://www.youtube.com/results?search_query=academic+fraud https://www.youtube.com/results?search_query=academic+fraud
- bumby 2y agoIs there good data on how many are fraudulent? I know there’s reasonable data on replicability issues, but that’s potentially different.
- benreesman 2y agoI think it’s important to remember that while the tidal wave of spam just starting to crest courtesy of the less scrupulous LLM vendors is uh, necessary to address, this century’s war on epistemology was well underway already in the grand traditions of periodic wars on the idea that facts are even aspirationally, directionally worthwhile. The phrase “alternative facts” hit the mainstream in 2016 and the idea that resistance is futile on broad-spectrum digital weaponized bytes was muscular then (that was around the time I was starting to feel ill for being a key architect of it). Now technology is a human artifact and always ends up resembling its creators or financiers or both: I’d have nice fonts on my computer in 2024 most likely either way, but it’s directly because of Jobs they were available in 1984 to a household budget. If someone other than Altman had or some other insight than “this thing can lie in a newly scalable way” was the escape velocity moment on LLMs then we’d still have test sets and metrics and just science going on in the Commanding Heights of the S&P 500, but these people are a symptom of our apathy around any noble instinct. If we had stuck firm on our values no effective altruism cult leader type would even make the press.
- __loam 2y agoPost-modernism was a mistake.
- 2y ago
- jeremynixon 2y agoThere is article shows no evidence of fabrication, fraud or misinformation, while making accusations of all of them. All it shows is that ChatGPT was used, which is wildly escalated into "evidence manipulation" (ironically without evidence). Much more work is needed to show that this means anything.
- viraptor 2y agoIf the result was not read even to check for obvious boilerplate GPT markers, then we can't expect anything else in them was. That means anything else, numbers, interpretation, conclusion was potentially never checked. The authors use fraud in a specific sense here: "using ChatGPT fraudulently or undeclared" where they proved that the produced text was included without proper review. They also never accused those papers of misinformation, so they don't need to show evidence of that.
- OutOfHere 2y agoJust because ChatGPT was used to help write a paper doesn't in itself mean that the data or findings are fabricated.
- ceejayoz 2y agoSure, but there are some... pretty egregious cases. https://mashable.com/article/ai-rat-penis-diagram-midjourney-science https://mashable.com/article/ai-rat-penis-diagram-midjourney...
- hakanderyal 2y agoThat’s the funniest piece of writing I’ve read in a longtime, thanks! I wonder what they were thinking submitting the paper.
- CatWChainsaw 2y agoThey let the machines think for them, that's the whole problem.
- riedel 2y agoTrue. I am seeing chatgpt used by my colleagues (mostly no English native speakers) day to day and it mostly improves their writing (except for those wotfd that pop up a bit too often [0] like utilize [1]). So not all bad. I am also hearing that a lot of reviewers and readers use it though. So we are often joking that PhD students (in CS) nowadays only write bullet point from their research. Generate prose that is used to generate bullet points. [0] https://www.scientificamerican.com/article/chatbots-have-thoroughly-infiltrated-scientific-publishing/ https://www.scientificamerican.com/article/chatbots-have-tho... [1] https://medium.com/learning-data/words-and-phrases-that-make-it-obvious-you-used-chatgpt-2ba374033ac6 https://medium.com/learning-data/words-and-phrases-that-make...
- CuriouslyC 2y agoScientific writing is pretty bad usually so I'll count this as an improvement
- hodgesrm 2y ago> Two main risks arise... First, the abundance of fabricated “studies” seeping into all areas of the research infrastructure... A second risk lies in the increased possibility that convincingly scientific-looking content was in fact deceitfully created with AI tools... A third risk: ChatGPT has no understanding of "truth" in the sense of facts reported by established, trusted sources. I'm doing a research project related to use of data lakes and tried using ChatGPT to search for original sources. It's a shitshow of fabricated links and pedestrian summaries of marketing materials. This feels like an evolutionary dead end.
- kenjackson 2y agoIt sounds like your use of AI is one of the worst uses. Standard semantic search would be much better and appropriate.
- HeatrayEnjoyer 2y agoHow do you run a semantic search
- hodgesrm 2y agoNo disagreement with that. My expectations were not high--but I was still surprised how bad it was. There are absolutely no guardrails.
- nis0s 2y agoIf summarization and analysis isn’t the main use of AI, then what is?
- passion__desire 2y agoExistence of LLMs make Google search even more relevant for cross-checking rather than less relevant for deep research. Daniel Dennett said we should have all levels of searches available for everyone i.e. from basic string matching to semantic matching. [0] [0] https://youtu.be/arEvPIhOLyQ?t=1139 https://youtu.be/arEvPIhOLyQ?t=1139
- deleted 2y ago[deleted]
- gerdesj 2y agoColour me surprised. An IT related search will generally end up with loads of returns that lead to AI generated wankery. For example, suppose you wish to back up switch configs or dump a file or whatever and tftp is so easy and simple to setup. You'll tear it down later or firewall it or whatever. So a quick search "linux tftp serevr" gets you to say: https://thelinuxcode.com/install_tftp_server_ubuntu/ https://thelinuxcode.com/install_tftp_server_ubuntu/ All good until you try to use the --create flag which should allow you to upload to the server. That flag is not valid for tftp-hpa, it is valid on tftpd (another tftp daemon) That's a hallucination. Hallucinations are fucking annoying and increasingly prevalent. In Windows land the humans hallucinate - C:\ SFC /SCANNOW does not fix anything except for something really madly self imposed.
- bongodongobob 2y agoIt's funny you mention this because yesterday I had it write me a shell script to set up a TFTP server from scratch. I had it walk me through the process first, then said "ok now make that into a script." And it did and it works.
- deleted 2y ago[deleted]
- viraptor 2y agoThat's not an AI hallucination. The content comes from Ubuntu community wiki https://help.ubuntu.com/community/TFTP https://help.ubuntu.com/community/TFTP - it was written in 2015. And at least in Debian, tftpd-hpa man page lists --create as valid https://manpages.debian.org/testing/tftpd-hpa/tftpd.8.en.html https://manpages.debian.org/testing/tftpd-hpa/tftpd.8.en.htm... Seems valid upstream too https://github.com/Distrotech/tftp-hpa/blob/5e95f248e8435eb3413eb2a2d377d17e37b712ff/tftpd/tftpd.c#L332 https://github.com/Distrotech/tftp-hpa/blob/5e95f248e8435eb3...
- mkl 2y agoIt says to put the --create option in /etc/default/tftpd-hpa. tftpd-hpa does support --create (at least on Ubuntu). The client program tftp-hpa (no d) doesn't support --create, but that's not what the instructions are talking about.
- deleted 2y ago[deleted]
- intellectualx 2y ago[flagged]
- Barrin92 2y agoHonestly what we need to do is establish much stronger credentialing schemes. The "only a good guy with an AI can stop a bad guy with an AI" approach of trying to filter out bad content is just a hopeless arms race and unproductive. In a sense we need to go back two steps and websites need to be much stronger curators of knowledge again, and we need some reliable ways to sign and attribute real authorship to publications. So that when someone publishes a fake paper there is always a human being who signed it and can be held accountable. There's a practically unlimited number of automated systems, but only a limited number of people trying to benefit from it. In the same way https went from being rare to being the norm because the assumption that things are default-authentic doesn't hold, the same just needs to happen to publishing. If you have a functioning reputation system and you can put on a price on fake information 99% of it is dis-incentivized.
- tbrownaw 2y agoIs this not already a thing? You can look up purported papers by DOI, and whatever journal it came from supposedly had it reviewed and should know who sent it to them. (And if that doesn't work, how is what you're suggesting meaningfully different?)
- Barrin92 2y agoIt's not at all a thing. Here's a recent study looking at citation fraud on Google Scholar including professional citation boosting services including with fake identities. It's widespread practice. https://arxiv.org/abs/2402.04607 https://arxiv.org/abs/2402.04607 Having a machine verifiable, cryptographic identity system that renders these kinds of things transparent, basically the equivalent of a ledger but instead of using it for get-rich schemes using it for identity would probably make verification enforceable.
- refibrillator 2y agoHmm there may be a bug in the authors’ python script that searches google scholar for the phrases "as of my last knowledge update" or "I don't have access to real-time data". You can see the code in appendix B. The bug happens if the ‘bib’ key doesn’t exist in the api response. That leads to the urls array having more rows than the paper_data array. So the columns could become mismatched in the final data frame. It seems they made a third array called flag which could be used to detect and remove the bad results, but it’s not used any where in the posted code. Not clear to me how this would affect their analysis, it does seem like something they would catch when manually reviewing the papers. But perhaps the bibliographic data wasn’t reviewed and only used to calculate the summary stats etc.
- viraptor 2y agoThat sounds important enough to contact the authors. Best case, they fixed it up manually; worst case, lots of papers are publicly accused of being made up and the whole farming/fish-focused summary they produced is completely wrong.
- hiddencost 2y agohttps://www.hb.se/en/research/research-portal/researchers/JUHA/ https://www.hb.se/en/research/research-portal/researchers/JU... Contact info for the first author
- Lerc 2y agoAs a tangent to the paper topic itself, what should be the standard procedure for publishing data gathering code like this? Given that they don't specify which version of any libraries or APIs used and that updates occur over time, API's change etc. inevitably resulting in code rot. It will eventually be impossible to figure out exactly what this code did. With meticulous version records it should at least be possible to ascertain what the code did by reconstructing that exact version (assuming stored back versions exist)
- jerpint 2y agoUsing a colab with printed outputs could be a good option to at the very least hint to reproducing results independently
- Strilanc 2y agoWhen I went to the APS March Meeting earlier this year, I talked with the editor of a scientific journal and asked them if they were worried about LLM generated papers. They said actually their main worry wasn't LLM-generated papers, it was LLM-generated reviews. LLMs are much better at plausibly summarizing content than they are at doing long sequences of reasoning, so they're much better at generating believable reviews than believable papers. Plus reviews are pretty tedious to do, giving an incentive to half-ass it with an LLM. Plus reviews are usually not shared publicly, taking away some of the potential embarrassment.
- matusp 2y agoWe already got an LLM generated meta review that was very clearly just summarization of reviews. There were some pretty egregious cases of borderline hallucinated remarks. This was ACL Rolling Review, so basically the most prestigious NLP venue and the editors told us to suck it up. Very disappointing and I genuinely worry about the state of science and how this will affect people who rely on scientometric criteria.
- reliabilityguy 2y agoWell, given that the only thing that matters for tenure reviews is the “service”, i.e., roughly a list of conferences the applicant reviewed/performed some sort of service at, this is barely a surprise. Right now there is now incentive to do a high quality review unless the reviewer is motivated.
- nextaccountic 2y agoCould you share it publicly or would you face adverse consequences? If you can please publish it and maybe post here on HN or reddit.
- Der_Einzige 2y agoWith NeurIPS 2024 reviews going on right now, I'm sure that a whole lot of these kind of reviews are being generated daily.
- known 2y ago[dead]
- oefrha 2y agoHow about people stop responding to titles for a change. This isn’t about papers that merely used ChatGPT and got caught by some cutting edge detection techniques, it’s about papers that blatantly include ChatGPT boilerplates like > “as of my last knowledge update” and/or “I don’t have access to real-time data” which suggests no human (don’t even need to be a researcher) read every sentence of these damn “papers”. That’s a pretty low bar to clear, if you can’t even bother to read generated crap before including it in your paper, your academic integrity is negative and not a word from you can carry any weight.
- RobotToaster 2y ago> which suggests no human (don’t even need to be a researcher) read every sentence of these damn “papers”. Which also suggests none of the so called reviewers or editors read the entire paper before including it in their journal...
- pcrh 2y agoThis kind of fabricated result is not a problem for practitioners in the relevant fields, who can easily distinguish between false and real work. If there are instances where the ability to make such distinctions is lost, it is most likely to be so because the content lacks novelty, i.e. it simply regurgitates known and established facts. In which case it is a pointless effort, even if it might inflate the supposed author's list of publications. As to the integrity of researchers, this is a known issue. The temptation to fabricate data existed long before the latest innovations in AI, and is very easy to do in most fields, particularly in medicine or biosciences which constitute the bulk of irreproducible research. Policing this kind of behavior is not altered by GPT or similar. The bigger problem, however, is when non-experts attempt to become informed and are unable to distinguish between plausible and implausible sources of information. This is already a problem even without AI, consider the debates over the origins of SARS-CoV2, for example. The solution to this is the cultivation and funding of sources of expertise, e.g. in Universities and similar.
- EnigmaFlare 2y agoNon-experts actually attempting to become informed (instead of just feeling like they're informed) can easily tell the difference too. The people being fooled are the ones who want to be fooled. They're looking for something to support their pre-existing belief. And for those people, they'll always find something they can convince themselves supports their belief, so I don't think it matters what false information is floating around. It seems to be kind of a new thing for laymen to be reading scientific papers. 20 years ago, they just weren't accessible. You had to physically go to a local university library and work out how to use the arcane search tools, which wouldn't really find what you wanted anyway. And even then, you couldn't take it home and half the time you couldn't even photocopy it because you needed a student ID card to use the photocopier.
- kgeist 2y agoI wonder how many of the GPT-generated papers are actually made by people whose native language is not English and who want to improve their English. That would explain various "as of my last knowledge update" still left intact in the papers, if the authors don't fully understand what it means.
- diggan 2y agoI'm guessing that we don't want people to write papers in a language where they don't understand "as of my last knowledge update", as probably a lot of terms in their paper have more advanced language than that. Would be better in those cases for people to write their paper in their native language and let readers translate it for themselves.
- EasyMark 2y agoIt’s not a black and white problem. Some people may have good ability to read but not write/speak a language (I’m that way with Spanish) so the cases will vary as to which would work best user or author translated, it could be good to include both version in any given paper and fix both problems.
- daghamm 2y agoLast time we discussed this, someone basically searched for phrases such as "certainly I can do X for you" and assumed that meant GPT was used. HN noticed that many of the accused papers actually predated openai. Hope this research is better.
- xandrius 2y agoHow else would that phrase go into a real paper then?
- tkgally 2y agoFor a paper that includes both a broad discussion of the scholarly issues raised by LLMs and wide-ranging policy recommendations, I wish the authors had taken a more nuanced approach to data collection than just searching for “as of my last knowledge update” and/or “I don’t have access to real-time data” and weeding out the false positives manually. LLMs can be used in scholarly writing in many ways that will not be caught with such a coarse sieve. Some are obviously illegitimate, such as having an LLM write an entire paper with fabricated data. But there are other ways that are not so clearly unacceptable. For example, the authors’ statement that “[GPT’s] undeclared use—beyond proofreading—has potentially far-reaching implications for both science and society” suggests that, for them, using LLMs for “proofreading” is okay. But “proofreading” is understood in various ways. For some people, it would include only correcting spelling and grammatical mistakes. For others, especially for people who are not native speakers of English, it can also include changing the wording and even rewriting entire sentences and paragraphs to make the meaning clearer. To what extent can one use an LLM for such revision without declaring that one has done so?
- RobotToaster 2y agoIf the papers are correct, what does it matter if the author used AI? If the papers are incorrect, then the reviewers should catch them.
- rosmax_1337 2y agoI think we might be entering a dark age of sorts.
- kazinator 2y ago> often controversial topics susceptible to disinformation: ... and computing Ouch!