5 ms·
First impressions: 1. The data in most of the plots (see the appendix) look fake. Real life data does not look that clean. 2. In May of 2022, 6 months before
by intoamplitudes 1y ago
First impressions:
1. The data in most of the plots (see the appendix) look fake. Real life data does not look that clean.
2. In May of 2022, 6 months before chatGPT put genAI in the spotlight, how does a second-year PhD student manage to convince a large materials lab firm to conduct an experiment with over 1,000 of its employees? What was the model used? It only says GANs+diffusion. Most of the technical details are just high-level general explanations of what these concepts are, nothing specific.
"Following a short pilot program, the lab began a large-scale rollout of the model in May of 2022." Anyone who has worked at a large company knows -- this just does not happen.
- pixl97 1y ago>The data in most of the plots (see the appendix) look fak Could a Benford's Law analysis apply here to detect that?
- constantcrying 1y agoHow would you apply it, why would it be applicable?
- tough 1y agoFake data is usually too clean
- constantcrying 1y agoAnd? How is that at all a relevant observation?
- tough 1y agoAnyone looking at inly the data objectively should br able to comento terms that is distribution is unnatural, as it turns out to fake organic isnt as easy
- constantcrying 1y agoThis was about Benford's law.
- tough 1y agoSorry for replying to the wrong thread, should have just been a general reply indeed
- pixl97 1y agoAnd what exactly do you think Benfords law is?
- btrettel 1y agoOn point 2, the study being apparently impossible to conduct as described was also a problem for Michael LaCour. Seems like an underappreciated fraud-detection heuristic. https://en.wikipedia.org/wiki/When_Contact_Changes_Minds https://en.wikipedia.org/wiki/When_Contact_Changes_Minds https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=f0d1d857ed05912c69f9e1689597656f828a8e5f https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d... > As we examined the study’s data in planning our own studies, two features surprised us: voters’ survey responses exhibit much higher test-retest reliabilities than we have observed in any other panel survey data, and the response and reinterview rates of the panel survey were significantly higher than we expected. > The firm also denied having the capabilities to perform many aspects of the recruitment procedures described in LaCour and Green (2014).
- constantcrying 1y agoA month by month record of scientists time spend on different tasks is on its face absurd. The proposed methodology, automatic textual analysis of scientists written records, giving you a year worth of a near constant time split pre AI is totally unbelievable. The data quality for that would need to be unimaginably high.
- mzs 1y ago% gunzip -c arXiv-2412.17866v1.tar.gz | tar xOf - main.tex | grep '\bI have\b' To summarize, I have established three facts. First, AI substantially increases the average rate of materials discovery. Second, it disproportionately benefits researchers with high initial productivity. Third, this heterogeneity is driven almost entirely by differences in judgment. To understand the mechanisms behind these results, I investigate the dynamics of human-AI collaboration in science. \item Compared to other methods I have used, the AI tool generates potential materials that are more likely to possess desirable properties. \item The AI tool generates potential materials with physical structures that are more distinct than those produced by other methods I have used. % gunzip -c arXiv-2412.17866v1.tar.gz | tar xOf - main.tex | grep '\b I \b' | wc 25 1858 12791 %
- rafram 1y agoNot sure what you’re trying to say.
- kccqzy 1y agoMaybe the point is that it is rare for a paper to have the pronoun "I" so many times. Usually the pronoun "we" is used even when there is a single author.
- pbhjpbhj 1y agoIt's a single author. https://arxiv.org/pdf/2412.17866 https://arxiv.org/pdf/2412.17866
- muhdeeb 1y agoAgreed! It’s pretty alien. I’ve seen brilliant single author work, but nothing that uses “I” unless it’s a blog post. The formal papers are always the singular “we”. Feels very communal that way! Nice to include the giants we stand on as implied coauthors.
- kragen 1y ago
- raphman 1y agoFWIW, in the q&a after a talk, he claims that it was a GNN (graph neural network), not a GAN. (In this q&a, the audience does not really question the validity of the research.) https://doi.org/10.52843/cassyni.n74lq7 https://doi.org/10.52843/cassyni.n74lq7
- mncharity 1y agoWayback of the Sloan School seminar page shows him doing one on February 24, 2025. I wonder how that went. I miss google search's Cache. As with the seminar, several other hits on MIT pages have been removed. I'm reminded of a PBS News Hour story, on free fusion energy from water in your basement (yes, really), which was memory holed shortly after. The next-ish night they seemed rather put out, protesting they had verified the story... with "a scientist". That cassyni talk link... I've seen a lot of MIT talks (a favorite mind candy), and though Sloan talks were underrepresented, that looked... more than a little odd. MIT Q&A norms are diverse, from the subtle question you won't appreciate if you haven't already spotted the fatal flaw, to bluntness leaving the speaker in tears. I wonder if there's a seminar tape.
- radicaldreamer 1y agoHere’s a podcast from Sloan where David Autor is talking about the work as if it is perfectly valid https://sloanreview.mit.edu/audio/feed-drop-how-ai-will-change-your-job-with-mits-david-autor/ https://sloanreview.mit.edu/audio/feed-drop-how-ai-will-chan...
- mncharity 1y agoThe feed drop appears to be this podcast[1], posted Feb 3. [1] https://www.youtube.com/watch?v=giGrMKDc0O0 https://www.youtube.com/watch?v=giGrMKDc0O0
- rdtsc 1y agoOh interesting. I haven't talked to any recent graduates but I would expect an MIT PhD student to be more articulate and not say "like" every other word. There was a question at the end that made him a little uncomfortable: [1:00:20] Q: Did you use academic labs only or did you use private labs? A: (uncomfortable pause) Oh private, yeah, so like all corporate, yeah... Q: So, no academic labs? A: I think it's a good question (scratches head uncomfortably, seemingly trying to hide), what this would look like in an academic setting, cause like, ... the goals are driven by what product we're going make ... academia is all, like "we're looking around trying to create cool stuff"... My 8 year-old is more articulated than this person. Perhaps they are just nervous, I'll give them that I guess.
- raphman 1y agoOh, he also claimed that he got IRB approval from "MIT’s Committee on the Use of Humans as Experimental Subjects under ID E-5842. JEL Codes: O31, O32, O33, J24, L65." before conducting this research, i.e., at a time when he wasn't even a PhD student.
- 3s 1y agoI agree with point 1, at least superficially. But re: point 2, there are a lot of companies with close connections to MIT (and other big institutions like Stanford) that are interested in deploying cutting edge research experiments, especially if they already have established ties with the lab/PI
- lumost 1y agoIf a paper is difficult to replicate in a high volume field.. will it ever be replicated? The question we should be asking is how many fraudulent papers are there in the field? I’ve even worked in places where some ML researchers seemingly made up numbers for years on end.