13 ms·
Half of my PhD thesis is considered "unpublishable" because, after doing the work, my supervisors felt it's actually "unsurprising" that it didn't work out. We
by kraetzin 6y ago
Half of my PhD thesis is considered "unpublishable" because, after doing the work, my supervisors felt it's actually "unsurprising" that it didn't work out. We took methods that had been exploited to improve on previous results for over a decade to their logical extreme, and found that this method no longer leads to improvements. After doing the work it seems obvious. A paper on the subject would almost be considered uninteresting, and a high ranking journal would ignore it (which is why it's considered "unpublishable"). However, nobody has published this information, and it would help others to not make the same mistake.
I wonder how many times similar "mistakes" have been made by PhD students across disciplines.
- owenshen24 6y agoThat sucks to hear. Null results are important, as you say, if only to dissuade others from doing the same. See also the "file-drawer problem" (https://en.wikipedia.org/wiki/Publication_bias https://en.wikipedia.org/wiki/Publication_bias). Also, with regards to the incentives in the field and the lack of null results, there's always Ioannidis's classic work (https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...).
- kraetzin 6y agoThose are some good reads, and have absolutely been my experience. It's depressing how many publications that I've come across don't provide the whole story, and are probably false. I've found that looking at what a paper doesn't report can be far more important that what they claim.
- mycall 6y agoI wonder how many novel techniques could come out of these types of reports if they were actually analyzed by ML or NLP.
- owenshen24 6y agoI definitely think there's more room for this sort of guided / ML analysis, but I'm not quite sure to make traction on extracting the structure of scientific papers...hopefully someone with more experience can chime in. I think paper discovery has recently suffered a huge boon thanks to ConnectedPapers, though. [https://www.connectedpapers.com/ https://www.connectedpapers.com/]
- mikk14 6y agoI disagree. Negative results are important. Null results are of very limited interest. The two are worlds apart. A null result simply means you tried something and it didn't work. But you don't know why. You haven't proven it didn't work. There are literally millions of reasons why something might not work. For instance, you could try to use compound X to cure disease Y, observe no effect, and conclude that X doesn't cure Y. But what if somewhere in the process of making X you made an uncaught mistake and you instead used X'? A negative result means that you tried something and you came to the proven conclusion it doesn't work. This is, crucially, as hard to obtain as a positive result. In my example, it would imply a much longer process than simply "apply X, see no effect in Y, make a few robustness checks, done". You could say "Well, publish the null anyway, somebody will catch the mistake". Unlikely. There are already so many papers out there that keeping up is impossible. If we were publishing also null results this number will grow tenfold at the very least. Nobody could possibly check everything. They will see a paper "X doesn't cure Y" and call it knowledge, stifling a possible cure virtually forever. Am I splitting hairs? Perhaps. But I think HN prizes itself to be a scientifically minded community, and thus it has a mandate to use terms correctly. Confusing "null" with "negative" is a sin. I hope one day I'll find a way to strongly and passionately argue against the "null results are as important as positive results" position. It is a bad meme. Charitably, I consider it most of the times a honest mistake. But sometimes it gives me the impression it is a cheap trick used by people to erode the reputation of academia.
- tralarpa 6y ago> Negative results are important. Null results are of very limited interest. Correct. There is a highly cited paper in CS where the author showed that a mathematical model that was widely used in research didn't actually work (anymore) in reality. That paper was the starting point of a lot of new research in that field.
- shrimpx 6y agoCan you add a citation for that paper?
- learnstats2 6y ago
- notagoodidea 6y agoBy experience, I want to say : way too much. Journals and published articles looks like a research lab full of PhD working on their owns without access to all the previous results of the labs (good, bad and everything in between) and do not talk to others unless they have a 10 minutes seminar every six months to show some stuff quickly.
- bumby 6y agoMy experience: I proposed a thesis to an advisor who deemed it unlikely to work. He ran it by a colleague who came to the same conclusion. I went to a different major and pursued the same thesis and invited the previous faculty who initially turned it down to the defense because it was relevant to their field. It was frustrating to hear them voice their opinions in the defense that they felt, “of course it world work.” After seeing the data, they took the exact opposite side claiming it was obvious to the point of being of limited publishing value.
- executesorder66 6y agoDidn't you ask them "But didn't you say this was uspublishable?" and what was their response?
- bumby 6y agoI did not. Maybe I was being weak or maybe I didn’t want to try and make them look bad in front of their peers but I did not bring up any of the previous conversations during the defense
- executesorder66 6y agoThanks for the response. I'm always just curious how people react to things like that.
- raincom 6y agoAcademia is 'full' of people who lack intellectual integrity. Imagine Frege's response to Bertrand Russell's letter concerning barber's paradox: 'it's obvious!' Instead of doing that, Frege openly acknowledged Russell's criticism in his book.
- PeterStuer 6y agoImagine how much worse it is outside where there's even no pretense of adhering to any intellectual rigor. Also within academia there is still a wide spectrum of intellectual rigor across the disciplines. Some things are just more verifiable than others.
- user32556 6y agoIMO, nowadays top-tier conference paper tends to focus too much on telling an interesting story. This makes researchers only show the surprising result in the paper. The unsurprising one is hardly mentioned. However, the uninteresting part would definitely help others to not make the same mistake. The uninteresting result is still result (and contribution), isn't it?
- shp0ngle 6y agoYeah, that's the TED Talk effect, as some people put it You cannot make a TED Talk about something that people already know
- pbourke 6y agoWhat if you presented it with a bunch of single-word slides and had a compelling frame? “What my 10 years among uncontacted tribes in the Amazon taught me about the boiling point of water”
- totalZero 6y agoSounds like a Columbus' Egg kind of situation. Your conclusions may be obvious in retrospect, but they weren't obvious at the time that you chose to pursue them and your supervisors gave you their blessing.
- skosuri 6y ago1. Other than your time, why not preprint it? 2. As an aside, I can't tell you how many times I've tried to work on stuff, it ends up working, and then I find papers and people saying what we did would never work. Sometimes the ignorance is good.
- bnegreve 6y ago> then I find papers and people saying what we did would never work. Seriously? What kind of scientific paper makes such claims?
- skosuri 6y agoTo be fair, it's mostly people that do this because negative results publishing is rare. There have definitely been papers saying this though; plenty of shade thrown at basically every new method in its infancy with papers saying why they won't work (you can find plenty of academic papers dunking on human genome project and shotgun sequencing; next-gen sequencing; talens/crispr's, gene therapy, immunotherapy, ai, etc).
- dguest 6y agoWhat field is this? In the physics papers I've worked on we generally try to state all the assumptions we made when we rule something out, but we do sometimes miss things, and I suppose that in some fields the preparation might me messier.
- alfonsodev 6y agoWould it be possible for you to upload the paper to researchgate.net ?
- JuettnerDistrib 6y ago> We took methods that had been exploited to improve on previous results for over a decade to their logical extreme, and found that this method no longer leads to improvements. This actually sounds like a really good review paper! Review papers serve multiple purposes: getting people up to speed on a subject, and putting your own spin on a subject to guide future investigation.
- korijn 6y agoSame experience for my master's thesis.
- sampo 6y ago> A paper on the subject would almost be considered uninteresting, and a high ranking journal would ignore it (which is why it's considered "unpublishable"). There is a range of journals from high ranking to solid mid-level to lower tiers to somewhat suspicious to downright obviously pay-to-publish. You can always find a level that will publish your article.
- valarauko 6y agoThe problem in such cases usually isn't in finding a willing journal, but constraints on the authors. For example, during my PhD at a leading biological research institute in India, there was an informal ban on sending manuscripts to open access journals - a rule instituted by the Director, who's office had to approve every submission. At some point this ban was extended to conference proceedings, or journals below a certain Impact Factor. These rules might have been overturned by subsequent administrations, I don't know.
- JamesBarney 6y agoWhat's even worse though is this creates enormous pressure to tweak results such that the findings are publishable. I know one person who basically couldn't get their PhD because they couldn't reproduce another experiment and after several years of trying is pretty much certain the original results were faked in order to be publishable.
- dmd 6y agoOne of the most important things my PhD advisor taught me was to design an experiment so that whichever way the result comes out, it tells you something interesting (even if one of those ways might be more surprising, and more interesting).
- ylem 6y agoNature scientific reports? A number of people have mixed feelings about the journal, but I think it does encourage people to publish work that is technically correct, but not exciting. I remember going through the pain of publishing a boring piece of work before that corrected a boring but incorrect study by someone else. It's tedious, but I try to think of it as community service.
- metters 6y agoYou could publish this in a blog (summarized of course) At least other scientists/researches would be warned.
- voxl 6y agoThat's all well and good, but PhD students have zero incentive to do this, and the blog would likely go completely ignored anyway. Not only does a blog post not help you graduate but in order for anyone to care about the results posted on your blog you have to market it!