7 ms·
Half the papers at NIPS would be rejected if the review process were rerun
- deleted 12y ago[deleted]
- deleted 12y ago[deleted]
- deleted 12y ago[deleted]
- deleted 12y ago[deleted]
- dang 12y agoWe didn't see the subtitle. The site guidelines explicitly ask you not to post questions like this in the threads, but rather to email them to hn@ycombinator.com. The minutiae of title editing are not on topic here.
- hyperbovine 12y agoWhat is going on with these graphics?
- rtkwe 12y agoIt's an XKCD styled graph by the looks of it. There are a couple generators out there that take data and make the hand drawn looking graphs. e.g http://xkcdgraphs.com/ http://xkcdgraphs.com/
- deleted 12y ago[deleted]
- jff 12y agoAnd they look fucking awful.
- flopto 12y agoThey're supposed to look hastily-made to reflect the imprecision in the estimates. https://www.chrisstucchio.com/blog/2014/why_xkcd_style_graphs_are_important.html https://www.chrisstucchio.com/blog/2014/why_xkcd_style_graph...
- desdiv 12y agoI love the general idea of comic-style graphs, but this particular implementation of it does indeed look "fucking awful" in my opinion. The graph itself is fine, but the axis is made of these faux wavy line that eerily repeats itself.
- userbinator 12y agoThe graph itself is fine, but the axis is made of these faux wavy line that eerily repeats itself. I see the same things in fonts that are supposed to look "hand-drawn", and CG renders of realistic scenes - it looks "imperfect" but the way the "imperfection" is itself perfect is what stands out. A little randomness goes a long way to avoiding that.
- bsder 12y agoExcept that there really isn't imprecision in the estimates. The numbers are really quite precise and the confidence interval is actually very good. Far better than an XKCD style should be used for. I use XKCD stuff for WAG's, not well-supported data.
- simonster 12y agoAlso, matplotlib, the most commonly-used Python plotting package, can generate xkcd-ized plots with a single additional command (see http://matplotlib.org/xkcd/examples/showcase/xkcd.html http://matplotlib.org/xkcd/examples/showcase/xkcd.html).
- at-fates-hands 12y agoIs this the Neural Information Processing Systems convention you're talking about? I'm sure more than a few people won't have any idea what "NIPS" stands for.
- davmre 12y agoTo be fair, calling it "Neural Information Processing Systems" isn't significantly more informative. The name is just a quirk of history; NIPS in its modern form includes research in all areas of machine learning, not just neural nets.
- _delirium 12y agoIn fact for some years neural networks were very out of fashion there, and it was almost purely a statistical machine learning conference. I tend to just think of it as a machine-learning conference named "NIPS", which stands for something historical (like Perl and Lisp do).
- techaddict009 12y agoSomething similar happened with IEEE[1] - It had accepted approx 120 papers generated by SCIgen[2]. [1] - http://www.nature.com/news/publishers-withdraw-more-than-120-gibberish-papers-1.14763 http://www.nature.com/news/publishers-withdraw-more-than-120... [2] - http://pdos.csail.mit.edu/scigen/ http://pdos.csail.mit.edu/scigen/
- sqrt17 12y agoIEEE may be more prone to precision errors (letting bad papers in) while NIPS may be prone to recall errors (throwing good papers out). With the way reviewing is done (no one can take a week off to read and fully comprehend the four papers they are given) you cannot achieve perfect separation - even if that were possible.
- onan_barbarian 12y agoCalling the process of "accepting a SciGen-generated paper into a allegedly peer-reviewed journal" a "precision error" is a bit on the optimistic side. It implies that someone was making a decision after reading the content of the paper, as opposed to, well, just accepting everything in sight. It doesn't take a "week off" to notice that a paper is gibberish, at the very least.
- userbinator 12y agoUnless the reviewer doesn't actually know anything at all about what he/she claims to. It wouldn't surprise me at all if most of the general public would be unable to distinguish a SciGen-generated paper from a real one.
- GFK_of_xmaspast 12y agoWhat is such a person doing reviewing for the IEEE.
- Teodolfo 12y agoHow is this similar to what the NIPS organizers did? It doesn't seem related at all.
- dhm 12y agoI am surprised the committees were "tasked with a 22.5% acceptance rate". Couldn't more than 77.5% of the submissions have been of poor quality?
- wsxcde 12y agoHighly (almost vanishingly) unlikely at a "top-tier" venue like NIPS.
- dhm 12y agoCan you say more about why you believe this is true?
- robotresearcher 12y agoWhat does "poor quality" mean? There is no absolute standard for quality. "Poor" is something like "less good than usual compared to the recent work in this community". So the top-scoring third-ish of papers sent to the currently-converged-on favourite venue of a community are pretty much by definition not poor. Unless something very weird indeed happens one year. There are usually only very few really excellent papers, though. Most papers are filler in retrospect. Also, conferences need to accept a decent number of papers so that people will show up and cover the costs of the meeting. Venues are usually booked long before the program is fixed.
- dhm 12y agoOk, we've gone from "top tier venue, basically impossible to have a large fraction of poor papers submitted" to "Most papers are filler in retrospect" and "conferences need to accept a decent number of papers so that people will show up and cover the costs of the meeting". I guess if I am deciding whether or not to hire a professor I would be tempted to disregard publications in this conference.
- robotresearcher 12y ago
- danieltillett 12y agoThis is the way "peer review" works. It is basically random. I have always found it comforting whenever I had a paper rejected as I would know it was nothing to do with the quality of my work. I would fix any of the typos found by the reviewers (you always get a spelling nazi as one of reviewers) and send it out again unchanged. I only have had one paper rejected twice and it was accepted unchanged on the third attempt.
- IndianAstronaut 12y agoIsn't this why Mendeley and EndNote exist? Just change up the citation formatting and resubmit to a different journal until one accepts your work.
- danieltillett 12y agoYes :) My favourite peer review story is when I submitted one on my articles to the top journal in my field at the time (Appied and Enviromental Microbiogy). It came back with the usual peer review trivial changes (cite this irrelevant paper of mine,etc) which I did (this nearly always easier than arguing with the reviewers). The editor made a mistake and instead of sending the updated manuscript out to the original reviewers, they sent it out to a new lot of reviewers. What was funny about the whole exercise was the second set of reviewers called the first set of reviewers idiots and told me to change everything back.
- Natsu 12y agoThey always have to find some way to leave their mark.
- danieltillett 12y agoThis is true, but there is always exceptions. The second paper I published I sent off to the journal and after a couple of months I had not heard anything (this was in the physical paper days where you had to mail everything). My supervisor decided to call the editor to ask what was happening. The editor said "oh we published it last month". The whole paper had gone straight through without a single change. This of course was the last time I ever had a paper accepted like this :)
- nl 12y agoFor those who didn't read the story: NIPS decided to run this experiment themselves to identify problems. That's a a really brave thing to do, and deserves serious credit.
- deepsearch 12y agoExactly, the same credit mimicking cognition along vector comparison deserves.
- ehurrell 12y agoAbsolutely, it's hugely important. The core idea that subjective beings will disagree holds obvious weight, but it takes serious commitment to improving the process to offer up this kind of experiment to prove it.
- army 12y agoI think people are reading more into this than there is. Reviewing papers is a highly subjective, high variance process, and very few papers get universally positive reviews. From the point of view of an author, if you get a paper rejected that you know is worthwhile, you just have to make whatever improvements you can and then submit it again.
- andrea_s 12y agoSince NIPS is a very prestigious conference, I'd expect a lot of submitted papers (perhaps a vast majority of them) would fall in a grey area between "clearly unsuitable" and "clearly suitable". I personally think there are too many factors at play in evaluating a series of papers - no objective sorting can really exist. A note for people outside the academic Machine Learning field: NIPS is widely believed to be on a different level from the rest - during my PhD, my advisor used to say that a NIPS paper would be on par with a paper published on a good journal, resume-wise. The difference is especially striking if you've had the chance to attend other conferences (including, alas, IEEE-sponsored events) - that are, with very few exceptions, fairly terrible from a scientific point of view.
- drpgq 12y agoCVPR is pretty good.
- graycat 12y agoFYI: Apparently NIPS abbreviates Neural Information Processing Systems.
- Canada 12y agoSelecting talks really subjective. It's more like choosing what songs to play at a party than an objective sorting process. You have limited speaking slots. You have to guess what the conference attendees will find interesting this year. You are biased by your own particular interests. Also, some of the submitters are your friends/colleagues, and even if they didn't tell you what they were submitting already, which is unlikely since your relationship is based on talking about this stuff, you can tell it's theirs in less than 250 words... However a conference tries to sell the fairness and objectivity of its process, you can't anonymize or double blind these things away.
- sytelus 12y agoFor two independent committees, 6% of papers were acceptable without disagreement, 25% were rejectable while the rest were coin flips. This means when your paper gets accepted or rejected, luck is playing huge part. This is not because judges are actually flipping coin but vast majority of people don't seem strikingly good or bad. So for a repeated trial outcome may not be same. Also, the asymmetry here is striking. Definitely bad papers dominates in number by 4X than definitely good papers. These are really great observations with deep implications. This same patterns might get applied in other aspects of life such as interviewing candidates or selection of mate or buying a shirt. In all these cases, we might have similar distribution at work. I have often wondered why is it so hard to have less mediocrity in world? Why is not every book, t-shirt or smartphone is just great? One obvious reason is that lot of times people create something out of obligation such as demand from job instead of out of urge to create. So subsequent question is that if it was possible that no one has to have any obligation to create, can above distribution turn its head over hill? For example, in that scenario would we have, say, 70% great papers, 5% mediocre and rest coin toss?
- dalke 12y ago"Why is not every book, t-shirt or smartphone is just great?" Different people have different ideas of what "great" means. Not everyone thinks the Harry Potter series are great books, while many do. We see that in movies where a movie does poorly at the box office while the critics. The definition of greatness changes over time, so "It's a Wonderful Life", now considered one of the most critically acclaimed films ever made, had only mediocre revenue when it came out. Greatness is sometimes situational, so "Dan Brown ... is the undisputed king of airplane books — the not-too-heavy, not-too-long potboilers perfect for a long layover." If you don't fly, then perhaps there's no time when Brown's works might appeal. Travel has its own category of "good enough." Visiting Germany once I bought a book from the limited English selection not because it was great, but because it was something to read on the long train ride. A lot of people watch sports, but surely it can't be that all sports games are great, so greatness can't be the only reason for keeping someone's interest. Since it's hard to predict greatness, people will test out ideas to see if there's a response. Sometimes this can lead to feedback and improvements. Sometimes this testing is through writing clubs. Sometimes (as with smartphone apps) this is with the market itself.
- djulius 12y agoThat's a very neat experiment. SIGMOD made an interesting move this year by accepting all papers reaching its standards. However not every paper will be given a presentation slot during the conference.