5 ms·

Yes. We can send papers to whomever we want. At one point, when you published a paper in a journal you’d get a stack of printed copies of your paper in the ma

by ms013 8y ago

Yes. We can send papers to whomever we want. At one point, when you published a paper in a journal you’d get a stack of printed copies of your paper in the mail from the publisher that you were free to share with anyone. I think I have a few of those stacks around from papers I published 20 years or so ago. Even if you technically aren’t allowed to share, that has NEVER stopped someone from sharing in my career. The only time I’ve had trouble with people sharing is if I ask for code to reproduce what they published. That usually is met with silence.

adw 8y ago
This is highly field-dependent. My old field – solid state physics – did reasonably well on reproducibility, not least because of a bunch of committed open-source folks who had been burnt by exactly this (and by licensing their previous code commercially and getting screwed by their publisher, preventing them from reproducing their own work) working on ABINIT (abinit.org), led by Xavier Gonze. There's a culture around sharing pseudopotentials, too, which is good. SIESTA (icmab.es/siesta) is now open too, I note. That's the team I used to work alongside.
ApostleMatthew 8y ago
In my field (computational biophysics), it's pretty common practice to post published code on a git repository. Code used to produce published results absolutely should be accessible.
- yiyus 8y ago
  > Code used to produce published results absolutely should be accessible. Although I agree with this sentiment, I think that if this was a requirement we would not get much more code but much less papers.
  - umanwizard 8y ago
    I'm not in academia so maybe I'm being flippant, but what is even the point of a paper whose results can't be reproduced? Are you really advancing knowledge in any meaningful sense if someone can't repeat what you did?
    - ms013 8y ago
      Exactly. In one recent case, someone published a paper about an interesting graph centrality metric that I was interested in trying out. Unfortunately, their description in the paper was far too vague to be useful - "implemented as a simple extension of Brandes' algorithm". In attempting to reproduce it, that meant I needed to go read up on the algorithm they extended, and then try to figure out how they actually extended it. In the end, I couldn't actually reproduce the work, and never heard back about the code that the authors used in their published work. That severely degrades the utility of the paper. Yes - the paper does contain some knowledge that they shared with the world, but it was difficult to build upon and replicate since they failed to describe what amounts to the experimental apparatus and setup that was used to obtain the results they published. Unfortunately, this is relatively common in CS (at least, the corners of CS where I work).
      - MrEldritch 8y ago
        I know this is a big issue in AI/ML right now. Deepmind's papers are notoriously hard to reproduce, because they will lay out the general terms of the architecture but not specific implementation details - things like filter length, stride, number of layers, number of hidden units, feature selection, and all the little tricks of initialization or normalization or a zillion other subtleties. The trouble being that those "specific implementation details" are typically non-obvious and absolutely crucial to getting the system described to work at all. For instance, as far as I know, nobody's managed to implement a WaveNet that sounds anything like as good as Google's samples. Neural Turing Machines - published three years ago - were so finicky that someone actually figuring out how to implement the damn thing and have it actually work as described was enough to warrant a paper of its own (Implementing Neural Turing Machines, https://arxiv.org/pdf/1807.08518.pdf https://arxiv.org/pdf/1807.08518.pdf). Not to mention how hard it is to iterate on failed replications when you aren't blessed with ten thousand Nvidia Teslas and custom tensor ASICs and have to wait eternities for models to train. At this point, I think most of the community just kind of looks at their papers, sighs in jealousy, and moves on.
    - Gauc2 8y ago
      I think a lot of this depend on the field and context. For example in physics there are a lot of commercially available simulation tools that help with analysis of an experiment. You of course describe the approach and setup of the simulation but you can not publish/reference the source code since you are just a licensee.

by ms013 8y ago

adw 8y ago

This is highly field-dependent. My old field – solid state physics – did reasonably well on reproducibility, not least because of a bunch of committed open-source folks who had been burnt by exactly this (and by licensing their previous code commercially and getting screwed by their publisher, preventing them from reproducing their own work) working on ABINIT (abinit.org), led by Xavier Gonze. There's a culture around sharing pseudopotentials, too, which is good. SIESTA (icmab.es/siesta) is now open too, I note. That's the team I used to work alongside.

ApostleMatthew 8y ago

In my field (computational biophysics), it's pretty common practice to post published code on a git repository. Code used to produce published results absolutely should be accessible.

yiyus 8y ago
> Code used to produce published results absolutely should be accessible. Although I agree with this sentiment, I think that if this was a requirement we would not get much more code but much less papers.
- umanwizard 8y ago
  I'm not in academia so maybe I'm being flippant, but what is even the point of a paper whose results can't be reproduced? Are you really advancing knowledge in any meaningful sense if someone can't repeat what you did?
  - ms013 8y ago
    Exactly. In one recent case, someone published a paper about an interesting graph centrality metric that I was interested in trying out. Unfortunately, their description in the paper was far too vague to be useful - "implemented as a simple extension of Brandes' algorithm". In attempting to reproduce it, that meant I needed to go read up on the algorithm they extended, and then try to figure out how they actually extended it. In the end, I couldn't actually reproduce the work, and never heard back about the code that the authors used in their published work. That severely degrades the utility of the paper. Yes - the paper does contain some knowledge that they shared with the world, but it was difficult to build upon and replicate since they failed to describe what amounts to the experimental apparatus and setup that was used to obtain the results they published. Unfortunately, this is relatively common in CS (at least, the corners of CS where I work).
    - MrEldritch 8y ago
      I know this is a big issue in AI/ML right now. Deepmind's papers are notoriously hard to reproduce, because they will lay out the general terms of the architecture but not specific implementation details - things like filter length, stride, number of layers, number of hidden units, feature selection, and all the little tricks of initialization or normalization or a zillion other subtleties. The trouble being that those "specific implementation details" are typically non-obvious and absolutely crucial to getting the system described to work at all. For instance, as far as I know, nobody's managed to implement a WaveNet that sounds anything like as good as Google's samples. Neural Turing Machines - published three years ago - were so finicky that someone actually figuring out how to implement the damn thing and have it actually work as described was enough to warrant a paper of its own (Implementing Neural Turing Machines, https://arxiv.org/pdf/1807.08518.pdf https://arxiv.org/pdf/1807.08518.pdf). Not to mention how hard it is to iterate on failed replications when you aren't blessed with ten thousand Nvidia Teslas and custom tensor ASICs and have to wait eternities for models to train. At this point, I think most of the community just kind of looks at their papers, sighs in jealousy, and moves on.
  - Gauc2 8y ago
    I think a lot of this depend on the field and context. For example in physics there are a lot of commercially available simulation tools that help with analysis of an experiment. You of course describe the approach and setup of the simulation but you can not publish/reference the source code since you are just a licensee.