7 ms·

My PhD adviser was a good friend of Chandler's, and a lot of my doctoral work was based on previous work done in his lab. I'm honestly really surprised one of h

My PhD adviser was a good friend of Chandler's, and a lot of my doctoral work was based on previous work done in his lab. I'm honestly really surprised one of his students/post-docs made a mistake of this magnitude -- generating a velocity distribution which does not follow the Boltzmann distribution is not something I'd expect to be a result of code coming from his lab. But we all make mistakes, and this really just lends more weight to the idea that all code used in producing published results should be posted along with the article itself, just like data sets often are (at least from studies with government funding).

ur-whale 8y ago
Refusing to share their code falls under the "we all make mistakes" umbrella you think?
- whateveryou381 8y ago
  Sharing code is not always the answer. Independent thinking implies independence, in code and more abstractly. The real 'mistake' is allowing the drama to escalate to the point that it is toxic. People who are truly interested don't care about who is right and who is wrong. When disagreements like this come up, having common and good test cases are probably the most important (and is indeed the way in which the problem, generation of non-Boltzman behavior, was found). In the end ST2 itself is flawed, and Princeton admitted that the discussion of water was not significantly advanced through this drama. Is it worth it? Understanding the argument and its importance should be the focus.
  - YorkshireSeason 8y ago
    Sharing code is not always the answer Why let the perfect be the enemy of the good? Sharing code is not the full answer, especially in case of Monte-Carlo simulations in physics, because that kind of algorithm is hard to test: What is the testing oracle? What is the specification? But sharing code is part of the answer. Setting up a culture where it is unacceptable to submit a paper without open-sourcing the code and suitable testing (for simple edge ases), and suitable scripts that make reproducing the software simulations easy, is good scientific 'hygiene'. See for example [1, 2] for efforts towards reproducible software submissions in computer science. Reproduciblility is the very essence of the scientific method. [1] http://evaluate.inf.usi.ch/artifacts http://evaluate.inf.usi.ch/artifacts [2] http://www.artifact-eval.org/ http://www.artifact-eval.org/
  - mannykannot 8y ago
    In this case, independent thinking did come up with independent results, but the refusal to share code significantly delayed the recognition of which line of inquiry had been more successful. You might want to be careful about using that "was it worth it?" argument - someone might reasonably ask, was it worth funding either team? If science doesn't show that it does good work, then its funding can be more easily questioned.
- s-shellfish 8y ago
  Sharing code can be just as confusing as it is enlightening.
  - mannykannot 8y ago
    Ordinary text can be confusing. That is not a reason for not publishing, and one of the purposes of a journal's editors is to keep the confused and confusing stuff at bay.
    - s-shellfish 8y ago
      So, black box, closed source - e.g. history repeats itself.
- ApostleMatthew 8y ago
  Oh, I meant making the error in the code fell under the we all make mistakes. Not the refusing to share it.

ur-whale 8y ago

Refusing to share their code falls under the "we all make mistakes" umbrella you think?

whateveryou381 8y ago
Sharing code is not always the answer. Independent thinking implies independence, in code and more abstractly. The real 'mistake' is allowing the drama to escalate to the point that it is toxic. People who are truly interested don't care about who is right and who is wrong. When disagreements like this come up, having common and good test cases are probably the most important (and is indeed the way in which the problem, generation of non-Boltzman behavior, was found). In the end ST2 itself is flawed, and Princeton admitted that the discussion of water was not significantly advanced through this drama. Is it worth it? Understanding the argument and its importance should be the focus.
- YorkshireSeason 8y ago
  Sharing code is not always the answer Why let the perfect be the enemy of the good? Sharing code is not the full answer, especially in case of Monte-Carlo simulations in physics, because that kind of algorithm is hard to test: What is the testing oracle? What is the specification? But sharing code is part of the answer. Setting up a culture where it is unacceptable to submit a paper without open-sourcing the code and suitable testing (for simple edge ases), and suitable scripts that make reproducing the software simulations easy, is good scientific 'hygiene'. See for example [1, 2] for efforts towards reproducible software submissions in computer science. Reproduciblility is the very essence of the scientific method. [1] http://evaluate.inf.usi.ch/artifacts http://evaluate.inf.usi.ch/artifacts [2] http://www.artifact-eval.org/ http://www.artifact-eval.org/
- mannykannot 8y ago
  In this case, independent thinking did come up with independent results, but the refusal to share code significantly delayed the recognition of which line of inquiry had been more successful. You might want to be careful about using that "was it worth it?" argument - someone might reasonably ask, was it worth funding either team? If science doesn't show that it does good work, then its funding can be more easily questioned.
s-shellfish 8y ago
Sharing code can be just as confusing as it is enlightening.
- mannykannot 8y ago
  Ordinary text can be confusing. That is not a reason for not publishing, and one of the purposes of a journal's editors is to keep the confused and confusing stuff at bay.
  - s-shellfish 8y ago
    So, black box, closed source - e.g. history repeats itself.
ApostleMatthew 8y ago
Oh, I meant making the error in the code fell under the we all make mistakes. Not the refusing to share it.