6 ms·

> the problem is that academia rewards producing papers, not stable software libraries. isn't this the good thing about academia? aren't you kids getting pai

by palvaro 12y ago

> the problem is that academia rewards producing papers, not stable software libraries. isn't this the good thing about academia? aren't you kids getting paid the big bucks to write and maintain the software libraries, while we work on novel problems for pennies?

Fede_V 12y ago
I am an academic myself! Aside from that - it's actually a bad thing, poor software quality is incredibly harmful when trying to create reproducible research. A few people are fighting this (Titus Brown, etc) but it's mostly swimming against the tide of bad incentives.
- palvaro 12y ago
  I agree that this is a serious issue in our community, but I am not sure I agree that stable libraries <=> reproducible experiments.
  - nitrogen 12y ago
    If those experiments involve automated data collection or computer models, then stable data collection or modeling libraries would be kind of important for reproducing them.
  - Fede_V 12y ago
    Try to reproduce the analysis published in a paper when all you have is a matlab script with one letter variable names and zero comments :)
    - _delirium 12y ago
      If you're running someone else's code, imo that's not reproduction in the first place, just like re-running an experiment using the original experimenter's preparations and lab apparatus is not what's usually meant by "reproducing" an experiment. Too much undocumented stuff can creep in if you don't independently reproduce, with independent apparatus, preparations of samples, etc. (I don't think having someone's code is useless, and it can be especially useful for elaborating on the original experiment, but I would purposely avoid looking at it if I were aiming for an independent reproduction.)
      - rntz 12y ago
        You are always running someone else's code. It starts the moment you boot up your machine.
        jsprogrammer 12y ago
        Not if you bootstrap from the silicon up.
    - zo1 12y ago
      If you even have access to the source code, detailed algorithm, or even a matlab script. It's either a citation or a plain old equation. Often times, and especially from what I've seen in the computer vision papers, the authors merely state what algorithm they used, and how they combined it with their novel method. And that algorithm is in another paper, by the way, probably by the same author. Definitely not the implementation you're working with, too, if you have it. It's almost as if they need a combined repository. And each paper that presents a novel algorithm, or implementation of an existing one, is a "changeset" or "branch". And the citations to algorithm's used in a paper would be changeset hashes, or branch names. Hey, it's the first thing that popped into mind for me to solve this horrendous problem.
      - Toenex 12y ago
        I certainly agree with this. The computer vision field is awash with papers proposing a 'new' algorithm which is then poorly compared to some select group of existing techniques under criteria chosen by the author. A paper is a very poor substitute for the code itself and really it should be mandatory for code to be submitted with the paper, especially in a field such a computer vision where the entire experimental apparatus could be packed into a zip file. That way any other group could take the code and independently evaluate the technique without reimplementation. Indeed my own experience is that often the maths described in the paper is not necessarily responsible for all the results! As you say this could even become the start of collaborative improvement. Unfortunately my experience is that too many academic groups believe that their source code is the route to untold riches.
    - GFK_of_xmaspast 12y ago
      Better than nothing. (Been there, done that).
  - ehurrell 12y ago
    I agree with you here, but I think 'stable libraries' is perhaps a good target for a few reasons, right now the culture isn't just bad code, it's "There is no advantage or benefit to showing your code". I would say a difference between computer scientists and programmers is that frequently the work isn't just the code, but still, nurturing something like an open-source scientific community would accelerate a lot of learning.
- blackkettle 12y ago
  i have started using Docker for this kind of stuff. you can build an isolated environment for your software and experiments, where you can absolutely guarantee that anyone who wants to can easily replicate your experiments, since they don't need to create the environment themselves - just pull the docker image for conference-paper# and run the scripts. if the experimental data is proprietary, or you want to keep it separate, you can set a mount point for it in the lxc.
  - seanmcdirmid 12y ago
    This reminds me of Phillip Guo's work; maybe this one? http://pgbovine.net/publications/CDE-create-portable-Linux-packages-overview_CiSE-2012.pdf http://pgbovine.net/publications/CDE-create-portable-Linux-p...
    - blackkettle 12y ago
      cool, i had not heard of this. i just started using docker for work and came to the conclusion that it was epically well-suited to this purpose as well. i think docker might be even nicer, since there is no special tools required [but ill definitely take a closer look at this work]
      - seanmcdirmid 12y ago
        Guo's work is a bit old, docker is a very new thing.
    - blackkettle 12y ago
      also it hadn't occurred to me that this might be something interesting to even publish a paper about. so thanks for that too [assuming someone else hasn't already done this too..] edit: well no surprise there i guess! http://www.nextflow.io/blog/2014/nextflow-meets-docker.html http://www.nextflow.io/blog/2014/nextflow-meets-docker.html
  - mcguire 12y ago
    "anyone who wants to can easily replicate your experiments" Replicate the experiments, or just repeat the results?
walshemj 12y ago
The bad thing is academia is badly paid so are the support staff which is why I left a world class RnD organisation to work in commercial software. have you read Cryptnomicon look at how Randy's first job at a University is described.
Chinjut 12y ago
Prestige should accumulate to anyone who does good work, of whatever kind. Stable software libraries can help science as much as producing papers (if nothing else, because of their effect on future production of papers!).
deleted 12y ago
[deleted]

by palvaro 12y ago

Fede_V 12y ago

I am an academic myself! Aside from that - it's actually a bad thing, poor software quality is incredibly harmful when trying to create reproducible research. A few people are fighting this (Titus Brown, etc) but it's mostly swimming against the tide of bad incentives.

palvaro 12y ago
I agree that this is a serious issue in our community, but I am not sure I agree that stable libraries <=> reproducible experiments.
- nitrogen 12y ago
  If those experiments involve automated data collection or computer models, then stable data collection or modeling libraries would be kind of important for reproducing them.
- Fede_V 12y ago
  Try to reproduce the analysis published in a paper when all you have is a matlab script with one letter variable names and zero comments :)
  - _delirium 12y ago
    If you're running someone else's code, imo that's not reproduction in the first place, just like re-running an experiment using the original experimenter's preparations and lab apparatus is not what's usually meant by "reproducing" an experiment. Too much undocumented stuff can creep in if you don't independently reproduce, with independent apparatus, preparations of samples, etc. (I don't think having someone's code is useless, and it can be especially useful for elaborating on the original experiment, but I would purposely avoid looking at it if I were aiming for an independent reproduction.)
    - rntz 12y ago
      You are always running someone else's code. It starts the moment you boot up your machine.
      - jsprogrammer 12y ago
        Not if you bootstrap from the silicon up.
  - zo1 12y ago
    If you even have access to the source code, detailed algorithm, or even a matlab script. It's either a citation or a plain old equation. Often times, and especially from what I've seen in the computer vision papers, the authors merely state what algorithm they used, and how they combined it with their novel method. And that algorithm is in another paper, by the way, probably by the same author. Definitely not the implementation you're working with, too, if you have it. It's almost as if they need a combined repository. And each paper that presents a novel algorithm, or implementation of an existing one, is a "changeset" or "branch". And the citations to algorithm's used in a paper would be changeset hashes, or branch names. Hey, it's the first thing that popped into mind for me to solve this horrendous problem.
    - Toenex 12y ago
      I certainly agree with this. The computer vision field is awash with papers proposing a 'new' algorithm which is then poorly compared to some select group of existing techniques under criteria chosen by the author. A paper is a very poor substitute for the code itself and really it should be mandatory for code to be submitted with the paper, especially in a field such a computer vision where the entire experimental apparatus could be packed into a zip file. That way any other group could take the code and independently evaluate the technique without reimplementation. Indeed my own experience is that often the maths described in the paper is not necessarily responsible for all the results! As you say this could even become the start of collaborative improvement. Unfortunately my experience is that too many academic groups believe that their source code is the route to untold riches.
  - GFK_of_xmaspast 12y ago
    Better than nothing. (Been there, done that).
- ehurrell 12y ago
  I agree with you here, but I think 'stable libraries' is perhaps a good target for a few reasons, right now the culture isn't just bad code, it's "There is no advantage or benefit to showing your code". I would say a difference between computer scientists and programmers is that frequently the work isn't just the code, but still, nurturing something like an open-source scientific community would accelerate a lot of learning.
blackkettle 12y ago
i have started using Docker for this kind of stuff. you can build an isolated environment for your software and experiments, where you can absolutely guarantee that anyone who wants to can easily replicate your experiments, since they don't need to create the environment themselves - just pull the docker image for conference-paper# and run the scripts. if the experimental data is proprietary, or you want to keep it separate, you can set a mount point for it in the lxc.
- seanmcdirmid 12y ago
  This reminds me of Phillip Guo's work; maybe this one? http://pgbovine.net/publications/CDE-create-portable-Linux-packages-overview_CiSE-2012.pdf http://pgbovine.net/publications/CDE-create-portable-Linux-p...
  - blackkettle 12y ago
    cool, i had not heard of this. i just started using docker for work and came to the conclusion that it was epically well-suited to this purpose as well. i think docker might be even nicer, since there is no special tools required [but ill definitely take a closer look at this work]
    - seanmcdirmid 12y ago
      Guo's work is a bit old, docker is a very new thing.
  - blackkettle 12y ago
    also it hadn't occurred to me that this might be something interesting to even publish a paper about. so thanks for that too [assuming someone else hasn't already done this too..] edit: well no surprise there i guess! http://www.nextflow.io/blog/2014/nextflow-meets-docker.html http://www.nextflow.io/blog/2014/nextflow-meets-docker.html
- mcguire 12y ago
  "anyone who wants to can easily replicate your experiments" Replicate the experiments, or just repeat the results?

walshemj 12y ago

The bad thing is academia is badly paid so are the support staff which is why I left a world class RnD organisation to work in commercial software. have you read Cryptnomicon look at how Randy's first job at a University is described.

Chinjut 12y ago

Prestige should accumulate to anyone who does good work, of whatever kind. Stable software libraries can help science as much as producing papers (if nothing else, because of their effect on future production of papers!).

deleted 12y ago

[deleted]