5 ms·

+1 on implementing papers, that's one of the best you can do to improve your skills (anywhere in science or engineering actually). A warning: I remember trying

+1 on implementing papers, that's one of the best you can do to improve your skills (anywhere in science or engineering actually). A warning: I remember trying to do this back in my uni / grad days and more often than not there is key information or things (perhaps even by accident) left out of the implementation descriptions. I was more in mechanical engineering so perhaps this is less common in AI oriented papers, but I still think it's a valid thing to look out for.

usmannk 4y ago
> I was more in mechanical engineering so perhaps this is less common in AI oriented papers No, you got it right. This is EXTREMELY prevalent in modern AI/ML papers, to everyone's detriment. In the majority of interesting cases, reproduction is only possible with the original code.
dotnet00 4y ago
I think it's actually often worse in AI papers. Fortunately at least some bigger journals/conferences encourage or require releasing source code, which makes it easier to track down subtle details that the authors didn't clearly mention in the paper. On top of that due to its dependence on data and the ability to 'fudge' statistics, a lot of AI papers aren't really that replicable even if there aren't any implementation subtleties. For example, I've run into papers on image generation which describe some trick to improve quality, but focus entirely on standardized scores without providing any visual comparisons (and thus as feared turning out to not have as much of a visual improvement as the scores would suggest on other datasets). While in a lot of sciences or engineering many things can be attributed to being standard practice for experts in the field, AI moves too fast to have such standards and tends to be a bit too arbitrary for such standards to mean much.
jmcgough 4y ago
This is an issue in biomedical research as well. Sometimes I've reached out to researchers who've done similar studies and ask them missing details in their methods.

usmannk 4y ago

> I was more in mechanical engineering so perhaps this is less common in AI oriented papers No, you got it right. This is EXTREMELY prevalent in modern AI/ML papers, to everyone's detriment. In the majority of interesting cases, reproduction is only possible with the original code.

dotnet00 4y ago

I think it's actually often worse in AI papers. Fortunately at least some bigger journals/conferences encourage or require releasing source code, which makes it easier to track down subtle details that the authors didn't clearly mention in the paper. On top of that due to its dependence on data and the ability to 'fudge' statistics, a lot of AI papers aren't really that replicable even if there aren't any implementation subtleties. For example, I've run into papers on image generation which describe some trick to improve quality, but focus entirely on standardized scores without providing any visual comparisons (and thus as feared turning out to not have as much of a visual improvement as the scores would suggest on other datasets). While in a lot of sciences or engineering many things can be attributed to being standard practice for experts in the field, AI moves too fast to have such standards and tends to be a bit too arbitrary for such standards to mean much.

jmcgough 4y ago

This is an issue in biomedical research as well. Sometimes I've reached out to researchers who've done similar studies and ask them missing details in their methods.