7 ms·
Implementing models directly from papers is typically pretty doable (and is of course more straightforward when the full implementation is open sourced). Often
by calebkaiser 2mo ago
Implementing models directly from papers is typically pretty doable (and is of course more straightforward when the full implementation is open sourced). Often there is some amount of specific knowledge, like particular hyperparameters, that is missing and has to be trial and errored by the community, but generally speaking, getting the core model architecture implemented is a reasonable task for most well documented models.
Reproducing the exact training run, however, is basically impossible without the original dataset and training pipeline (here meaning all of the code + infra involved in actually executing the pre and post training loops). Also, it would be exorbitantly expensive to do if you weren't also a lab trying to train a similar model.
But you can still scale the architecture down and experiment as a solo researcher using the published research. There are probably some open source implementations already on GitHub for any given big open model release.
- marcyb5st 2mo agoThe exact training run is basically impossible anyway. Randomness plays a role. Even if you fix your RNG seed, in a distributed training scenario like this one some weight updates might come at different times and be included in different update steps. Should have minimal impact on the final outcome, but would still be a different model as some of the weights will differ in the end.
- eru 2mo ago> Should have minimal impact on the final outcome, [...] I share this expectation. But this is an interesting empiric question that deserves study; even if just to confirm what 'everyone knows'.
- pcmasterr 2mo agoReproducibility isn’t something that has been considered desirable in many pipelines until recent years. Heck even ffmpeg introduce randomness when stitching together downloaded chunks from youtube. By design.
- eru 2mo agoReproducing a stationary probability distribution that subsequent runs draw from is also a kind of reproduction. And presumably for ffmpeg you can fix the random seed? Tell me more about that design, please.