6 ms·

Assessing accuracy? If you're running the same architecture with the same training regime, why would you expect differing accuracy numbers (other than a bug of

Assessing accuracy? If you're running the same architecture with the same training regime, why would you expect differing accuracy numbers (other than a bug of some sort)? Seems a bit strange to include.

AndrewKemendo 9y ago
Didn't you hear? Azure has an entropy converter that makes deep nets deterministic!
minimaxir 9y ago
From the CNTK vs. TensorFlow page: https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-to-switch-from-tensorflow-to-cntk https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-t... > TensorFlow shared the training script for Inception V3, and offered pre-trained models to download. However, it is difficult to retrain the model and achieve the same accuracy, because that requires additional understanding of details such as data pre-processing and augmentation. The best accuracy that were achieved by a third party (Keras in this case) is about 0.6% worse that what the original paper reported. Researchers in the CNTK team worked hard and were able to train a CNTK Inception V3 model with 5.972% top 5 error, even better than the original paper reported! This suggests that an improvement in accuracy is possible by switching to CNTK, hence why I included an accuracy metric from both frameworks. (and also for sanity checking, as you note)
- sirfz 9y ago
  That's not really related to the network. When you build a NN, you shouldn't expect different accuracy using different frameworks (given that you're using the same hyper parameters and training/evaluating on the same data).
- ntenenz 9y ago
  I mean this in the nicest possible way (so please don't take offense), but I think you're missing what is being said there. The framework itself should have no impact, positive or negative, on model accuracy. That being said, it can be extremely challenging to reproduce results given the stochasticity of random batches and asynchronous updates. Furthermore, precisely specifying the methods of data augmentation can be tedious, thus the protocol is often only partially detailed in published work which further exacerbates the challenges of reproducing results.
  - jacquesm 9y ago
    We could introduce a simple rule to get rid of this kind of problem: if it can't be reproduced with the data supplied then it isn't true so you can't publish.
  - nerdponx 9y ago
    Not to mention specifying the random seed and PRNG algorithm.
- benjismith 9y ago
  No, if you re-read the first sentence there, it says that the different results are attributed to differences in "pre-processing and augmentation". The choice of NN framework is essentially irrelevant.
  - sdenton4 9y ago
    Should be irrelevant... It's still worth testing though, to see that the default implementations are actually doing what it says on the box...
  - dgacmu 9y ago
    Note, though, that the preprocessing and augmentation is (at least in TF) done within the framework itself. I helped debug the pure-TensorFlow version of the Inception input pipeline, and getting it to match the earlier DistBelief version was agonizing -- it really shows all of the differences (and bugs) in the image processing ops. And there can be subtle effects -- differences in which image resizing algorithm you use, for example. But it's worth noting that this code is all released: https://github.com/tensorflow/models/blob/master/inception/inception/image_processing.py#L198 https://github.com/tensorflow/models/blob/master/inception/i... It may be hard to replicate that across all platforms, though -- as an example, the distortions include using four different image resizing algorithms. Some of it was true preprocessing, i.e., cleaning up the imagenet data. I wrote a bit about that here: https://da-data.blogspot.com/2016/02/cleaning-imagenet-dataset-collected.html https://da-data.blogspot.com/2016/02/cleaning-imagenet-datas... (tl;dr - there are some invalid images and bboxes, etc., and some papers chose to deal with the "blacklisted" images differently.)
nl 9y ago
While this sounds like it should be true, in practice it isn't. See for example the TF vs Torch implementations of Pix2Pix https://github.com/affinelayer/pix2pix-tensorflow https://github.com/affinelayer/pix2pix-tensorflow (at the bottom of the page). The reasons for this are many: different order of operations, failure to propagate random seeds, different order of processing files etc. I don't think there is a structured study of this so that would be a great thing to do!

AndrewKemendo 9y ago

Didn't you hear? Azure has an entropy converter that makes deep nets deterministic!

minimaxir 9y ago

From the CNTK vs. TensorFlow page: https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-to-switch-from-tensorflow-to-cntk https://docs.microsoft.com/en-us/cognitive-toolkit/reasons-t... > TensorFlow shared the training script for Inception V3, and offered pre-trained models to download. However, it is difficult to retrain the model and achieve the same accuracy, because that requires additional understanding of details such as data pre-processing and augmentation. The best accuracy that were achieved by a third party (Keras in this case) is about 0.6% worse that what the original paper reported. Researchers in the CNTK team worked hard and were able to train a CNTK Inception V3 model with 5.972% top 5 error, even better than the original paper reported! This suggests that an improvement in accuracy is possible by switching to CNTK, hence why I included an accuracy metric from both frameworks. (and also for sanity checking, as you note)

sirfz 9y ago
That's not really related to the network. When you build a NN, you shouldn't expect different accuracy using different frameworks (given that you're using the same hyper parameters and training/evaluating on the same data).
ntenenz 9y ago
I mean this in the nicest possible way (so please don't take offense), but I think you're missing what is being said there. The framework itself should have no impact, positive or negative, on model accuracy. That being said, it can be extremely challenging to reproduce results given the stochasticity of random batches and asynchronous updates. Furthermore, precisely specifying the methods of data augmentation can be tedious, thus the protocol is often only partially detailed in published work which further exacerbates the challenges of reproducing results.
- jacquesm 9y ago
  We could introduce a simple rule to get rid of this kind of problem: if it can't be reproduced with the data supplied then it isn't true so you can't publish.
- nerdponx 9y ago
  Not to mention specifying the random seed and PRNG algorithm.
benjismith 9y ago
No, if you re-read the first sentence there, it says that the different results are attributed to differences in "pre-processing and augmentation". The choice of NN framework is essentially irrelevant.
- sdenton4 9y ago
  Should be irrelevant... It's still worth testing though, to see that the default implementations are actually doing what it says on the box...
- dgacmu 9y ago
  Note, though, that the preprocessing and augmentation is (at least in TF) done within the framework itself. I helped debug the pure-TensorFlow version of the Inception input pipeline, and getting it to match the earlier DistBelief version was agonizing -- it really shows all of the differences (and bugs) in the image processing ops. And there can be subtle effects -- differences in which image resizing algorithm you use, for example. But it's worth noting that this code is all released: https://github.com/tensorflow/models/blob/master/inception/inception/image_processing.py#L198 https://github.com/tensorflow/models/blob/master/inception/i... It may be hard to replicate that across all platforms, though -- as an example, the distortions include using four different image resizing algorithms. Some of it was true preprocessing, i.e., cleaning up the imagenet data. I wrote a bit about that here: https://da-data.blogspot.com/2016/02/cleaning-imagenet-dataset-collected.html https://da-data.blogspot.com/2016/02/cleaning-imagenet-datas... (tl;dr - there are some invalid images and bboxes, etc., and some papers chose to deal with the "blacklisted" images differently.)

nl 9y ago

While this sounds like it should be true, in practice it isn't. See for example the TF vs Torch implementations of Pix2Pix https://github.com/affinelayer/pix2pix-tensorflow https://github.com/affinelayer/pix2pix-tensorflow (at the bottom of the page). The reasons for this are many: different order of operations, failure to propagate random seeds, different order of processing files etc. I don't think there is a structured study of this so that would be a great thing to do!