5 ms·

This is really intersting. > In particular, model parallelism is efficient when the amount of computation per neuron activity is high (because the neuron activ

This is really intersting. > In particular, model parallelism is efficient when the amount of computation per neuron activity is high (because the neuron activity is the unit being communicated), while data parallelism is efficient when the amount of computation per weight is high (because the weight is the unit being communicated). In this case, for fully connected layers, the amount of computation per neuron is high because it is (fully) connected to and from every other neuron in the pervious and following layer, and therefore we want more GPU's to work on those in parallel, so it isn't a time bottleneck? What does it mean then for to have a high "computation per weight". >• Convolutional layers cumulatively contain about 90-95% of the computation, about 5% of the parameters, and have large representations. • Fully-connected layers contain about 5-10% of the computation, about 95% of the parameters, and have small representations. Wouldn't fc's require more computation since each parameter has more incoming and outgoing values? I don't doubt this is all accurate and I'm just missing some obvious intuition, but for the sake of learning feel free to explain it if you wish!

radarsat1 4y ago
Convolutional kernels are matrix-multiplied over and over again in a sliding window across the feature map. The fully connected layer is matrix-multiplied exactly once. So the number of times each element of the weight matrix is used depends on the size of the layer for FC, but depends on the size of the feature map for Conv layers. Usually the feature map is much larger than the kernel, for the majority of layers, so the kernel weights are re-used a lot. The other possible interpretation is that typically an FC is only applied on a reduced dimensional embedding, after reducing the input data dimensionality iteratively by several convolutional layers. So there is overall more computation done by Conv layers, to get that final embedding, which his then fed to one or two FC layers for taking global context into account -- so usually you'll find more Conv layers than FC layers in a given ANN architecture, thus they make up a larger percentage of the overall computation.
liuliu 4y ago
The paper is very dated. You have to put on the lens that at that time, the networks we are talking about are AlexNet (the author) and VGGNet. For these, FCN layers has much more parameters than CNN layers, and all-reduce for gradients on these FCN layers are not economical (BTW, that's why Transformers do mixed model / data parallelism as well, or do large batches, they have a lot parameters and want to do these all-reduce as late as possible (or avoid at all, with model parallelism)). Otherwise the peer comment answered your question really well!

radarsat1 4y ago

Convolutional kernels are matrix-multiplied over and over again in a sliding window across the feature map. The fully connected layer is matrix-multiplied exactly once. So the number of times each element of the weight matrix is used depends on the size of the layer for FC, but depends on the size of the feature map for Conv layers. Usually the feature map is much larger than the kernel, for the majority of layers, so the kernel weights are re-used a lot. The other possible interpretation is that typically an FC is only applied on a reduced dimensional embedding, after reducing the input data dimensionality iteratively by several convolutional layers. So there is overall more computation done by Conv layers, to get that final embedding, which his then fed to one or two FC layers for taking global context into account -- so usually you'll find more Conv layers than FC layers in a given ANN architecture, thus they make up a larger percentage of the overall computation.

liuliu 4y ago

The paper is very dated. You have to put on the lens that at that time, the networks we are talking about are AlexNet (the author) and VGGNet. For these, FCN layers has much more parameters than CNN layers, and all-reduce for gradients on these FCN layers are not economical (BTW, that's why Transformers do mixed model / data parallelism as well, or do large batches, they have a lot parameters and want to do these all-reduce as late as possible (or avoid at all, with model parallelism)). Otherwise the peer comment answered your question really well!