Activations of individual neurons are hardly relevant, quoting Szegedy et al. from [1] we find that there is no distinction between individual high level units

Activations of individual neurons are hardly relevant, quoting Szegedy et al. from [1] we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks. [1] Intriguing properties of neural networks