I've seen lots of alternate approaches to constructing neural nets with varying operations and one of the common themes is that it doesn't really matter that mu

I've seen lots of alternate approaches to constructing neural nets with varying operations and one of the common themes is that it doesn't really matter that much w.r.t. accuracy, so you may as well optimize for performance. This seems to indicate to me that neural nets are some variation on a yet unidentified more elegant mathematical structure that might enable better theoretical understanding about training phenomena. The spin-glass equivalencies I think further suggest more elegant underlying structures. Once we understand that underlying structure we might be able to do really cool things, i.e. identify the nature and size of training data set required for solving a given problem, or train much, much faster.