The SVD connects the four fundamental subspaces of a linear system. I was fortunate to receive this great insight from one of Strang's lectures. I highly recomm

The SVD connects the four fundamental subspaces of a linear system. I was fortunate to receive this great insight from one of Strang's lectures. I highly recommend you watch one of his videos online on this particular topic. This post doesn't do full justice to the beauty of the SVD. Intuitively, you are trying to compute a transformation that diagonalizes the covariance matrix of the data. Computing the covariance has two problems: 1) this is a O(n^2) operation and 2) can lead to big numerical errors for really small values in the matrix. By creative use of elementary matrix operations, the SVD gives you the transformation on the original matrix. If you are interested in just the first few singular vectors, certain math libraries also support an economical mode that does just this.