Read through Understanding Software Dynamics by Richard Sites and work through the example problems in the book. He discusses exactly what you're describing (L

Read through Understanding Software Dynamics by Richard Sites and work through the example problems in the book. He discusses exactly what you're describing (L1/2/3 cache hit rates, their performance implications, how compiler optimizations can fool us into thinking we have a good hit rate, etc). Also take a look into Intel VTune and Processor Tracing to understand how performance counters like Instructions per Cycle are calculated.