5 ms·
Exploring a Language Runtime with Bpftrace
- mrlongroots 1y agoThis is cool! eBPF is incredible: I think of it as flipping the relationship between you and the perf tool. Instead of the tool giving you information, it asks you "what do you want to know?". I don't know if it's google search but I find it hard to discover blogs etc. when I'm hunting for the syntax/language etc, having documented "case studies" is super useful.
- panic 1y agoCool to see Linux catching up to dtrace :)
- indiosmo 1y agobpftrace was instrumental in helping me diagnose a latency spike on a hot path. I call into a vendor library that eventually calls into Solarflare's TCPDirect API to send packets to the network. This call usually stays under 5us and has very little variance, but I was seeing the occasional 30ms or 60ms outlier. I attached probes to track the latency of each call down the stack and it became quite easy to pin down the specific function. Eventually tracking it down to a single assignment to a variable that was mmapped to a file, and getting spikes due to page cache writeback. After figuring it out I found these two articles that go into more detail on the problem. https://rigtorp.se/virtual-memory/ https://rigtorp.se/virtual-memory/ https://tungdam.medium.com/our-lessons-on-linux-writeback-do-dirty-jobs-the-right-way-fa1863a5a3dc https://tungdam.medium.com/our-lessons-on-linux-writeback-do... Here's one of the probes producing a histogram of the latency of the call to the TCPDirect function, which was surprising in how simple it was. sudo bpftrace -p "$PID" -e ' // —— TCPDirect —— uprobe:/lib/x86_64-linux-gnu/libonload_zf.so.1:zft_send_single { @zf_start[tid] = nsecs; } uretprobe:/lib/x86_64-linux-gnu/libonload_zf.so.1:zft_send_single /@zf_start[tid]/ { $lat = nsecs - @zf_start[tid]; @h_zft = hist($lat / 1000); delete(@zf_start[tid]); } '