9 ms·

Author here. This post is a write-up of a performance-debugging rabbit hole I hit while trying to saturate NICs with NVMe reads using io_uring and RDMA. The sh

by MrCroxx 3mo ago

Author here. This post is a write-up of a performance-debugging rabbit hole I hit while trying to saturate NICs with NVMe reads using io_uring and RDMA. The short version: READ_FIXED fixed the obvious per-I/O GUP overhead in a small demo, but the larger deployment still got stuck at roughly half of line rate. After ruling out io-wq backlog, request splitting, fd lookup, and CRC arithmetic, the actual wall turned out to be dTLB misses from scanning 1,028 KiB buffers backed by 4 KiB pages. Moving the read arena to hugepages brought the system close to NIC saturation. The funny part is that an AI agent suggested hugepages early and got the optimization right, but its explanation was wrong. This post is mostly about reconstructing the evidence for why it worked. I’d be very interested in feedback from people who have used AI to debug performance issues in a complex system.

ozgrakkurt 3mo ago
I disagree with the AI part. Because hugepages is one of the things that can be guessed to improve performance when doing something with substantial amount of data. So anyone familiar with the space could have suggested something like that without knowing the details of the problem. Hence it is not useful advice IMO. That aside, the blog post was really cool to read and a instant favorite, wish there were more english posts on the blog. Especially like the hardware limit based expectations, detailed measurements and the writing style.
- MrCroxx 3mo ago
  Thank you for liking this blog. I agree with your point. Actually, I’ve just recently transitioned from building data infrastructure on the cloud to taking on a high-performance computing role that truly handles massive amounts of data. So, although I’d heard about the benefits of hugepages before, I had never actually reproduced these issues in my own environment. This time, even though I initially suspected the problems were related to hugepages and the TLB, I didn’t write this blog from a seasoned perspective. Instead, I wanted to methodically investigate and eliminate all other possible issues I could think of. (Interestingly, my agent attributed the effectiveness of hugepages to the root causes of these bugs, which piqued my curiosity and drove my deeper exploration.) Finally, thank you very much for your appreciation, which means a lot to me. Previously, I was working on open-source projects, but now that I’ve changed jobs, I may not have the same amount of energy to contribute to open-source code as before. However, I think blogging might be a new way for me to contribute. I hope I can keep it up. (My English writing skills are poor, so I wrote in Chinese and used AI to translate it; I hope you don’t mind.)
  - MrCroxx 3mo ago
    But I'm trying my best to practice it. Hope one day I can produce some solid posts in English directly. qwq

by MrCroxx 3mo ago

ozgrakkurt 3mo ago

I disagree with the AI part. Because hugepages is one of the things that can be guessed to improve performance when doing something with substantial amount of data. So anyone familiar with the space could have suggested something like that without knowing the details of the problem. Hence it is not useful advice IMO. That aside, the blog post was really cool to read and a instant favorite, wish there were more english posts on the blog. Especially like the hardware limit based expectations, detailed measurements and the writing style.

MrCroxx 3mo ago
Thank you for liking this blog. I agree with your point. Actually, I’ve just recently transitioned from building data infrastructure on the cloud to taking on a high-performance computing role that truly handles massive amounts of data. So, although I’d heard about the benefits of hugepages before, I had never actually reproduced these issues in my own environment. This time, even though I initially suspected the problems were related to hugepages and the TLB, I didn’t write this blog from a seasoned perspective. Instead, I wanted to methodically investigate and eliminate all other possible issues I could think of. (Interestingly, my agent attributed the effectiveness of hugepages to the root causes of these bugs, which piqued my curiosity and drove my deeper exploration.) Finally, thank you very much for your appreciation, which means a lot to me. Previously, I was working on open-source projects, but now that I’ve changed jobs, I may not have the same amount of energy to contribute to open-source code as before. However, I think blogging might be a new way for me to contribute. I hope I can keep it up. (My English writing skills are poor, so I wrote in Chinese and used AI to translate it; I hope you don’t mind.)
- MrCroxx 3mo ago
  But I'm trying my best to practice it. Hope one day I can produce some solid posts in English directly. qwq