4 ms·

Are both using a PCIe switch? If one is and the other isn't, it could be about PCIe credit based flow control kicking in.