Thats why you should also track other metrics when AB testing performance changes. During my time at IG, even small perf improvements almost always corresponded

Thats why you should also track other metrics when AB testing performance changes. During my time at IG, even small perf improvements almost always corresponded to stat-sig improvements in engagement metrics. 100ms is a pretty significant performance win, definitely not noise - the other thing is that those wins often scale by the quality of device & connection. So a 100ms win for a high end desktop on fiber internet could be multiple seconds for a low end mobile on 3G.