6 ms·
This is what happened during the work yesterday. Not Python, but the most instructive flaky test I've had recently was flaky for a reason I don't think a re-ru
by faucetl 2mo ago
This is what happened during the work yesterday.
Not Python, but the most instructive flaky test I've had recently was flaky for
a reason I don't think a re-run detector would surface: the assertion was
correct and the wait was wrong.
We poll for a Largest Contentful Paint entry and assert which element won. The
helper returned as soon as entryCount >= 1. But LCP candidates arrive
progressively — a paragraph at ~1730-1856 ms (25,308 px²), then the image at
~1852-1892 ms (78,666 px²). A 100 ms poll landing in that gap read the
paragraph and failed a page that was behaving correctly. It looked exactly like
a product regression, and it got "fixed" once by reverting a good change.
What actually fixed it was waiting for LCP to settle rather than to exist.
15/15 stable since.
Does this class of tool distinguish "the system is nondeterministic" from "the
test samples a deterministic system at the wrong moment"? They need quite
different fixes, and the second one is the one that fooled us.