Calculating the number of tests needed to reach statistical certainty is not straightforward at all. The equation for simple A/B tests relies on the change of

Calculating the number of tests needed to reach statistical certainty is not straightforward at all. The equation for simple A/B tests relies on the change of conversion rate from a control. More info here: https://developer.amazon.com/sdk/ab-testing/reference/ab-math.html https://developer.amazon.com/sdk/ab-testing/reference/ab-mat... That's for simple A/B tests - this is not simple A/B tests, instead it is setting up a search space of possible variations of a page and comparing many to many to find which is best. Splitting the data based on visitor referral, country, and more make the sample size needed even larger. TLDR: This solution needs a decent amount of traffic to have reasonable results. The exact numbers are not definite. Source: I co-created xander.io which is basically this concept but more open source.