You're right. That's a bit more reasonable, so back to my 10% change in base rate (1% abs.) but with a 90% power: > power.prop.test(p1=0.1, p2=0.11, power=0.

You're right. That's a bit more reasonable, so back to my 10% change in base rate (1% abs.) but with a 90% power: > power.prop.test(p1=0.1, p2=0.11, power=0.9,sig.level=0.05) Two-sample comparison of proportions power calculation n = 19746.62 p1 = 0.1 p2 = 0.11 sig.level = 0.05 power = 0.9 alternative = two.sided NOTE: n is number in *each* group Requires about 40,000 samples per test. I would strongly recommend anyone serious about doing this look in to MAB testing, as A-B testing is way too expensive for reasonable scale testing (unless you have a strong a priori hypothesis to test).