8 ms·

A test of statistical significance can answer that question easily - I think most optimizers use it.

• bermanoid 14y ago
  With only 13 purchases on each side of the test, a test of statistical significance is only going to pick out very strong effects. Ones that bump conversion rates more than 10%, typically, which is a pretty huge change. For more subtle effects, you need more observations, plain and simple; it doesn't matter how you do the math if you don't have the data to reach the significance levels you care about.
• adrianhoward 14y ago
  As well as the issue with the low numbers only showing a very significant effect at this point, there's also the assumption that the redesign will act immediately. Some products don't have a "search, find and purchase immediately" pattern to sales. Especially when you move out of the B2C market. Some businesses sales can look more like "Visit half a dozen different sites. Go away for a week and think. Visit best sites again. Go away for a few days and come to a decision. Visit final option, browse and purchase". Tracking these multiple visits can be non-trivial/impossible since it may be different people and different browsers visiting the site at the different stages. It also leads to long lead-times for the effects that design changes make.