7 ms·
One of the dangers of A/B testing that the author didn't discuss is actually measuring statistical validity. Simply split testing with a low sample size or low
by mark_story 14y ago
One of the dangers of A/B testing that the author didn't discuss is actually measuring statistical validity. Simply split testing with a low sample size or low level of difference between versions could just be random chance. I find it is always important to figure out the statistical significance of your results to ensure its not just the roll of the dice.
- itsbanderson 14y agoThis is a good point, and something we definitely consider. Our A/B testing tool, ABBA, performs the necessary statistics and is mentioned in the "Test everything" section of the post.
- thejosh 14y agoABBA is a fantastic name for a A/B testing tool. Props to the creators for that name.
- itsbanderson 14y agoABBA: http://www.thumbtack.com/labs/abba http://www.thumbtack.com/labs/abba
- jiggy2011 14y agoWhat is a good way to determine when your sample size is big enough to to warrant A/B testing?
- harlanlewis 14y agohttps://www.google.com/webhp?ie=UTF-8#hl=en&output=search&sclient=psy-ab&q=sample%20size%20statistically%20significant https://www.google.com/webhp?ie=UTF-8#hl=en&output=searc...
- itsbanderson 14y agoHere's a sample size calculator tool: http://www.cardinalpath.com/tools/sample-size-calculator-tool/ http://www.cardinalpath.com/tools/sample-size-calculator-too... And here's a post that addresses this issue: http://37signals.com/svn/posts/3004-ab-testing-tech-note-determining-sample-size http://37signals.com/svn/posts/3004-ab-testing-tech-note-det...
- adrianhoward 14y agoOne of the dangers of A/B testing that the author didn't discuss is actually measuring statistical validity Isn't that actually one of the problems of thinking you're doing A/B testing when you're actually not ;-) A proper analysis is an intrinsic part of what A/B testing is. It's like doing TDD without the refactoring step. TDD without refactoring is, well, not TDD.