6 ms·

Ask HN: What is your one tip about A/B testing you would share?

4 points by raycloyd 12y ago

Wondering what everyone would tell someone new to a/b testing for website and conversion rate optimization. Could be anything from expectations to approach, etc. Thanks!

onion2k 12y ago
Learn what "statistical power" (http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power) means, and then understand that the fact n% of users prefer one option doesn't necessarily make it a good idea. Statistical significance is far more important than the actual result. This is a really good paper about it: http://www.qubitproducts.com/sites/default/files/pdf/most_winning_ab_test_results_are_illusory.pdf http://www.qubitproducts.com/sites/default/files/pdf/most_wi...
Someone1234 12y ago
That most A/B tests don't prove what they aim to prove. People just add a new version of the site, then wait until 100 people have seen it, measure a small improvement and migrate to the new version. However if you then run the test a second time, but make the "old" site the alternative, you might find it too sees a small measure of improvement and thus you could bounce back and forth between A and B until the end of time. I am not a statistician so I won't try to give you advice on what a statistically significant result is. However many /many/ articles have been written on that topic, and many products have been designed which you can slot your data into to see if the 'B' option is legitimately better.
mtmail 12y ago
If you radically change a feature then regular users will first start playing with it. Simply because it's new. In that case your test needs be longer or exclude regular users.
domrdy 12y ago
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky http://nginx.org/en/docs/http/ngx_http_upstream_module.html#... - to make clients 'stick' to upstreams, if you're using nginx.
siddharthdeswal 12y ago
Don't A/B test your credibility. Not everything should be (or can be) subjected to hypothesis testing.
dairgram 12y ago
Before you get too far, try an A/A test. By this, I mean let the A and B choices be identical. You would certainly expect the outcomes to be equal. Right? I have seen statistically significant differences in outcomes in A/A testing. A/B testing has value but being sure to A/A test may temper your expectations and/or point at problems in your setup before you get too far.
seekingcharlie 12y ago
Ensure that your testing things that are actually going to impact your conversion rate or goals. You don't have to test everything - just the things that matter.
catman01 12y ago
Kittens will always improve conversion rates.
hkiely 12y ago
Choose and test one independent variable at a time. Then, check your results for statistical significance.

4 points by raycloyd 12y ago

Wondering what everyone would tell someone new to a/b testing for website and conversion rate optimization. Could be anything from expectations to approach, etc. Thanks!

onion2k 12y ago

Learn what "statistical power" (http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power) means, and then understand that the fact n% of users prefer one option doesn't necessarily make it a good idea. Statistical significance is far more important than the actual result. This is a really good paper about it: http://www.qubitproducts.com/sites/default/files/pdf/most_winning_ab_test_results_are_illusory.pdf http://www.qubitproducts.com/sites/default/files/pdf/most_wi...

Someone1234 12y ago

That most A/B tests don't prove what they aim to prove. People just add a new version of the site, then wait until 100 people have seen it, measure a small improvement and migrate to the new version. However if you then run the test a second time, but make the "old" site the alternative, you might find it too sees a small measure of improvement and thus you could bounce back and forth between A and B until the end of time. I am not a statistician so I won't try to give you advice on what a statistically significant result is. However many /many/ articles have been written on that topic, and many products have been designed which you can slot your data into to see if the 'B' option is legitimately better.

mtmail 12y ago

If you radically change a feature then regular users will first start playing with it. Simply because it's new. In that case your test needs be longer or exclude regular users.

domrdy 12y ago

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky http://nginx.org/en/docs/http/ngx_http_upstream_module.html#... - to make clients 'stick' to upstreams, if you're using nginx.

siddharthdeswal 12y ago

Don't A/B test your credibility. Not everything should be (or can be) subjected to hypothesis testing.

dairgram 12y ago

Before you get too far, try an A/A test. By this, I mean let the A and B choices be identical. You would certainly expect the outcomes to be equal. Right? I have seen statistically significant differences in outcomes in A/A testing. A/B testing has value but being sure to A/A test may temper your expectations and/or point at problems in your setup before you get too far.

seekingcharlie 12y ago

Ensure that your testing things that are actually going to impact your conversion rate or goals. You don't have to test everything - just the things that matter.

catman01 12y ago

Kittens will always improve conversion rates.

hkiely 12y ago

Choose and test one independent variable at a time. Then, check your results for statistical significance.