6 ms·

Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement

51 points by divia 17y ago

rbanffy 17y ago
Hi folks. I am conducting an experiment (this is phase 2) and need about 50 downvotes. Would you please downvote this post?
- deleted 17y ago
  [deleted]
  - rbanffy 17y ago
    Wondering how much karma would I lose if I were flagged... Please, go ahead.
  - rbanffy 17y ago
    And if I were a comment spammer, I would probably embed a link or two in my message ;-)
- minsight 17y ago
  There is a time and a place for experiments. I'm pretty sure it's not here and not now.
  - rbanffy 17y ago
    If an experiment has HN as its subject, this is precisely not the right but the only place it can happen. As for the time, well... I got 40+ upvotes today (I started this morning needing about half a dozen). The plan is to reach 1337 karma points (did it yesterday, but I wasn't monitoring to get a screenshot).
    - minsight 17y ago
      Then that's not an experiment, but a spammy attempt at satisfying your vanity. This is not what HN is for.
      - rbanffy 17y ago
        Actually, I am trying to hack the meatware that operates on HN. That's worthy of hacker news, isn't it? As long as I am kept downvoted, my posts will be a nice grey and way down the list. Very little harm comes from that.
        rbanffy2 17y ago
        If you're going to be a twat, you shouldn't be so googleable. ricardo@dieblinkenlights.com rbanffy@gmail.com ------ http://identi.ca/rbanffy http://identi.ca/rbanffy http://www.linkedin.com/in/ricardobanffy http://www.linkedin.com/in/ricardobanffy http://profiles.yahoo.com/rbanffy http://profiles.yahoo.com/rbanffy http://www.dieblinkenlights.com/ http://www.dieblinkenlights.com/ http://pt-br.facebook.com/rbanffy http://pt-br.facebook.com/rbanffy ------- http://maps.google.com/maps?f=d&source=s_d&saddr=&daddr=-23.553084,-46.650077&hl=en&geocode=&mra=mi&mrsp=0&sz=16&sll=-23.553739,-46.648936&sspn=0.011664,0.015943&ie=UTF8&ll=-23.552343,-46.649837&spn=0.011664,0.015943&z=16 http://maps.google.com/maps?f=d&source=s_d&saddr=&#3...
        sjf 17y ago
        This isn't reddit, no one is impressed that you can google a user name.
videophile 17y ago
There is some "New Math" being invoked here. If the conversion rate goes from 24.4% to 29.6%, it's disingenuous to claim a 21.2% improvement. I stopped at the first misuse of statistics, but by that point, they had taken out informative links from their web site and turned it into a landing page. Frankly, a 5% change in conversion rate is in the noise compared to the effect of targeted advertising (and I have no faith that these sloppy reporters actually did a properly controlled experiment). Edit: Downmodded for asking for real math over marketing math?
- jbm 17y ago
  It's been a while since I've taken a math class, but I'm not sure what the problem is with calling it a 21.2% improvement. The old conversion rate was 24.4%. The new one was 29.6%. (29.6 - 24.4) / 24.4 = .213 = 21.3%. Is it misleading because their number of conversions is so low? (Not trying to be snarky, I have no idea)
  - videophile 17y ago
    The improvement in the conversion rate is (29.6 - 24.4) = 5.2%. Reporting "a percent of a percent" is downright misleading because it is entirely dependent on the starting baseline and also because conversion rates have high natural variance. Imagine an improvement from 2% conversion (for a truly awful site) to 2.6% (for a site just as awful). That "30% improvement" just isn't. If they were intellectually honest and called it a "0.6% improvement," anyone would be able to see that the claimed improvement is well within stddev.
    - brown9-2 17y ago
      But the underlying metric that actually matters, the number of users converted, did actually improve by 21.1%. I think it's just a matter of using confusing labels to refer to the percentages/improvements. In other words, you don't really care about the second derivative (the rate at which the conversion rate) is improving as much as you care about the initial rate.
    - patio11 17y ago
onreact-com 17y ago
"Conclusion: Simplified design improved conversion by an average of 20.45%." is the most important sentence in this post. To be honest you don't have to use Google Website Optimizer to notice that.
- patio11 17y ago
  It is easy to believe the things you want to believe. A/B testing is useful for convincing you of the things you don't want to believe. Example: I'm releasing my Rails A/B testing framework this Sunday, with a case study writeup taken from my site. My site currently allows people to defer signup for the trial by means of a guest login. That sort of option is popular here. I'm told it increases usability and user engagement, right? Without ruining the surprise: that is a testable hypothesis. ;)
  - onreact-com 17y ago
    A/B testing is great, no doubt. The lesson of solely simplifying is just not a great outcome of such a test. It's like driving a car for the first time and then telling everybody that it's faster than walking.
- ramit 17y ago
  Yeah you do. I ran a test on a friend's site, convinced that a short version would outperform the long-copy version. Conversions dropped by about 30%, which would have cost him tens of thousands of dollars per year.

51 points by divia 17y ago

Hi folks. I am conducting an experiment (this is phase 2) and need about 50 downvotes. Would you please downvote this post?

deleted 17y ago
[deleted]
- rbanffy 17y ago
  Wondering how much karma would I lose if I were flagged... Please, go ahead.
- rbanffy 17y ago
  And if I were a comment spammer, I would probably embed a link or two in my message ;-)
minsight 17y ago
There is a time and a place for experiments. I'm pretty sure it's not here and not now.
- rbanffy 17y ago
  If an experiment has HN as its subject, this is precisely not the right but the only place it can happen. As for the time, well... I got 40+ upvotes today (I started this morning needing about half a dozen). The plan is to reach 1337 karma points (did it yesterday, but I wasn't monitoring to get a screenshot).
  - minsight 17y ago
    Then that's not an experiment, but a spammy attempt at satisfying your vanity. This is not what HN is for.
    - rbanffy 17y ago
      Actually, I am trying to hack the meatware that operates on HN. That's worthy of hacker news, isn't it? As long as I am kept downvoted, my posts will be a nice grey and way down the list. Very little harm comes from that.
      - rbanffy2 17y ago
        If you're going to be a twat, you shouldn't be so googleable. ricardo@dieblinkenlights.com rbanffy@gmail.com ------ http://identi.ca/rbanffy http://identi.ca/rbanffy http://www.linkedin.com/in/ricardobanffy http://www.linkedin.com/in/ricardobanffy http://profiles.yahoo.com/rbanffy http://profiles.yahoo.com/rbanffy http://www.dieblinkenlights.com/ http://www.dieblinkenlights.com/ http://pt-br.facebook.com/rbanffy http://pt-br.facebook.com/rbanffy ------- http://maps.google.com/maps?f=d&source=s_d&saddr=&daddr=-23.553084,-46.650077&hl=en&geocode=&mra=mi&mrsp=0&sz=16&sll=-23.553739,-46.648936&sspn=0.011664,0.015943&ie=UTF8&ll=-23.552343,-46.649837&spn=0.011664,0.015943&z=16 http://maps.google.com/maps?f=d&source=s_d&saddr=&#3...
        sjf 17y ago
        This isn't reddit, no one is impressed that you can google a user name.

videophile 17y ago

There is some "New Math" being invoked here. If the conversion rate goes from 24.4% to 29.6%, it's disingenuous to claim a 21.2% improvement. I stopped at the first misuse of statistics, but by that point, they had taken out informative links from their web site and turned it into a landing page. Frankly, a 5% change in conversion rate is in the noise compared to the effect of targeted advertising (and I have no faith that these sloppy reporters actually did a properly controlled experiment). Edit: Downmodded for asking for real math over marketing math?

jbm 17y ago
It's been a while since I've taken a math class, but I'm not sure what the problem is with calling it a 21.2% improvement. The old conversion rate was 24.4%. The new one was 29.6%. (29.6 - 24.4) / 24.4 = .213 = 21.3%. Is it misleading because their number of conversions is so low? (Not trying to be snarky, I have no idea)
- videophile 17y ago
  The improvement in the conversion rate is (29.6 - 24.4) = 5.2%. Reporting "a percent of a percent" is downright misleading because it is entirely dependent on the starting baseline and also because conversion rates have high natural variance. Imagine an improvement from 2% conversion (for a truly awful site) to 2.6% (for a site just as awful). That "30% improvement" just isn't. If they were intellectually honest and called it a "0.6% improvement," anyone would be able to see that the claimed improvement is well within stddev.
  - brown9-2 17y ago
    But the underlying metric that actually matters, the number of users converted, did actually improve by 21.1%. I think it's just a matter of using confusing labels to refer to the percentages/improvements. In other words, you don't really care about the second derivative (the rate at which the conversion rate) is improving as much as you care about the initial rate.
  - patio11 17y ago

onreact-com 17y ago

"Conclusion: Simplified design improved conversion by an average of 20.45%." is the most important sentence in this post. To be honest you don't have to use Google Website Optimizer to notice that.

patio11 17y ago
It is easy to believe the things you want to believe. A/B testing is useful for convincing you of the things you don't want to believe. Example: I'm releasing my Rails A/B testing framework this Sunday, with a case study writeup taken from my site. My site currently allows people to defer signup for the trial by means of a guest login. That sort of option is popular here. I'm told it increases usability and user engagement, right? Without ruining the surprise: that is a testable hypothesis. ;)
- onreact-com 17y ago
  A/B testing is great, no doubt. The lesson of solely simplifying is just not a great outcome of such a test. It's like driving a car for the first time and then telling everybody that it's faster than walking.
ramit 17y ago
Yeah you do. I ran a test on a friend's site, convinced that a short version would outperform the long-copy version. Conversions dropped by about 30%, which would have cost him tens of thousands of dollars per year.