8 ms·

>In the end I decided the only real way to tell if what we were doing was working, was to turn it all off, for several months to see if there was a measurable d

>In the end I decided the only real way to tell if what we were doing was working, was to turn it all off, for several months to see if there was a measurable difference. What? If this person genuinely thinks that there is no way to measure an ad's performance while it is running, they are either too ignorant to be in business, or they hired someone who lied to them.

gk1 4y ago
Don't pretend attribution is simple or solved. If that were true there wouldn't be dozens of attribution products, who are still far from perfect. The Dunning Kruger effect is strong in marketing. The more you learn about marketing the more reason you have to be unsure.
yojo 4y ago
If you’re full e-commerce the only metric that really matters is conversions. How do you know your ads aren’t just taking credit for customers who were going to buy anyway? You can get great CTR and conversion rate running ads against your brand name. But if your SEO is sound you’re pretty much just canibalizing your organic traffic. I’m not aware of a way to determine whether an ad just claimed credit for a customer it didn’t earn. I’m interested to learn though.
- boplicity 4y ago
  > How do you know your ads aren’t just taking credit for customers who were going to buy anyway? If you can't answer that question, then you're not running a proper advertising campaign. You're just throwing money around blindly. Plenty of people do have good answers to that question, though. Unfortunately, plenty of people don't.
  - Strom 4y ago
    > If you can't answer that question, then you're not running a proper advertising campaign. What is this proper advertising campaign? Does it include turning off ads for terms that rank organically high anyway?
    - boplicity 4y ago
      > What is this proper advertising campaign Well, for our business, proper advertising campaigns have opened new channels for growth, allowing us to reach new people, and build our email list significantly, with a predictable ROI. It's pretty easy to tell if an ad is working when you turn it on, and your email list growth rate instantly doubles -- and all the traffic is coming from a new traffic source. (Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point.)
      - Strom 4y ago
        > It's pretty easy to tell if an ad is working when you turn it on, and your email list growth rate instantly doubles -- and all the traffic is coming from a new traffic source. Maybe for a real tight definition of instantly, but generally this is not true. Imagine the following scenario: 1. You run an ad for ComapnyName. 2. An enthusiastic customer promotes your business in a local bar. 3. The bar attendees search for CompanyName and click on the first result. Because you ran an ad for CompanyName, the first result is going to be your ad. You will see very nice ROI on that ad. What you won't see in any stats, is that these people would have probably found the organic link anyway, because they were already motivated and searching for CompanyName in particular. > Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point That wouldn't help either in this imaginary scenario, because these bar attendees are a spike in traffic. Plus if the ad has been running for a longer period, then you won't have accurate organic search traffic stats anymore either, because it's already cannibalized.
        boplicity 4y ago
        > Imagine the following scenario "Imagination" is the core of the problem here. Plenty of people are imagining various scenarios, whereas people with successful advertising campaigns are just raking in the money -- no need for imagination. I've turned off plenty of ads when they stopped working. I didn't need to turn them off to discover they stopped working. Sure, you can imagine a dozen scenarios where an ad campaign doesn't work. Fortunately, I've managed to learn how to focus on the reality of the situation -- and have reaped the rewards in the process. > you won't have accurate organic search traffic stats anymore either, because it's already cannibalized. That' not true for us -- as I said before, a successful campaign for us brings in new sources of traffic from new marketing channels. One of the early lessons I learned was to not cannibalize what's already working. For example, we grew our Facebook Page (and email list) from scratch, when we had no Facebook traffic, by using Facebook advertising. Operating under different brands -- even just for testing purposes, is a one another of the way we deal with this. Of course you can continue to imagine scenarios where we might be making mistakes. I do that as well -- it's called planning. Though, none of that really matters until money is spent (and made or lost).
- tetsusaiga 4y ago
  I agree that only conversions matter, but you should be able to track whether a customer originated from an ad click vs. SEO, whether they clicked one ad and had 4 sessions since, etc. While it's not a perfect system, you should know with a reasonable degree of accuracy how many dollars you get back for every dollar you put in to ad spend. As far as the SEO traffic issue, this can also be accounted for once you have scaled your ads beyond the levels of what SEO could yield. At low budgets this can be more of an issue, at high budgets much less so.
  - Strom 4y ago
    > you should be able to track whether a customer originated from an ad click vs. SEO That's easy to track, but answers the wrong question. It answers whether the person clicked on the ad or the organic link. What we actually want to know is whether this person would convert even without seeing an ad. The only way to answer whether a person would still convert without seeing an ad is to make sure that they don't see an ad.
    - tetsusaiga 4y ago
      My major failure in this thread was not specifying from that start that I'm not so much talking about paid search ads (google ads), as much as I mean FB, YouTube, etc. That's the arena where you've really gotta question if you're competing with yourself. On the flip side of that, most of HN seems to think purely in terms of paid search ads, when there are many other types of online advertising that exist, and that don't have this issue baked into them. Such is the nature of internet dialogue, I guess lol.
      - Strom 4y ago
        That's a good point. It's a lot harder to cannibalize organic traffic with non-search ads.
      - m3047 4y ago
        Are you conflating a call to action on a channel you already control with advertising? Because in that case you might try f(A), f(A,B) and also f(B): could be a synergistic effect, or they could just click on anything.
- jefftk 4y ago
  > How do you know your ads aren’t just taking credit for customers who were going to buy anyway? The word for this in the business is "incrementality", and there are several ways of measuring it. The simplest conceptually, is that you run two ads to two random groups of users: one for the product and one for something irrelevant like a charity. Then you compare conversions between the two groups. (There are fancier ways to do it that don't require you to spend half your budget on an irrelevant ad, but that's the basic idea.)
  - whimsicalism 4y ago
    I work in ads. You appear to be one of the few people on this thread who actually knows what they're talking about. +1 to everything you've said.
    - deleted 4y ago
      [deleted]
  - yojo 4y ago
    Thank you. I wasn’t aware of this, it makes perfect sense.
  - yobbo 4y ago
    > is that you show two ads: one for the product and one for something irrelevant like a charity. This is would be another experiment that does not test what you need to in order to demonstrate some sort of causality.
    - jefftk 4y ago
      Why doesn't that demonstrate causality? It is a randomized controlled trial.
      - yobbo 4y ago
        The control would be to not show ads. This was tried in the article, with time as the variable to partition the control groups.
- dangerface 4y ago
  > How do you know your ads aren’t just taking credit for customers who were going to buy anyway? If you are doing advertising right you build a funnel from awareness through to conversion and track every part of it so you then a/b test your adverts and channels. Getting advert channel fit is the key to success and can only be done by testing and measuring your adverts. The fact that op couldn't measure the effectiveness of their adverts shows they weren't doing this and their bad outcome should be 100% expected. If you are just buying adverts in the hope you get more conversions then you are putting those adverts out to die. Burn your money you will probably get more eyes for doing that than untested adverts.
  - whimsicalism 4y ago
    This is now impossible to do on iOS devices.
    - dangerface 4y ago
      How is it impossible on iOS?
      - whimsicalism 4y ago
        Click attribution is basically not possible due to new Apple policy on tracking, unless you opt-in.
bschne 4y ago
Well that works in some ways, but it can also be very hard to disentangle the actual causal impact of an ad campaign! Absent running an experiment, you don't really know if people would or would not have bought through some other channel without your campaign. The most striking example is probably the experiment Tadelis convinced eBay to run on their brand-name ads, though the subtlety applies to less obvious cases of questionable ad spend too — https://faculty.haas.berkeley.edu/stadelis/Tadelis.pdf https://faculty.haas.berkeley.edu/stadelis/Tadelis.pdf
- tetsusaiga 4y ago
  I'll just reference my other replies to the first part here, but thank you for that link too. I'll have to give it a read when I have more time later today
civilized 4y ago
One problem is the ad industry is quite good at finding ways to take credit for conversions that would have happened regardless, and analytics are all systematically biased in a way that conveniently maximizes the ability to do so. The idea to simply turn off ads and see the effect is born of a healthy distrust of ad analytics industry bluster.
- josefresco 4y ago
  > One problem is the ad industry is quite good at finding ways to take credit for conversions that would have happened regardless This is a great counter-point when someone mentions "use conversion tracking!" Conversion tracking is great, but not if they store a cookie for 7/15/30 days and "award" the conversion to the ad, when the customer took a different and varying path to purchase. Sure the ad contributed "some" to the conversion, but not 100%.
  - civilized 4y ago
    I strongly suspect they sometimes show the ad and credit it for the conversion after the user shows initial interest in some specific product, i.e. when the user was already highly likely to buy before seeing the ad. This would explain why you're often flooded with ads for something after you search for it, go to its website, or purchase something there. (We've all had that experience of buying something and then being tailed by ads for that thing for weeks afterwards, even though no sane person would think we were going to buy that thing again in such a short time frame.) A story to illustrate: there was once a pizza store that had two guys go out into the city to distribute promotional coupons. The coupons had codes on them so the business could attribute sales to each coupon distributor. John went out into the city and tried his best to drum up new business. Chad stood next to the door of the pizza place and handed a coupon to anyone who was walking in. After a month, 98% of the coupons used were from Chad. Chad got a big bonus and John was let go.
deleted 4y ago
[deleted]
yobbo 4y ago
> this person genuinely thinks that there is no way to measure an ad's performance while This is the case in many situations. The time between initial visit to site to conversion can be weeks, months, or years. Meanwhile, customers will visit site from various platforms. There is no way of attributing value to a single click, it needs to be approximated as an aggregate.
LightHugger 4y ago
Companies that sell ads will do everything they can to convince you there is a measurable difference, but often while the metrics presented to the client are real, they do not always translate to real world impact. There is no easy way to determine whether this is the case without just comparing when the ads are on and off and somehow dealing with the confounding variables, which isn't easy.
m3047 4y ago
_How would you measure this:_ Does this ad improve conversion compared to not having the ad at all?
mbesto 4y ago
> If this person genuinely thinks that there is no way to measure an ad's performance while it is running, they are either too ignorant to be in business, or they hired someone who lied to them. Doing a blackout month is a legitimate technique. I've deeply studied ad performance metrics and they are NOT as conclusive as people might think. If they were, you could just keep increasing the amount of $ spent and the conversions would go up. Ok, not exactly, there is usually a diminishing returns aspect, but you get the point. A great example of why this requires more analysis is branded vs non-branded search terms on AdWords. Let's say you spend $10k/month on the brand "Mattel" (aka branded) and $10k/month on "toys" (aka non-branded). Mattel likely gets you like a 20x ROAS because people are probably searching for a specific Mattel toy. It's also potentially likely that if you completely turned off branded search you would net the same results. Ad attribution is NOTORIOUSLY difficult: https://www.optimizesmart.com/what-is-attribution-problem-in-online-marketing/ https://www.optimizesmart.com/what-is-attribution-problem-in... so to pretend like "this guy is an idiot, he doesn't know what he's doing" is disingenuous.

gk1 4y ago

Don't pretend attribution is simple or solved. If that were true there wouldn't be dozens of attribution products, who are still far from perfect. The Dunning Kruger effect is strong in marketing. The more you learn about marketing the more reason you have to be unsure.

yojo 4y ago

If you’re full e-commerce the only metric that really matters is conversions. How do you know your ads aren’t just taking credit for customers who were going to buy anyway? You can get great CTR and conversion rate running ads against your brand name. But if your SEO is sound you’re pretty much just canibalizing your organic traffic. I’m not aware of a way to determine whether an ad just claimed credit for a customer it didn’t earn. I’m interested to learn though.

boplicity 4y ago
> How do you know your ads aren’t just taking credit for customers who were going to buy anyway? If you can't answer that question, then you're not running a proper advertising campaign. You're just throwing money around blindly. Plenty of people do have good answers to that question, though. Unfortunately, plenty of people don't.
- Strom 4y ago
  > If you can't answer that question, then you're not running a proper advertising campaign. What is this proper advertising campaign? Does it include turning off ads for terms that rank organically high anyway?
  - boplicity 4y ago
    > What is this proper advertising campaign Well, for our business, proper advertising campaigns have opened new channels for growth, allowing us to reach new people, and build our email list significantly, with a predictable ROI. It's pretty easy to tell if an ad is working when you turn it on, and your email list growth rate instantly doubles -- and all the traffic is coming from a new traffic source. (Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point.)
    - Strom 4y ago
      > It's pretty easy to tell if an ad is working when you turn it on, and your email list growth rate instantly doubles -- and all the traffic is coming from a new traffic source. Maybe for a real tight definition of instantly, but generally this is not true. Imagine the following scenario: 1. You run an ad for ComapnyName. 2. An enthusiastic customer promotes your business in a local bar. 3. The bar attendees search for CompanyName and click on the first result. Because you ran an ad for CompanyName, the first result is going to be your ad. You will see very nice ROI on that ad. What you won't see in any stats, is that these people would have probably found the organic link anyway, because they were already motivated and searching for CompanyName in particular. > Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point That wouldn't help either in this imaginary scenario, because these bar attendees are a spike in traffic. Plus if the ad has been running for a longer period, then you won't have accurate organic search traffic stats anymore either, because it's already cannibalized.
      - boplicity 4y ago
        > Imagine the following scenario "Imagination" is the core of the problem here. Plenty of people are imagining various scenarios, whereas people with successful advertising campaigns are just raking in the money -- no need for imagination. I've turned off plenty of ads when they stopped working. I didn't need to turn them off to discover they stopped working. Sure, you can imagine a dozen scenarios where an ad campaign doesn't work. Fortunately, I've managed to learn how to focus on the reality of the situation -- and have reaped the rewards in the process. > you won't have accurate organic search traffic stats anymore either, because it's already cannibalized. That' not true for us -- as I said before, a successful campaign for us brings in new sources of traffic from new marketing channels. One of the early lessons I learned was to not cannibalize what's already working. For example, we grew our Facebook Page (and email list) from scratch, when we had no Facebook traffic, by using Facebook advertising. Operating under different brands -- even just for testing purposes, is a one another of the way we deal with this. Of course you can continue to imagine scenarios where we might be making mistakes. I do that as well -- it's called planning. Though, none of that really matters until money is spent (and made or lost).
tetsusaiga 4y ago
I agree that only conversions matter, but you should be able to track whether a customer originated from an ad click vs. SEO, whether they clicked one ad and had 4 sessions since, etc. While it's not a perfect system, you should know with a reasonable degree of accuracy how many dollars you get back for every dollar you put in to ad spend. As far as the SEO traffic issue, this can also be accounted for once you have scaled your ads beyond the levels of what SEO could yield. At low budgets this can be more of an issue, at high budgets much less so.
- Strom 4y ago
  > you should be able to track whether a customer originated from an ad click vs. SEO That's easy to track, but answers the wrong question. It answers whether the person clicked on the ad or the organic link. What we actually want to know is whether this person would convert even without seeing an ad. The only way to answer whether a person would still convert without seeing an ad is to make sure that they don't see an ad.
  - tetsusaiga 4y ago
    My major failure in this thread was not specifying from that start that I'm not so much talking about paid search ads (google ads), as much as I mean FB, YouTube, etc. That's the arena where you've really gotta question if you're competing with yourself. On the flip side of that, most of HN seems to think purely in terms of paid search ads, when there are many other types of online advertising that exist, and that don't have this issue baked into them. Such is the nature of internet dialogue, I guess lol.
    - Strom 4y ago
      That's a good point. It's a lot harder to cannibalize organic traffic with non-search ads.
    - m3047 4y ago
      Are you conflating a call to action on a channel you already control with advertising? Because in that case you might try f(A), f(A,B) and also f(B): could be a synergistic effect, or they could just click on anything.
jefftk 4y ago
> How do you know your ads aren’t just taking credit for customers who were going to buy anyway? The word for this in the business is "incrementality", and there are several ways of measuring it. The simplest conceptually, is that you run two ads to two random groups of users: one for the product and one for something irrelevant like a charity. Then you compare conversions between the two groups. (There are fancier ways to do it that don't require you to spend half your budget on an irrelevant ad, but that's the basic idea.)
- whimsicalism 4y ago
  I work in ads. You appear to be one of the few people on this thread who actually knows what they're talking about. +1 to everything you've said.
  - deleted 4y ago
    [deleted]
- yojo 4y ago
  Thank you. I wasn’t aware of this, it makes perfect sense.
- yobbo 4y ago
  > is that you show two ads: one for the product and one for something irrelevant like a charity. This is would be another experiment that does not test what you need to in order to demonstrate some sort of causality.
  - jefftk 4y ago
    Why doesn't that demonstrate causality? It is a randomized controlled trial.
    - yobbo 4y ago
      The control would be to not show ads. This was tried in the article, with time as the variable to partition the control groups.
dangerface 4y ago
> How do you know your ads aren’t just taking credit for customers who were going to buy anyway? If you are doing advertising right you build a funnel from awareness through to conversion and track every part of it so you then a/b test your adverts and channels. Getting advert channel fit is the key to success and can only be done by testing and measuring your adverts. The fact that op couldn't measure the effectiveness of their adverts shows they weren't doing this and their bad outcome should be 100% expected. If you are just buying adverts in the hope you get more conversions then you are putting those adverts out to die. Burn your money you will probably get more eyes for doing that than untested adverts.
- whimsicalism 4y ago
  This is now impossible to do on iOS devices.
  - dangerface 4y ago
    How is it impossible on iOS?
    - whimsicalism 4y ago
      Click attribution is basically not possible due to new Apple policy on tracking, unless you opt-in.

bschne 4y ago

Well that works in some ways, but it can also be very hard to disentangle the actual causal impact of an ad campaign! Absent running an experiment, you don't really know if people would or would not have bought through some other channel without your campaign. The most striking example is probably the experiment Tadelis convinced eBay to run on their brand-name ads, though the subtlety applies to less obvious cases of questionable ad spend too — https://faculty.haas.berkeley.edu/stadelis/Tadelis.pdf https://faculty.haas.berkeley.edu/stadelis/Tadelis.pdf

tetsusaiga 4y ago
I'll just reference my other replies to the first part here, but thank you for that link too. I'll have to give it a read when I have more time later today

civilized 4y ago

One problem is the ad industry is quite good at finding ways to take credit for conversions that would have happened regardless, and analytics are all systematically biased in a way that conveniently maximizes the ability to do so. The idea to simply turn off ads and see the effect is born of a healthy distrust of ad analytics industry bluster.

josefresco 4y ago
> One problem is the ad industry is quite good at finding ways to take credit for conversions that would have happened regardless This is a great counter-point when someone mentions "use conversion tracking!" Conversion tracking is great, but not if they store a cookie for 7/15/30 days and "award" the conversion to the ad, when the customer took a different and varying path to purchase. Sure the ad contributed "some" to the conversion, but not 100%.
- civilized 4y ago
  I strongly suspect they sometimes show the ad and credit it for the conversion after the user shows initial interest in some specific product, i.e. when the user was already highly likely to buy before seeing the ad. This would explain why you're often flooded with ads for something after you search for it, go to its website, or purchase something there. (We've all had that experience of buying something and then being tailed by ads for that thing for weeks afterwards, even though no sane person would think we were going to buy that thing again in such a short time frame.) A story to illustrate: there was once a pizza store that had two guys go out into the city to distribute promotional coupons. The coupons had codes on them so the business could attribute sales to each coupon distributor. John went out into the city and tried his best to drum up new business. Chad stood next to the door of the pizza place and handed a coupon to anyone who was walking in. After a month, 98% of the coupons used were from Chad. Chad got a big bonus and John was let go.

deleted 4y ago

[deleted]

yobbo 4y ago

> this person genuinely thinks that there is no way to measure an ad's performance while This is the case in many situations. The time between initial visit to site to conversion can be weeks, months, or years. Meanwhile, customers will visit site from various platforms. There is no way of attributing value to a single click, it needs to be approximated as an aggregate.

LightHugger 4y ago

Companies that sell ads will do everything they can to convince you there is a measurable difference, but often while the metrics presented to the client are real, they do not always translate to real world impact. There is no easy way to determine whether this is the case without just comparing when the ads are on and off and somehow dealing with the confounding variables, which isn't easy.

m3047 4y ago

_How would you measure this:_ Does this ad improve conversion compared to not having the ad at all?

mbesto 4y ago

> If this person genuinely thinks that there is no way to measure an ad's performance while it is running, they are either too ignorant to be in business, or they hired someone who lied to them. Doing a blackout month is a legitimate technique. I've deeply studied ad performance metrics and they are NOT as conclusive as people might think. If they were, you could just keep increasing the amount of $ spent and the conversions would go up. Ok, not exactly, there is usually a diminishing returns aspect, but you get the point. A great example of why this requires more analysis is branded vs non-branded search terms on AdWords. Let's say you spend $10k/month on the brand "Mattel" (aka branded) and $10k/month on "toys" (aka non-branded). Mattel likely gets you like a 20x ROAS because people are probably searching for a specific Mattel toy. It's also potentially likely that if you completely turned off branded search you would net the same results. Ad attribution is NOTORIOUSLY difficult: https://www.optimizesmart.com/what-is-attribution-problem-in-online-marketing/ https://www.optimizesmart.com/what-is-attribution-problem-in... so to pretend like "this guy is an idiot, he doesn't know what he's doing" is disingenuous.