5 ms·
I appreciate your thoughtful criticism of this. The reason I say this though, is because if they knew how to run ads properly they would have been tracking the
by tetsusaiga 4y ago
I appreciate your thoughtful criticism of this.
The reason I say this though, is because if they knew how to run ads properly they would have been tracking their results from the start, and would have known much earlier whether they were getting a return on their investment.
If your only way of measuring advertising results is to "turn it off", you're just flying blind, and one can't expect success with a (lack of) strategy like that.
That said, you are not wrong that a meaningful percentage of advertising is being run with similarly insufficient statistical power, and to that I would say those businesses are also incorrect, for the most part. I delineate because at some point, say when you're Apple or Microsoft, you are so big that "brand awareness" advertising takes over performance advertising. For the most part though, I'd say those aren't the types of businesses we are discussing in a context like this one.
- cj 4y ago> The reason I say this though, is because if they knew how to run ads properly they would have been tracking their results from the start Part of the problem is your brand keywords will typically show up as being one of your best converting keywords.
- staticautomatic 4y agoA problem in what sense?
- jefftk 4y agoIt's a problem in that those keywords are some of the places where you are at least likely to be generating counterfactual conversions: most of that traffic was probably coming to anyway.
- VBprogrammer 4y agoA phenomenon which reminds me of the canonical survivorship bias story. In WWII they conducted studies to determine where the bullet holes where on aircraft which returned from bombing sorties, in order to determine which parts of the aircraft required armour. It took a statistician to point out that they actual needed to armour those places where they rarely saw damage on returning aircraft, as those parts are most likely the parts where being hit caused the aircraft to not return at all. Sometimes it requires a bit of a leap of imagination in order to resolve these things.
- tomrod 4y agoCausal inference has made a lot of improvements since WWII, and "if" the advertising company knows what they are doing they run effective A/B or MAB testing; that said, statistical power is typically low because of insufficient sample size for individual companies. You could pool all ads together, but since each advertising company is independent you get into all kinds of weird path dependencies. While I wouldn't claim to be an adtech practitioner, I did at one point help a few F500 work through conceptual models of multitouch attribution and other statistical issues. These are very nontrivial issues -- proving advertising effectiveness is very difficult!
- Retric 4y agoTracking advertising effectiveness is ridiculously difficult and multiple people inside and outside your company are incentivized to overstate impact. Statistical power for example assumes independence which can be very difficult. Great you spend X million to convince people to buy an AC in March, did you actually benefit or would those same customers want an AC as soon as the first heat wave hit? Spreading demand can be useful, but it’s also really easy to to draw false conclusions from statistics if you don’t understand the domain. And that’s just one of the many pitfalls involved.
- tomrod 4y agoYou hit the nail on the head, for the record. Thank you!
- tomrod 4y agoHow do you know any strategy is attributable to success or failure without testing it? Pre/post analysis may be temporally correlated but this isn't proof because you haven't captured a baseline comparison. A/B and MAB testing are helpful but not magic bullets. Shapley values (marginal impact) is a nice mathematical outcome to have for multitouch attribution but as usually implemented is only a single statistic and can be a fluke without additional testing.