16 ms·
Back in the early 2010s I worked in ad tech on a data science team, and one of the things we were pushing for was causal A/B testing; basically turn off a campa
by rm999 6y ago
Back in the early 2010s I worked in ad tech on a data science team, and one of the things we were pushing for was causal A/B testing; basically turn off a campaign's advertising to a % of people and correlate it with sales to measure ROI.
As we were kicking this off I was at a conference chatting with an executive at another ad tech company. His response: "oh yeah I know a guy who tried that, he's not in the industry anymore."
We almost immediately came to realize our launch clients were getting negative ROI, sometimes severely so. AFAIK our efforts fizzled out, and I believe none of the people on my team are in the industry anymore.
- saganus 6y agoI might be misunderstanding... but, negarive ROI? I.e, the more they spent on ads, the less sales they got? Feels like a missed something, as that sounds... counterintuivie. (Not in any way related to the ad industry so pardon my ignorance)
- jmfisch 6y agoAs I read it, the amount they were spending on the ads themselves was more than the converted revenue from ad clicks. While the decline was probably (guessing) not linear, spending more on ads led to less than proportionally more revenue. If that had happened I can imagine calling that negative ROI.
- saganus 6y agoThat makes more sense! Thanks
- qeternity 6y agoNo. A negative ROI just implies that the ratio of benefit/cost is less than one. If I incur a cost/investment of $100 but it creates value of $200 then I have an ROI of 100%. If this same expenditure instead only produced $80 value then I would have -20% ROI as in the value I’m realizing from my investment is 20% less than the cost of the investment.
- alacombe 6y agoRatio can't be negative, they can either be above of below 1. What you are talking about is "$benefit - $cost", not "$benefit / cost".
- deleted 6y ago[deleted]
- ddulaney 6y agoROI is (net benefit)/(cost), where net benefit is (gross benefit - cost). Net benefit can be negative, making ROI negative if cost exceeds benefit. GP was using benefit as a shorthand for net benefit, which can certainly be negative.
- loeg 6y agoEven gross benefit could conceivably be negative, if the ads are bad enough! (I.e., a terrible ad might prevent a sale.)
- rocqua 6y agoWe tend to call most rations below 1 'negative'. Its technically wrong but most people get what you mean when you say it.
- ducttapecrown 6y agoPositive and negative are the right words on a log scale :D
- benjam47 6y agoRatio can't be negative, but ROI can. Positive ROI is the percentage of the ratio being 1 or higher. Negative ROI is the percentage of the ratio being less than 1. Or, from a "common sense" perspective, if I spent $1000 on advertising, and earned $100 in revenue from that investment, my ROI is definitely negative.
- jacob019 6y ago
- syndacks 6y agoWhat do you mean by the two references to not being in the ad industry anymore? Quit? Fired? Why?
- karaterobot 6y agoDid they leave the industry because they realized their product wasn't providing a positive ROI, or were they fired for pointing that out to other people? I think the former is probably what you mean, I just want to be sure.
- rm999 6y agoGood question. In the first case it was left vague but I understood it as being fired and then not trying to find a new ad tech job. In the case of my team it was finding better industries to work in.
- ben509 6y agoYeah, it could be that many startups try to do advertising ethically, realize that it doesn't work, and drop out. That's necessarily going to leave all the firms willing to sell snake oil behind. You can get firms selling stuff that doesn't work that are "trusted" simply because they've been around for years; there are plenty of distinguished brands selling homeopathic remedies, audiophile speaker cables, timeshares and MLM schemes.
- cmrdporcupine 6y agoSimilar time frame for me, worked at an ad-tech startup, when retargeting was first becoming a big thing and we were pivoting the startup from the bad business idea it started to to doing retargeting as a demand"-side-platform" (DSP) on ad various exchanges. Tried myself to confirm whether any of it had any positive effect, I couldn't really discern any increase in the click-through-rate (CTR) for various approaches, but I'm not a stats expert, etc. so talked to the founder about how we should employ someone with a stats background. That convo went nowhere, and for that and other reasons I was out the door within a couple months. That was the era of the ad-exchange DSP bubble. After that I went to work at another company that was on the other side of the exchange pipeline and I could see all these DSPs just plugging away doing their thing and none of it looked (to me) like it was accomplishing much. It was all bottomfeeding off of lower quality inventory but I suspect making big promises to investors. That startup eventually pivoted a couple more times and sold to a bigger player a few years later, making some money for the founder but I suspect no value to the buyer.
- dmix 6y agoReading about startups that pivot dramatically multiple times make me cringe so hard. The goal of pivoting isn’t throwing spaghetti at the wall until you hit something before running out of VC or angel money. That just shows terrible product/marketing leadership. Your prior story is unsurprising combined with that.
- bsenftner 6y agoAs a person with just shy of a stats graduate degree, the early 2010's were a nightmare of trying to get ad tech companies to grasp their incompetency. I met a dizzying number of slick talking frauds and became quite jaded.
- cmrdporcupine 6y agoOk good, it wasn't just me being overly negative (because I usually am). I am not a stats grad, my math is weak, but it didn't look good to me, and I tried to get my employer to bring a stats specialist in to analyze. That got a cold response.