9 ms·

A/B testing gets misused to juice metrics in the short term

• Kaotique 4y ago
  AB testing shows zero respect to your customers. It is the equivalent of testing your theories on lab rats. Instead try to improve the customer experience, make better products, improve customer service.
  • rgavuliak 4y ago
    I have experience where the company paid a UX agency to create a flow that was by all standards better customer experience and a better product, nicer too. They ran an AB test, turns out people were more likely to pay with the old version. AB testing is good that it challenges what UX people think is better experience or product with hard metrics.
    • nkrisc 4y ago
      Which is why A/B testing is an important part of the UX toolkit. It's a tool among others, and is one way to validate assumptions. A good UX designer will try to base their designs on data and reasonable hypotheses drawn from the data, but a new design or flow is necessarily based on some amount of assumptions, so it requires validation. That said, an A/B test does not tell you why something didn't work. You can make further assumptions based on the results and develop new hypotheses, but it never tells you why. Typically you would do some kind of qualitative UX research on a prototype or even static concepts beforehand to identify these kinds of issues before you even expend the effort to do a live A/B test. Far cheaper to do a study with 6-12 people and a prototype than to build out a full, functioning A/B test experience. It's possible the flow they created was generally better but perhaps it had one fatal flaw. Perhaps that flaw could easily be remedied once identified. A/B testing is just one small part of a good UX process.
  • smeyer 4y ago
    >Instead try to improve the customer experience AB testing can be (although isn't always) used to improve the customer experience. Assuming you know exactly what will make the customer experience best without actually testing it can also lead to a worse experience.
    • throwaway290 4y ago
      A/B testing helps you maximize a metric, not make customer experience better. Those are different things.
      • ssharp 4y ago
        Better customer experiences often times lead to increased metrics. They are not totally different things.
        • wongarsu 4y ago
          Maybe if you are very aware of the fact that your goal and metric aren't totally aligned, but that often gets lost. As a result A/B testing for longer website visits can make websites that make it more obvious that the information you want is there, but also make the path to actually get it longer. A/B testing for engagement might promote divisive behavior and fights. A/B testing for read rate or clicks might lead to trust loss. I think a lot of lessons from AI safety apply surprisingly well to A/B testing, mainly around how hard it is to align your actual goals with the metrics you use for optimization, and how disastrous the consequences can be. It doesn't have to go wrong, but it's incredibly hard to ensure it goes right, especially if it's the only feedback you have.
          • noirbot 4y ago
            But that's just optimizing for bad metrics. At this point, anyone who thinks "engagement" and "time spent on page" are customer-positive metrics is in a different mental space than you and I. There's a lot of ineffable things that make up good customer experience that would be hard-to-impossible to A/B test, but it doesn't mean that A/B testing is "unsafe" just because it could be used to optimize for bad things any more than any other telemetry or metrics gathering could be bad because you could optimize for evil things. And at the same time, bad management and product leadership can optimize and develop towards bad goals with plenty of tools that aren't A/B testing. It seems to miss the point to blame/stigmatize a specific tool because it's been used poorly by a few bad actors in a public way.
  • Saturdays 4y ago
    That's a very weak blanket statement, there are totally reasonable A/B tests you can run that don't deteriorate a user's experience, and the results can guide you to a better customer experience overall.
    • Kaotique 4y ago
      It did not mean it too seriously, of course there are also good AB tests, but there are a lot of bad ones out there. Those are what the article was about. (edited for clarification)
      • HelloNurse 4y ago
        Obviously, it depends on what the A/B test is about. Molesting or not the customer for the sake of some shortsighted metric is a bad choice; deciding what content should go above the fold or not (e.g. Amazon places images, short description, details/specs and similar products in that order) is a good choice.
  • treis 4y ago
    >Instead try to improve the customer experience, make better products, improve customer service. Without a metric to say what is "better" and a method to measure it this is empty advice.
    • josefx 4y ago
      But does AB testing provide the only or even best metric for that? It probably is the cheapest way requiring the least engagement with the lab mice.
      • treis 4y ago
        AB testing is how you isolate a change and measure the impact. It's the only real way to be able to associate cause and effect. Best you can do otherwise is measure something over time while making changes. You can try and correlate changes with outcomes but it's hard to be sure the change is what drove the outcome.
        • oba2311 4y ago
          That sounds pretty accurate. Anyone who's gone through some econometric classes would know that split testing is (almost) the only way to get to (close to) causality - all the rest is black magic called correlation which can also lead you to the conclusion that Nicolas Cage's movies lead to people drowning: https://www.thenationalnews.com/uae/nicolas-cage-movies-linked-to-drownings-and-other-spurious-correlations-1.450759#:~:text=An%20analysis%20of%20data%20by,that%20correlation%20is%20not%20causation https://www.thenationalnews.com/uae/nicolas-cage-movies-link.... https://www.wnycstudios.org/podcasts/otm/articles/spurious-correlations https://www.wnycstudios.org/podcasts/otm/articles/spurious-c...
    • Kaotique 4y ago
      There are many other metrics that do not involve AB testing. You can just survey customer experience before, during and after a purchase for example. I never said to throw out all metrics. With AB testing your are optimising for a specific outcome. Usually higher conversion. As pointed out in the article eventually you'll end up with a bunch of colourful buttons and scary texts that persuade the user to click. A lot of the "only 2 seats/rooms available" are lies to scare the user into a conversion.
• Eddy_Viscosity2 4y ago
  I'd say this is a sub-category of the saying: "There is nothing in this world that an MBA can't and won't make worse". It happens when people change perspectives from building and sustaining businesses to exploiting and squeezing every employee, supplier, and customer for the last drop.
  • jollybean 4y ago
    This is just casual bigotry.
    • verve_rat 4y ago
      Calling that bigotry is offensive to anyone that has been subjected to actual bigotry. You should be ashamed.
      • jollybean 4y ago
        • grapeskin 4y ago
          MBAs aren’t something people are born into or end up through no choice of their own. Bigotry against MBAs is impossible. It’s like being a loan shark or pimp. People actively choose that life and society talks poorly about them because of the social harm they bring. There’s nothing stopping someone from washing their hands of any of those things and being productive.
          • jollybean 4y ago
            • verve_rat 4y ago
              Your behaviour here isn't a good argument for this thread's feelings about MBAs being "obstinate or unreasonable".
• readingnews 4y ago
  Hrm, looks like A/B testing has destroyed the website.
• EricMausler 4y ago
  /rant AB testing is and always has been fish oil for management. The only things it can actually prove, are more easily identifiable by common sense. So wherever it actually works, it was probably a waste of time / overkill for evidence. - sincerely, a business analyst
  • tbranyen 4y ago
    Have to disagree. I've found plenty of issues that affect real production users through the use of AB testing. Problems that were small enough to escape review, testing, and reporting, but large enough to be stat-sig. They always lead to a bug, or issue with test vs control. I will always use AB testing for uncertain code in the future. I was skeptical when I first started writing AB tests, but they have proven their worth over and over again.
    • ratww 4y ago
      Sure, but that's not really A/B testing, those are more often called staged rollouts or progressive rollouts.
      • tbranyen 4y ago
        I'm talking about running week or month long tests with control and multi test cells containing new functionality, configuration, or code to determine the viability of a single or combination of changes by analyzing statistical output driven by p-value and pre-determined target metrics. These types of experiments are extremely valuable in uncovering hard-to-find bugs, assuming you have sufficient logging and confidence around your metrics. They let you know a problem exists and roughly where it is in the product. From there you can drill down and investigate your source code until the discrepancy is found.
        • EricMausler 4y ago
          This makes sense to me. Not the kind of AB testing I had in mind, but fair point. I was thinking more about decision making processes, not operations troubleshooting.
• nuc1e0n 4y ago
  This kind of optimisation for short term gains at the expense of long term sustainability is what is causing climate change and the collapse of the global economy. But the politicians and heads of industry who preside over this situation will all be retired/dead before it becomes a problem. Or so they thought.
• causi 4y ago
  The problem with AB testing is that it's a short-term strategy. For example, if a news site runs AB testing with headlines, they'll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles get the most alluring headlines.
  • dr_dshiv 4y ago
    I’m a huge fan of metrics. Huge! But they are worthless when not combined with qualitative experience. AB testing needs to be combined with human-centered “actually talking to people about their experiences.” Otherwise, you drift and the metrics no longer match the objective.
  • boruto 4y ago
    Indeed, If do more ad placements increase revenue is the test and then there is 20% jump what are you as an engineer going to do? Tell to management that its bad?
  • goodside 4y ago
    There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made the statistics more complicated.
    • im3w1l 4y ago
      Did you A/B test the matching algorithms?
      • goodside 4y ago
        Don’t recall specifically. There maybe may have been randomized A/B tests on the “special blend” at some point, I think — it was never spelled out on the site, but I think that was the experimental mix du jour and we tended to use that instead of forced randomization.
• mschuster91 4y ago
  My biggest issue with A/B testing isn't even mentioned here... gaslighting your customers is absolutely not OK. Particularly with older people, the constant "where the f..k did Outlook now put feature XYZ" (in the case that comes to my mind, the CC bar which used to be tab-reachable, now you have to tab+space or manually click on the tiny gray "cc" letters) onslaught is just absurd. When you change how applications behave without telling the users, it's a direct attack on their muscle memory at best and makes them question their sanity at worst. My second biggest issue is: it's rare that companies offer actual, live-human support these days anyway. When marketing adds A/B testing, shit becomes really annoying if something breaks as a result - usually the phone lines are suddenly flooded, the agents have no idea what has happened either and try to reproduce and figure out what's going on (and sometimes can't because they aren't part of the test group!), and so even people who haven't been in the testing group are going to be very pissed off. IMHO, A/B testing without explicitly notifying the customers in advance should be banned by law, and that ban be harshly enforced. Customers are not guinea pigs, and with the rise of elderly people on the Internet this becomes an actual public safety issue (as ever-changing stuff makes it easier for scammers!).
  • mjburgess 4y ago
    You're downvoted, but this issue is more common than it seems; and I agree, more serious than it seems. You're describing adversarial UI changes to small populations of then unsupported customers. This can have outsized impacts on vulnerable populations, eg., esp. the elderly.
  • emsixteen 4y ago
    > it's rare that companies offer actual, live-human support these days anyway. This is one of my most intense frustrations in the modern age. Complete and utter disrespect for your customers' time and knowledge.
  • Akronymus 4y ago
    Not to mention the lovely "This option that used to be here no longer is here." getting the response of "its still here (for me)". Youtube specifically loves to do this to me.
• kjhgkjghkj 4y ago
  Intentional or not, one outcome on sites that are relentlessly A/B tested is that the resulting UI design lets users know that content they want is there, they just need to click and scroll a bit more to find it. Having left FB years ago, I now watch people "navigate" their site/apps with disbelief.
  • regularfry 4y ago
    Isn't that exactly the problem? The resulting UI isn't designed, it's aggregated across a disjointed set of granular tweaks.
    • kjhgkjghkj 4y ago
      A problem for who? Given that people already invested in the product ecosystem seem to have almost limitless patience to scroll for the right content, I'm sure it improves almost every user time and attention metric. It's why I saw it as my moral duty to leave (as well as the other FB properties), so that at least in a small way, I "produce content" that is only available by interacting with me as a person.
    • ssharp 4y ago
      This is part of the "unchecked" part of AB testing the headline mentions. You, of course, need to ensure the granular tweaks can be rolled up into something usable as the granular tweaks prove successful. You can't just keep bolting on UI changes while losing sight of the larger experience. Each incremental A/B test is testing against a previously successful variant so eventually the control is radically different from where it started and you're only concerned about beating the control. Using a longer-term holdout group or reseting the control experience during incremental testing can help mitigate this and get you zoomed out a bit from the local maxima.
• jerf 4y ago
  "As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money." This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page, and some stupid interstitial pops up. Dude, I'm trying to enter my credit card information! Back off! Especially stupid for a "sign up for our newsletter" popup; we all know that unclicking the "yes, we can email you every 17 seconds from now until the heat death of the universe with valuable offers from 'our affiliates' which we define as 'anyone we share a species with'!" box on the checkout form is mandatory, and if we don't see it immediately we'd best go hunting for it. You've already default populating the checkbox to "yes" on this very screen, get out of the way! Less unbelievably stupid, but related, is when I'm examining product X and just after I scroll down a bit to read more you pop up something related to... well... anything other than product X! I'm signalling interest in product X as hard as I can, and you've AB tested that this is a great time to jangle your keys over there instead? Your AB testing is stupid and can't possibly fail to be some stupid statistical fluke or other terrible error. What fisherman goes out on his boat, hooks a fish, and then rushes to throw another completely different lure out to the hooked fish and get them on that hook instead? This is another good touchstone for being "overoptimized".
  • tshaddox 4y ago
    The thing is, you haven’t really shown that these sites aren’t successfully optimizing for conversions. Couldn’t it very well be the case that UI which annoys some high-intent users by interrupting them or adding steps to the checkout process also increases overall conversions?
    • jl6 4y ago
      True, A could be “annoy users” and B could be “don’t annoy users”, and A could perform better overall, but in this framework you might be missing C which is “annoy users except those already deep in the funnel”.
• test1235 4y ago
  archive: https://archive.ph/fuUPG https://archive.ph/fuUPG
• benja123 4y ago
  I say this a lot and I will keep saying it. Conversion != customer obsession. There is a place for A/B testing. It is necessary and can be extremely beneficial in helping your customers enjoy and use your product more successfully. The main issue is that people mix conversion with customer obsession! Whenever you work on a product or feature you should be asking yourself "Is this really good for my customer" - if the answer is no, then no matter what the A/B tests/conversion rates show you don't do it. Unfortunately we mostly hire the wrong people as PMs, who then hire clones of themselves. They are not truly customer obsessed and use A/B tests incorrectly which results in products that trick or force customers to do things they don't understand/want to do. Long term this is bad for the product and company
  • jfoster 4y ago
    Yeah, this is key. Improving a product in the direction of customer intent vs against customer intent.
  • 10x-dev 4y ago
    My 'favorite' silly thing PMs do is UX research studies (typically on 5-10 people) and essentially ask completely untrained people if we should go with X/Y or Z. It's a super-effective way of avoiding responsibility for product decisions ("the data suggest we should go with Y"). If only building good products were as easy as asking what customers think they want.
    • throwaway98797 4y ago
      only listen to customers problems and never their solutions
    • HWR_14 4y ago
      Either they're doing the UX research wrong or (more likely) you're misunderstanding the process. You don't ask them if you should do X/Y/Z. You ask them to do X in the program, and see that none of them can find widget Y which controls it because they keep clicking on widget Z. It's about observing the users fumble through your UX when you know their motivation.
• hbn 4y ago
  I've talked about this on HN but I'll say it again. A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a big part of that was the constant A/B/C/D/E/F testing. I never felt like I was using a complete product, everything felt like a constant beta that could be changed or rearranged at any point, and I was just doing free testing work for them while they harvest all my data. Every app update was a risk of the app rearranging itself, or features appearing/disappearing. Eventually it didn't even come from app updates in the Play Store, and new interfaces would just appear one day when a server somewhere marked your account as being in the group that gets the new UI. This app that you were familiar with could at any point be rearranged when you open it on any given day. Then maybe a week later you open it and it's back to how it was before. A button you thought was here suddenly isn't, and you question whether something actually changed or if you're losing your mind. It's a subtle gaslighting that eventually I couldn't stand any more. To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. That's not to say a company like Apple is creating products out of the goodness of their heart, but at least it feels like it was developed by humans who made an opinionated call as to what they thought was the right design decision, and what they would want to use. And in my 2 years of owning an iPhone, I've never opened my reminders app to find out that it's completely unrecognizable, or my messages app has been renamed or rethemed for the umpteenth time.
  • aceazzameen 4y ago
    You nailed it. Google is constantly forcing users to relearn most of their products year after year. Give me Google products from a decade ago and I'd still be happy. Now I'm moving on from Google also. It's an untrustworthy brand.
  • costcofries 4y ago
    "To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. " Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive more value from the product. Bad A/B testing tries to make things more discoverable, where discoverability is often just noise and distractions. Good A/B testing ensures that the money machine, as you put it, pays its dues to users by making the product experience delightful.
• AndrewThrowaway 4y ago
  Imagine this beautiful business software which during the years and numerous A/B tests, "best UX practices", design languages and whatnot became this all "applesque", minimalist UI with 80% of it being a white space. By the way winning numerous design awards. However entering e.g. client's information take a lot of steps, you are constantly clicking "Next" throughout these beautiful wizards and pages. After some time everybody starts to feel that there must be a better way. What is the solution? Spreadsheet import! Where you can just do everything in this "complicated" UI of Microsoft Excel, with formulas, and hundred buttons at once on the screen. Fill in hundreds of rows of information and just import it to the "beautiful business system".
• axg11 4y ago
  A/B testing is local optimization. It should only be done on a mature(-ish) product when you have given up on finding a global minimum. Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through. Early stage startups should never A/B test. When you're searching for product market fit, you're doing global optimization within the search space. Your product will change drastically as you make new learnings. Premature optimization (A/B tests) will only be detrimental.
  • andsoitis 4y ago
    Even for a mature product where you might be doing A/B tests to explore hypotheses that you think will improve the product for the user, it is also worth considering doing mountain tests where you try to escape the local maxima.
  • ravivyas 4y ago
    "Running experiments and A/B tests are popular" ... because you can give up on your own judgment and opinions and say "the data says"
    • 411111111111111 4y ago
      > give up on your own judgment and opinions and say "the data says" The beauty of AB testing is that you don't have to give up your opinion. You can just change irrelevant things until the result you desire gets proven by chance and now you've got data to base your opinion on!
  • jaggederest 4y ago
    > Running experiments and A/B tests are popular because it is _guaranteed_ to give you signal. If you have a large engineering team and you're not sure how to filter the quality of results, gating everything through A/B tests is a well understood methodical way to ensure only positive work makes it way through. It's almost guaranteed to ensure only false positive work makes its way through. If you're picking 0.05 as your P value, and you're running dozens to hundreds of tests, your false positives are almost certain to exceed your actual positives. When I'm working for clients that do a lot of A/B testing, I suggest that they should always run A/A tests to ensure that they're not incorrectly rejecting the null hypothesis. If your A/A tests are showing significant differences, you have a problem in your testing pipeline that by definition can't be cured by more testing. You need holdout groups and selectivity about what to test, instead of just throwing everything at the proverbial wall.
• twawaaay 4y ago
  I think the main issue is that testing is misused to create better version of something when it should be used to create knowledge. So if you do testing and it gives you some kind of result, the crucial step is trying to understand what it really means, is there something we can learn from it. Unfortunately, this is also the hard part that requires actual effort and intelligence and is difficult to scale -- and so is frequently skipped.
• fleddr 4y ago
  I absolutely agree that A/B testing in the way described in the article is a catalyst for creating dark patterns in a UI. Because dark patterns work, they deliver short term increases in particular metrics. The author's idea is that this short term gain damages longer term metrics. That sounds logical and agreeable, but that doesn't make it true. Not in my experience anyway. Probably the people complaining the most about annoying UI patterns weren't going to convert anyway. Whilst those coming with a specific conversion goal to your site will convert even if annoyed in the process. Anyway, the true root cause goes all the way to the top. When you give a team a 20% sales increase target and "deliver by next quarter or be fired"...this is what you get. If the executive level dismisses a healthier, more sustainable long term growth model, then there's pretty much no way to stop this. It's so hard to stop because it actually works. It works short term and evidence that it harms long time is typically lacking or it simply isn't true.
• redleggedfrog 4y ago
  I think that you have to take into account the popularity of these methods when evaluating whether to implement them. It would seem the more sites that do these obtrusive UI patterns the less effective they become. Anecdotally nearly every method described in the article is an automatic back button off the site for me.
• _the_inflator 4y ago
  Maybe Apple will come up with a reality distortion field and will remove "urgency" warnings and informations from websites on Safari, as well as blocking "Join our Newsletter now and get a discount" pop-ups. What once was ads everywhere, is now psychological gaming. I hope someone comes up with a Google Extension, and maybe Apple with a new "Access Website" mode. These messages are boring to be honest. Once you noticed them everywhere, game over for me. Time to move on.
• ravivyas 4y ago
  In todays world of algorithms optimising marketing, and constant updates on marketing channels, it is hard to say if an A/B test worked as quality of users is never consistent. I currently work in a game publishing company, here are 2 anecdotes from it 1. We run an A/B for game performance but we keep changing the bids for our games, and thus get varied quality of users, A/B tests don't really help in such a case 2. Once by mistake we ran the same creative on FB for 2 different ads.. both ended up having totally different metrics
• mattgreenrocks 4y ago
  I recall when Booking.com rolled out the false urgency features. I was amazed at how utterly trashy and desperate they were. The problem is it's not subtle at all; there's a handful of those features that, when combined, end up being overbearing and noisy: "3 people looked at this listing within the past 3 days! 12 rooms left at this rate!" I don't care. I'm looking to book business or vacation travel. If a spot fills up I'll just go somewhere else. It'll be fine either way. I don't use them anymore for that reason. Old soul (me) is old. (I'm probably in a minority, judging by their advertisement budget.)
  • iamben 4y ago
    It's amazing how much this works though. I remember getting a call from my mum saying "the website is telling me London is already 78% booked for these dates!" It felt ridiculous having to say "Mum, it's March. You're staying there in November. I promise you it'll be fine..."
  • coldcode 4y ago
    Having worked in the OTA space, every time I see their (sometimes funny) ads, I want to call them Booking.Nope OTA make comparisons a bit easier, but everything is negotiated and contractually controlled to keep people from just going to the hotel directly. Secret hotel prices (like HotWire if that still exists (Expedia) or Travelocity's Top Secret hotels if that still exists (also Expedia)) are an even more crazy negotiation. Hotel Tonight at least used to contact the hotel chains every day for that day's options, though since they were bought by AirBnB who knows what they do. These days I just find a nice hotel and book with them/their system directly. Airlines too, since airlines fail to give all their options to the OTAs. In some ways its sad that aggregators don't work all that well in the main travel industry (Flight/Hotel/Car) but travel is extremely complicated, highly competitive and still very fractured except for airlines. Pricing comparisons are not very useful since they are so mangled and obfuscated that you may as well just go to several sites and do it yourself by hand. For example Spirit Airlines used to give us prices for their tickets at $X and were always cheaper than everyone else; yet once you booked at that price they hit you for everything extra (bags, res, for all I know oxygen) then our customers complained we were fooling them and the real cost was higher.
• andreareina 4y ago
  Getting a 503, so here's an archive: https://web.archive.org/web/20220712122630/https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-destroys-everything-it-touches/ https://web.archive.org/web/20220712122630/https://www.zumst...
• happimess 4y ago
  I had a PM who pushed us to A/B test _everything_. We hired a new graphic designer who suggested that we change our product links from ALL CAPS to Title Case (a very popular idea on the team, and his first real suggestion after a few weeks with us), and she insisted that we A/B test it first. It felt like an insult to him, and a dumb test since title case looked way better. The three key outcomes I observed from the relentless A/B testing were UI antipatterns, team burnout, and a well-attended conference talk about "how we ran 105 A/B tests in a year, and what we learned".
  • gorbachev 4y ago
    Was one of the learnings "Everyone hated the product manager"?
  • edmundsauto 4y ago
    I've had a similar experience, although my learning was "people, even experts, are really terrible at understanding which treatment will perform better". We always run >=3 variants, surveyed the dozen team members on which one they thought would run. Over the years, there was no clear pattern over who could make that prediction. IE, it's not possible to predict which is the most effective treatment, even when you include a really bad idea in the treatments!
• chunkyks 4y ago
  "If a study came out that said deafening high-pitched noises increased conversion rates, we would all be bleeding from our ears by end of business tomorrow, right?" Netflix auto play? Is that you? You were a hateful idea, no one liked you, yet you stubbornly hung on for far too long
  • _tom_ 4y ago
    I'm convinced that Netflix uses "number of hours watched" as a success metric. Autoplay raises that. I'd pay twice as much to watch half as much quality programming, but that would tank what they think is a positive metric.
    • chunkyks 4y ago
      I'm pretty confident that that horrorshow was a case study in A/B testing: People can't decide what to watch and agitate over flicking through titles, so some bright spark had the idea of just starting stuff in the background to see if that helped. It did! People started watching stuff sooner! Mostly because it was so incredibly irritating that I'd start a show just to make the random autoplay cease. Of course, what it really meant was that I'd go to amazon and agonize over what to watch, but at least I was doing it at my own speed and not being harassed by noise autoplay.
• amluto 4y ago
  Reading this makes me think of the handful of sites, often targeted at professionals, that highly optimize for the experience of actually buying things. McMaster-Carr comes to mind. Their users shop there over and over, and McMaster wants to keep them. So you can find things for $2 or $2000, shipping prices are inoffensive, customer service is friendly but rarely needed, and there are minimal distractions on the way to checking out or even after checking out. The only real issues are mostly related to the fact that they sell so many products that one can get lost in the 4000+ items that all match the search. Well done. This is an interesting contrast to Amazon that also makes checkout easy but bombasts the user with thousands of listings, mostly mildly fraudulent and consisting of absolute crap, and still somehow gets repeat business.
  • rightbyte 4y ago
    McMaster-Carr might be the single thing I miss the most from my time in the US. It is like ... stupid good. Their listings catalogization is like godlike compared to alternatives. The Amazon or Google way of throwing all things into the bin and spew it out to the users is BS. We are saying we live in an information age but I firmly believe stuff were way better catalogizised back when it was done manually by paid gatekeepers.
  • int_19h 4y ago
    https://www.usplastic.com/ https://www.usplastic.com/ is another "industrial" example.
  • saagarjha 4y ago
    > This is an interesting contrast to Amazon that also makes checkout easy Hey, would you like Prime with that? Do you know we provide free two-day shipping with Prime? If you sign up for Prime today you can get a $100 discount!
• ryanmarsh 4y ago
  For many businesses revenue is a function of aggressive deal making. Full stop. In an undifferentiated market of discretionary (impulse) purchases if you don't hustle the customer you make less. The author of this article is confusing companies that are bad at hustling with hustling being bad. One time offers, limited time offers, mailing list signups, up-sells, and cross sells are time tested ways to increase sales dating as far back as radio era telephone and catalog sales. Steve Madden is a perfect example of this. They sell undifferentiated popular shoe styles less expensive than high fashion but more expensive than knockoffs. They have to hustle you to get you on their mailing list (for 10% off your order) in the hopes that you'll make another impulse purchase later when you get a text or email from them. If they weren't as aggressive you might never make another impulse purchase with them again as there are tons of brands selling nearly identical products. Some companies are just horrible at hustling so they actually get in the way of you completing your purchase. In a competitive market this is a self correcting problem.
• fairity 4y ago
  It doesn’t seem like the author has any hard data that supports his claim that long term LTV and K-factor losses outweigh short term conversion rate wins. Maybe I missed it? Without said evidence, it’s probably safe to assume his generalized claim is wrong in most cases.
• weeksie 4y ago
  Bright eyed PMs whispering "statistically significant" to themselves over and over as they nervously scan their data aggregation dashboards for wiggles.
• commandlinefan 4y ago
  > Next to some hotels, a message that supply was limited. It's also worth noting that there's no way in hell they actually know that with any sort of precision. No GDS has proper up-to-date knowledge of bookings from all the various sources that hotel reservations actually go through (they overbook airline flights). What they're really saying is that the small inventory of rooms that are reserved for them to book exclusively are almost gone.
• WhitneyLand 4y ago
  What an excellent write up. I agree with the sentiment on AB testing but I think the bigger insight is that we need to be reminded to see the forest for the trees with any process, tool, or goal. Sometimes these intangibles are hard to measure and almost need to be sensed. It reminds me of how you can see the exact same development methodology used at two different companies, where at one company it works beautifully and at the other it becomes a bureaucratic albatross.
• forgotmypw17 4y ago
  I have developed a personal strategy of ridding the Web of these things. Anytime it happens, I close the tab and move along. Very little of value is lost.
  • donmcronald 4y ago
    This is basically what I do. Anything that pops up or tries to grab my attention gets instant closed before I look at it and if I can't find the control to close it in 1s I just close the whole tab.
• walrus01 4y ago
  If you really want to see a massive amount of additional offers and small/partially hidden "no thanks" links, check out the work flow to reserve and rent a small light duty trailer with U-Haul. You have to click through at least 10 pages of additional offers (and many extra price things that are added by default!) before you get to the actual checkout page.
• epolanski 4y ago
  Anecdotal: we released plenty of improved features, like a better gallery to see the items in our shop, users used it a lot +250%, but conversion rate went down 4%. They spent more time seeing the items and..didn't like the pics and conversion went down. In the end we reverted to the crap gallery we had before, they don't click it anymore and conversion went back up again..
  • bee_rider 4y ago
    If it possible that there'd be a long term effect like: * Users know you have a nice gallery * They are more likely to shop at your store * In the end, you get more sales despite the lower conversation rate
    • epolanski 4y ago
      I agree with your point, but after finding out that in this industry you just need to be able to monkey some code to be called an engineer, random people are now data analysts because they can pull "experiment A revenue up, experiment B revenue down" and call it a day.
• ghostly_s 4y ago
  Nothing torpedoes my opinion of a brand more effectively than one of those insulting "Yes, spam me!"/"No, I'm a moron who hates saving money" popups. Absolutely mind-boggling that any thinking person thought that was an okay way to talk to customers.
• gumby 4y ago
  Such perfect timing: I just tried to place a take out lunch order with a restaurant. Opening the page popped up a modal box that said "Join Our List Subscribe to find out about new specials, community events, store openings and more." There were no buttons to click, no place to enter my email address (had I wanted to) and clicking did not dismiss it. The modal had a background that obscured the actual page. I finally opened the inspector and deleted it, so that I could use the menu to select "order online", which took me to a page ... with the same modal.
• ryanmcbride 4y ago
  I give an incredibly similar warning every time a company I'm working for starts trying to dip their toe into A/B testing. I have a lot of experience with it at scale (one at a fortune 100 company) and I've even built an a la carte testing framework in aws for a company that didn't like Target or Optimizely. Every single time I warn them about how the bill of goods they've been sold with A/B testing is almost completely unattainable, especially in the way that they want to go about it. They won't magically start getting more conversions by changing a button color. Even if they start getting more clicks, they rarely start getting more complete conversions, because the increased numbers is usually from people who weren't good leads in the first place. On top of that every company I've worked with has no idea what the real methodology for good tests is, no matter how many times I explain it or put it in a slide deck. I would constantly get requests to use A/B testing for feature rollout. Them: "Hey, could you do an A/B test of our existing site design and our upcoming redesign?" Me: "if the old design performs better are you going to toss out the redesign?" Them: "No we're going with the redesign but we want metrics on how it'll affect traffic" Me: "Those metrics are useless if you aren't going to listen to them, and if the results come back and the old design performs better, you're not even going to put it in a presentation because it's counter to your planned actions. There's literally no point in running this test" Them: "Run it anyway"
• londons_explore 4y ago
  You can get long term results from AB tests long after the test has ended... For example, you can see if Group A or Group B from a test are more likely to still use the site 1 year later. You hypothesize that those ways to 'juice the metrics in the short term' hurt the user experience in the long term... Well if your hypothesis is right, these long term AB results should show it.
  • epolanski 4y ago
    > For example, you can see if Group A or Group B from a test are more likely to still use the site 1 year later. This isn't very feasible on most products and certainly limited by the amount of data collected.
• tracerbulletx 4y ago
  If you're upset about internet retail, I hope you're also upset about milk being in the back of the store to get you to walk through the whole thing, because this has been merchandising's bread and butter for a very long time.
  • slotrans 4y ago
    This is a long-standing and oft-repeated myth. Milk is in the back of the store because that's where it makes sense to have a refrigerated wall. Milk is increasingly available in smaller quantities in compact refrigeration units at the front of the store.
    • tracerbulletx 4y ago
      It's probably both. But where products are placed in a store are uncontroversially designed to sell more not to make your life more convenient. https://www.npr.org/2014/08/01/337034378/everyone-goes-to-the-store-to-get-milk-so-whys-it-way-in-the-back https://www.npr.org/2014/08/01/337034378/everyone-goes-to-th...
  • epolanski 4y ago
    Eggs and salt are a common dark pattern, milk is in the fridges with diary products in most of europe, hard to miss.
• lifeplusplus 4y ago
  A nice way to summarize this article to think about local maxima and global maxima. A/B testing right now is done on cohort basis and tests are ran for weeks to couple of months. This means where lifetime span of a customer is beyond few weeks and months, it's really not possible to tell if global maximum was missed. I.e. you increase the number of promotional emails the customers get per week. You do it for 3 weeks and see that customers who got those emails had higher conversion. But you didn't get to see that customers who kept getting those higher number of emails completely unsubscribed after 3 months of pain. But by this time all customers are on higher frequency group so it's hard to tell what would be driving the unsubscriptions. I'm no expert but here are some solutions: 1. You should have really delayed long running control groups. Preferably going well beyond average duration your customer sticks around. These groups should get onto new things a year after. But even then it'd be not possible to take out WHAT feature is affecting them, because in 1 year main group would have accumulated lot of features. But still something... 2. You should really have lots of secondary KPIs that measure things that affect long term KPIs. Sure conversion is better, but is time spent reading newsletters increasing? Are buyers feeling good about their experience with the brand... some of these KPI are more qualitative and can't be just automated. what else?
• scotty79 4y ago
  Giraffe neck is the result of a/b testing. If you known its inside anatomy you know what I mean.
• phendrenad2 4y ago
  A/B testing can be powerful, but you quickly lose your editorial voice and your headlines become the same clickbait garbage that works for bottom-tier blogspammers. Look at a site like The Register. Could they use A/B testing to pick headlines? If they do, it's a light usage, because the clever and witty headlines have an internal consistency that I've come to enjoy and expect.
• fairity 4y ago
  Yes, it's possible to use A/B testing in a short-sighted fashion. Yes, I'm sure there are plenty of examples thereof. No, that doesn't mean A/B testing is inherently short-sighted. It's entirely possible to measure long-term secondary effects of an A/B test. Just save a record of treatment groups, and remember to come back and compare long-term metrics like LTV down the road. We do this all the time at my startup, and of the dark patterns that we've tested, we rarely see a long-term negative impact on LTV that outweighs the positive conversion rate impact. If you want to make a valid argument against dark patterns (which is basically what 90% of this thread is trying to do), it's unlikely to be grounded in efficacy. This is coming from a business owner who spends seven figures monthly on advertising, constantly split tests, and is heavily invested in only making decisions that are in the long-term interest of the business.
  • epups 4y ago
    Yes, the author was not talking about A/B testing at all, but about popups and annoying UX patterns that get rewarded by poorly used A/B tests or short-term thinking. But there is nothing stopping anybody from using long-term A/B tests (I assume most companies do, like you). Unfortunately dark patterns might simply be profitable.
• langsoul-com 4y ago
  Nobody gets a promotion for a long term oriented ab test. Hence short term is here to stay. Saying you made a 10% purchase rate improvement in a month is an easy pay rise.
• morelandjs 4y ago
  I work in this field. Author makes great points. Here are some additional thoughts: * Don't use bad optimization to discredit all optimization * Incremental A/B testing assumes that all accumulated features are independent with no interactions, which is almost assuredly a poor assumption as a website grows and space becomes limited * A/B testing is the only reliable tool to test hypotheses; causal inference is great but not a substitute * Incremental A/B testing (small changes) should be periodically coupled with mutational A/B testing (large changes) to tunnel from local optima to global optima
  • civilized 4y ago
    Whenever someone says "X tends to be bad", people will show up to say "not all X is bad, only bad X is bad" which is really beside the point.
• earthmancash 4y ago
  This post is right up my alley, as I'm a.) the CEO of an AB experimentation platform www.geteppo.com and b.) I'm an Airbnb alum where we rolled out an analogous feature that labeled a listing as "this is a rare find!" And the funny thing is, I agree with this article. Both the content and the heading of this hackernews article: 1. Notification/scare spam can have long term retention ramifications. The previous generation of experiment platforms made long term metrics literally impossible to read. But now companies can use holdout groups and long term metrics like retention to give more clarity. 2. Even if you can read long term metrics that include retention, the scare/notification spam could lead to less word of mouth growth. For travel, I am guessing that you will be swayed to drive WoM growth more by differentiated inventory, reliable service, and cost, so maybe it's just merely annoying but not a risk to the business. 3. Notice that Airbnb's UI is very, very different from Booking, Expedia. We made a conscious choice to always make sure Airbnb came across as a "sincerely helpful friend" as a booking platform. An AB experiment showing that metrics improve doesn't mean that you have to launch it. You can look at those results and say, "this metric lift isn't worth how ugly it's making my site", and that's a completely valid choice. (a choice we made often at Airbnb)
• zeroonetwothree 4y ago
  Reminds me of this post: https://biggestfish.substack.com/p/data-as-placebo https://biggestfish.substack.com/p/data-as-placebo
• ninkendo 4y ago
  As a rule, I open sites I’ve never been to in a new tab, with my hand hovering over the Cmd and W keys. The moment I get an popup of any kind that obscures content offering me to sign up for a mailing list or offer me some bullshit coupon, my fingers come down and I close the tab. I really hope the A/B tests actually show a bounce here, but I doubt it because I’m actually closing the tab, not using the back button (too many sites hijack it) and I’m not sure they can detect that I’ve left. I also do it so quickly now that they probably don’t even get the signal that I saw the popup either. Site owners: please stop doing this. You’re turning the web into a cesspit. You’re part of the problem.
• slotrans 4y ago
  The way A/B testing is practiced by actual people in the actual world, it is fundamentally broken. No one EVER tests for mean-reversion over time. 17 years I've seen companies do A/B tests. I doubt I've seen a single convincing, durable result the whole time.
• mdip 4y ago
  The "Hotel Tonight" example hit home with me, recently. I used to use that app all the time, then kids happened and spontaneous hotel reservation became rare. Fast forward a few years and a circumstance came up that made me think "Hotel Tonight". I discovered it wasn't installed on my new phone so I grabbed it. It was unrecognizable. Maybe the prices were as good, maybe it could still be used the way I used it previously, but it looked like it turned into a hotel booking app when what I wanted to see was a small selection of good hotels nearby with unusually low prices. One of the features was the lack of choice.