6 ms·

This kind of advice pops up on HN from time to time and while I believe it to be valuable, I guess it is at least incomplete. Here are two stories from my faile

This kind of advice pops up on HN from time to time and while I believe it to be valuable, I guess it is at least incomplete. Here are two stories from my failed ventures to augment this advice: 1) Years ago I started a small side-business with a few colleagues. It was a web-based service for a very specific group of clients. Best of all: The idea had already been validated. We had our first paying customer before we even built the product. And we quickly found a few other customers. But after a few sign-ups things were not looking so good anymore. Even though our early adopters were well known in the industry, other companies would not follow. Everyone we talked to loved the product, but as we had to learn the hard way nobody was willing to pay for it. Some even went so far as to implement possibly illegal solutions instead of paying us a few bucks. Lesson learned: Even if you talk to a few possible customers, question hard if those really are representative of your target audience. Also, liking a product/idea is not the same as being willing (or able) to pay for it. 2) A while back I wanted to find out if there would be any commercial interest in an app I had already built. So I set up a landing page and collected email addresses. Since I had already built the app I could easily hand out demo accounts for every interested party. I quickly collected around 60 addresses and set them all up with a demo account. Only a single person ever logged in at all! I sent out reminder emails but did not get a single response. To this day I do not fully understand what went wrong. Lesson learned: Newsletter- or Beta-signups from your landing page is NOT the same as validation for your idea.

gchokov 12y ago
That's is all well put together, and illustrates my idea in the other comments, why validation is not the silver bullet. Thank you for sharing your experience!
w1ntermute 12y ago
Note that these sorts of problems are not specific to startups. They apply to any sort of polling situation. For example, political campaigns face similar challenges.
mason55 12y ago
> Also, liking a product/idea is not the same as being willing (or able) to pay for it. A great thing to do is when you're validating products you ask the prospect how much they'll pay. Then say "Ok, so if I put a contract in front of you that says you'll pay $x if I deliver this product you will sign it?" You'll find out very quickly whether someone is just being nice or whether they are serious.
- gchokov 12y ago
  It's one thing to say how much you will pay, and different to pull out your wallet and actually pay.
  - enraged_camel 12y ago
    Yes, but asking someone how much they would pay for it takes them almost all the way, mentally, to actually paying for it. You're basically asking them to make a verbal and very informal commitment, but that's enough to make most people think about it seriously (as opposed to saying "omg I love it!!!" just to be nice).
    - wpietri 12y ago
      That's not my experience; I now avoid asking hypotheticals because the results for me have been very unreliable. I'm certainly unreliable that way. There are a number of Kickstarters where I've said, "OMG I'd pay for that!" But if I come back to it a couple of days later, I'll never actually click on the signup button. I'm even worse with things like this in person. I don't like disappointing people, so I'll try hard to find the most positive thing to say about their product. When interviewing, I might ask people a question like, "What's a fair price for X?" But that for me would be more about the follow-up question: "How did you pick that number?" An insight into what they see as equivalent products, related value propositions, or personal value metrics would definitely help me think about pricing.
      - enraged_camel 12y ago
DanielBMarkham 12y ago
Yeah. 1) You are not looking for somebody to "like" your idea. You're looking for something to build a machine around. That usually means "I'm ready to write the check" 2) You are not validating the market, you are validating the machine. As you point out, it's possible to run into a few folks that have checkbooks. Doesn't mean you can find any more of them. The "machine" part is "I go out on the street corner every day and run into 5 new people that will give me money" This is something that can scale. To build that, of course, you have to talk to those first 5 people. For what it's worth, the web is a sucky place to try to get anything done. Everybody is on and they're all trying to automate everything. While the landing page/email thing is great, probably much more useful to physically interact with as many people as possible.
smegmalife 12y ago
>Lesson learned: Newsletter- or Beta-signups from your landing page is NOT the same as validation for you idea. Sure, Beta and Newsletter signups are not full validation for your ideas. But the reality is that you'll never get full validation. For those who are looking to take the plunge and quit their job, or are deciding which project to work on, a simple test like this can be used as a proxy for demand. Just because people click sign up on your landing page doesn't mean your product will succeed. But it is infinitely more indicative than just guessing. If you're going to do a startup, you're going to shoot and miss several times before you succeed. I've been there before. But having Beta signups at least shows you're aiming in the right direction.
wpietri 12y ago
Definitely true. The landing-page thing is the IT equivalent of "is it plugged in?" If you can't get anybody to give you an email address, then you'll never get them to give you money. But just because people are interested doesn't mean they'll pay. In both these cases, I would have taken them as signs that I had some unarticulated, untested hypothesis in my head, and that I'd have to do user interviews to figure out what the issue was. People are weird, and business is extra-weird. Debugging a value chain is often way harder than debugging software.
exelius 12y ago
Sounds like two things here: With #1 you found a problem, but failed to find a market. It's hard to say without knowing the product and/or situation, but it sounds like your product was in the unfortunate position of being an enterprise B2B product that was not solving a problem in the SMB space. The typical strategy for a B2B product is to sell it to SMBs early on, then use that revenue to fund the massive feature set that large enterprises usually need. This is one of the things that happens in startups -- sometimes you don't know your market is a dud until after you have a product. It's not a sure thing. With #2, you just didn't have a large enough sample set. Response rates for e-mails collected via landing pages like this are around 1-2% in my experience. It's a bit of a chicken and egg problem to be honest, and it's hard enough that many startups will throw a marketing budget into it.
ryanSrich 12y ago
> Lesson learned: Newsletter- or Beta-signups from your landing page is NOT the same as validation for your idea. The importance of this comment lies with the false reality of believing at face value that someone loves your product and says they'd use it. Until their money makes it into your bank account, take everything they say with a grain of salt. It's also worth noting that changing the habits of individuals is a massive feat of social engineering. I've seen products that save time and money and yet the market will stick with what they've always done.
jonnathanson 12y ago
The problem with the landing-page-and-email-addresses method is that it's impersonal. It's also a low-commitment activity. It's so easy to sign up for something, and it requires so little conscious thought. You will get a lot of false positives because of that. It's counterintuitive to some extent, but consider a two-phase signup process: a landing page plus an email validation of some kind, or perhaps a two-step signup process on the landing page. There will be a noticeable drop-off between Step 1 and Step 2, and that's good. Those are probably people who would have washed out anyway, and you no longer have to waste time catering to them as false positives. [1] I'm not saying this was necessarily your issue, but a lot of people run into problems with hypothesis testing by conflating it with marketing or growth hacking. It's not. You're not trying to optimize a hypothesis test for pure signup volume. You're trying to use a hypothesis test as a filter: for the right audience, for your problem/solution statement, and for the value proposition you've chosen to test. You don't want signup to be a total pain in the ass, but at the same time, you don't want it to be so easy that it loses meaning as a signal of intent. [1] Caveat being that you still want to probe, and perhaps ask people who've bounced between 1 and 2 why they did so. Sometimes it really is just a UX issue. But in my experience, if someone really has the problem you're addressing, he or she will stick through a two-step signup or validation process.
bane 12y ago
> but as we had to learn the hard way nobody was willing to pay for it. I was part of a startup a few years ago with some really cool technology and good market validation. Somehow we got connected to a very well known media and technology company and were brought in to sell our product to them. The founders, star struck, pulled out all the stops on our presentation, flying people cross country, buying up demo data, the works. They saw the potential install base * our license cost (even at discount) and were thinking immediate exit. During the demo there were oohs and ahhs at all the appropriate places especially around certain features we really wanted to show off. Questions were constant and detailed and follow ups were planned before we even left the room. High fives were given in the parking lot. Over the following weeks calls were vague, "hard to coordinate all these people" "he's on travel" "she's changed departments" Weeks turned into months, calls just stopped being returned. The deal was put into back burner status with lots of finger pointing. One day, a year or so later, one of our employees came back from a tradeshow pointing at a flyer they had picked up from that company's booth. It described a new feature in their technology offering that was an exact copy of some of what we had shown in our demo. The copy even used a term we had coined for the demo. The screenshot looked more or less like that piece of our product, reskined for their technology. Lawyers were consulted. The end result was that they hadn't done anything wrong, just copied an "idea" they had seen. But as a result we were effectively locked out of their market segment. They had simply done the math, develop it in house or pay a license fee per use and it had been cheaper to do it all in house. By buying relevant data and making a demo so complete you could make actual decisions off of it, we had even shown them what they should aim for, what worked, what didn't work. We helped them optimize their development process and made the decision to do it all in house even easier.
- jrmiii 12y ago
  Been there. I think you have to hold some trade secrets back to avoid this fate.

gchokov 12y ago

That's is all well put together, and illustrates my idea in the other comments, why validation is not the silver bullet. Thank you for sharing your experience!

w1ntermute 12y ago

Note that these sorts of problems are not specific to startups. They apply to any sort of polling situation. For example, political campaigns face similar challenges.

mason55 12y ago

> Also, liking a product/idea is not the same as being willing (or able) to pay for it. A great thing to do is when you're validating products you ask the prospect how much they'll pay. Then say "Ok, so if I put a contract in front of you that says you'll pay $x if I deliver this product you will sign it?" You'll find out very quickly whether someone is just being nice or whether they are serious.

gchokov 12y ago
It's one thing to say how much you will pay, and different to pull out your wallet and actually pay.
- enraged_camel 12y ago
  Yes, but asking someone how much they would pay for it takes them almost all the way, mentally, to actually paying for it. You're basically asking them to make a verbal and very informal commitment, but that's enough to make most people think about it seriously (as opposed to saying "omg I love it!!!" just to be nice).
  - wpietri 12y ago
    That's not my experience; I now avoid asking hypotheticals because the results for me have been very unreliable. I'm certainly unreliable that way. There are a number of Kickstarters where I've said, "OMG I'd pay for that!" But if I come back to it a couple of days later, I'll never actually click on the signup button. I'm even worse with things like this in person. I don't like disappointing people, so I'll try hard to find the most positive thing to say about their product. When interviewing, I might ask people a question like, "What's a fair price for X?" But that for me would be more about the follow-up question: "How did you pick that number?" An insight into what they see as equivalent products, related value propositions, or personal value metrics would definitely help me think about pricing.
    - enraged_camel 12y ago

DanielBMarkham 12y ago

Yeah. 1) You are not looking for somebody to "like" your idea. You're looking for something to build a machine around. That usually means "I'm ready to write the check" 2) You are not validating the market, you are validating the machine. As you point out, it's possible to run into a few folks that have checkbooks. Doesn't mean you can find any more of them. The "machine" part is "I go out on the street corner every day and run into 5 new people that will give me money" This is something that can scale. To build that, of course, you have to talk to those first 5 people. For what it's worth, the web is a sucky place to try to get anything done. Everybody is on and they're all trying to automate everything. While the landing page/email thing is great, probably much more useful to physically interact with as many people as possible.

smegmalife 12y ago

>Lesson learned: Newsletter- or Beta-signups from your landing page is NOT the same as validation for you idea. Sure, Beta and Newsletter signups are not full validation for your ideas. But the reality is that you'll never get full validation. For those who are looking to take the plunge and quit their job, or are deciding which project to work on, a simple test like this can be used as a proxy for demand. Just because people click sign up on your landing page doesn't mean your product will succeed. But it is infinitely more indicative than just guessing. If you're going to do a startup, you're going to shoot and miss several times before you succeed. I've been there before. But having Beta signups at least shows you're aiming in the right direction.

wpietri 12y ago

Definitely true. The landing-page thing is the IT equivalent of "is it plugged in?" If you can't get anybody to give you an email address, then you'll never get them to give you money. But just because people are interested doesn't mean they'll pay. In both these cases, I would have taken them as signs that I had some unarticulated, untested hypothesis in my head, and that I'd have to do user interviews to figure out what the issue was. People are weird, and business is extra-weird. Debugging a value chain is often way harder than debugging software.

exelius 12y ago

Sounds like two things here: With #1 you found a problem, but failed to find a market. It's hard to say without knowing the product and/or situation, but it sounds like your product was in the unfortunate position of being an enterprise B2B product that was not solving a problem in the SMB space. The typical strategy for a B2B product is to sell it to SMBs early on, then use that revenue to fund the massive feature set that large enterprises usually need. This is one of the things that happens in startups -- sometimes you don't know your market is a dud until after you have a product. It's not a sure thing. With #2, you just didn't have a large enough sample set. Response rates for e-mails collected via landing pages like this are around 1-2% in my experience. It's a bit of a chicken and egg problem to be honest, and it's hard enough that many startups will throw a marketing budget into it.

ryanSrich 12y ago

> Lesson learned: Newsletter- or Beta-signups from your landing page is NOT the same as validation for your idea. The importance of this comment lies with the false reality of believing at face value that someone loves your product and says they'd use it. Until their money makes it into your bank account, take everything they say with a grain of salt. It's also worth noting that changing the habits of individuals is a massive feat of social engineering. I've seen products that save time and money and yet the market will stick with what they've always done.

jonnathanson 12y ago

The problem with the landing-page-and-email-addresses method is that it's impersonal. It's also a low-commitment activity. It's so easy to sign up for something, and it requires so little conscious thought. You will get a lot of false positives because of that. It's counterintuitive to some extent, but consider a two-phase signup process: a landing page plus an email validation of some kind, or perhaps a two-step signup process on the landing page. There will be a noticeable drop-off between Step 1 and Step 2, and that's good. Those are probably people who would have washed out anyway, and you no longer have to waste time catering to them as false positives. [1] I'm not saying this was necessarily your issue, but a lot of people run into problems with hypothesis testing by conflating it with marketing or growth hacking. It's not. You're not trying to optimize a hypothesis test for pure signup volume. You're trying to use a hypothesis test as a filter: for the right audience, for your problem/solution statement, and for the value proposition you've chosen to test. You don't want signup to be a total pain in the ass, but at the same time, you don't want it to be so easy that it loses meaning as a signal of intent. [1] Caveat being that you still want to probe, and perhaps ask people who've bounced between 1 and 2 why they did so. Sometimes it really is just a UX issue. But in my experience, if someone really has the problem you're addressing, he or she will stick through a two-step signup or validation process.

bane 12y ago

> but as we had to learn the hard way nobody was willing to pay for it. I was part of a startup a few years ago with some really cool technology and good market validation. Somehow we got connected to a very well known media and technology company and were brought in to sell our product to them. The founders, star struck, pulled out all the stops on our presentation, flying people cross country, buying up demo data, the works. They saw the potential install base * our license cost (even at discount) and were thinking immediate exit. During the demo there were oohs and ahhs at all the appropriate places especially around certain features we really wanted to show off. Questions were constant and detailed and follow ups were planned before we even left the room. High fives were given in the parking lot. Over the following weeks calls were vague, "hard to coordinate all these people" "he's on travel" "she's changed departments" Weeks turned into months, calls just stopped being returned. The deal was put into back burner status with lots of finger pointing. One day, a year or so later, one of our employees came back from a tradeshow pointing at a flyer they had picked up from that company's booth. It described a new feature in their technology offering that was an exact copy of some of what we had shown in our demo. The copy even used a term we had coined for the demo. The screenshot looked more or less like that piece of our product, reskined for their technology. Lawyers were consulted. The end result was that they hadn't done anything wrong, just copied an "idea" they had seen. But as a result we were effectively locked out of their market segment. They had simply done the math, develop it in house or pay a license fee per use and it had been cheaper to do it all in house. By buying relevant data and making a demo so complete you could make actual decisions off of it, we had even shown them what they should aim for, what worked, what didn't work. We helped them optimize their development process and made the decision to do it all in house even easier.

jrmiii 12y ago
Been there. I think you have to hold some trade secrets back to avoid this fate.