5 ms·

I have worked at two startups now where we made the fatal mistake of being profitable. If you make this mistake then the investors will swoop in and demand you

I have worked at two startups now where we made the fatal mistake of being profitable. If you make this mistake then the investors will swoop in and demand you spend more on marketing and AWS infrastructure, because we're scaling up to 5 billion users of course. Of course we started spending all the money on new people and AWS, and soon there was no money. At one point we were dumping like $15K a month on AWS for a dozen unnecessary over-engineered toys that nobody was using. This is the real cost of AWS. I'd love to see Amazon's data on money invested vs actual user traffic for small startups, that's got to be some of the most interesting and valuable data on earth. Forget companies, I'll bet Jeff is sitting around predicting when entire industries rise and fall weeks before anyone else based just on this data.

grogenaut 7y ago
How is Amazon supposed to know actual user numbers? That's app specific.
- deleted 7y ago
  [deleted]
- fouc 7y ago
  They probably meant page views and such.
- biesnecker 7y ago
  You can't get exact numbers, but for consumer-facing products you can estimate it via unique IPs hitting the load balancers, etc.
  - grogenaut 7y ago
    That would require amazon gathering customer information and possibly GDPR data on the usage of all of it's customers, which is a thing amazon doesn't do, and you don't want your cloud providers doing.
tyingq 7y ago
On the other side of the spectrum, AWS's extensive cost report metrics via tagging are great for big companies. I can now show exactly which departments and dev teams are driving all the costs, and on what (CPU, storage, network). In a way that I never could for on-prem stuff.
- dillonmckay 7y ago
  Can you elaborate on that or do you have any best-practices for tagging and correlating costs?
  - tyingq 7y ago
    We use a commercial tool. But, the most important tags are environment (dev, test, prod), application name, app version, and owning team. For some apps, perhaps a "component" or "service" tag would also be important.
- joncrane 7y ago
  ...sure, as long as they tag their resources properly. The closest I got to an org that did this well was a big company that ran Cloud Custodian in all their AWS accounts and if you launched an EC2 instance, it would terminate it immediately with extreme prejudice if it didn't have values for three required tags, one to identify the "owner" individually and two for accounting purposes. The only problem with that is there's no mechanism to make sure that the values of the cost centers values were correct. There was a bit of a scandal when one group (who presumably just copied and pasted a bunch of CloudFormation from another group's repo) was running 5 figures a months of infrastructure under the other group's billing codes. ALSO, as many have said, bandwidth is a big part of the cost, and at this time it's nearly impossible to do showback/chargeback on bandwidth. There may be a way to do it using Flow Logs by correlating IP addresses to instances and using those tags, but I've never heard of someone doing this successfully.
  - tyingq 7y ago
    Egress charges leverage tags now. You can get down to good detail. Here's an image showing it will use tags if you set them: https://blog.cloudability.com/wp-content/uploads/2017/02/data-transfer-costs-4-768x167.png https://blog.cloudability.com/wp-content/uploads/2017/02/dat... In this case, a service tag, set in some cases, not in others.
matwood 7y ago
AWS or not, investors invest for the sole purpose of spending money with the hope of finding a unicorn. If an investor just wants to sit on their money, they can do that without giving it to a company.
- icxa 7y ago
  Then those investors are stupid. Hope is not a strategy. But more likely is this take is wrong. Also sounds to me like GP maybe missed another consideration: architecting your application in such a manner that scaling is straightforward (not easy, straightforward, there is a difference). Using AWS as an example, at one of the businesses I worked at, we used Kinesis as an event bus. One shard handles 2MB/sec of output. This worked pretty well for thousands of messages a second, we even got up to the 100's of thousands by compressing the payload of the event message. After that, you can employ any number of strategies that work easily, such as shading and adding additional streams, and use a lambda to pipe the output of one stream to another. It scaled up to millions of messages by essentially pushing buttons in the AWS Console. Take a look at your architecture. More than likely, outside of a FB/Google/Netflix traffic scenario there is probably an easier and more straightforward architecture you can use that scales to your realistic use cases. Worry about the billions when you get there, which you yourself probably wont at that point because you would have exited, or moved into a higher role most likely by now.
pbhjpbhj 7y ago
5 billion users? That's substantially more than Facebook .. was that an error?
- archgoon 7y ago
  Most likely that was an exaggeration of the demands made by the investors.
- manigandham 7y ago
  That was a joke.
dillonmckay 7y ago
What is the rule of thumb for cloud-spending costs of production versus development/CI? Somebody mentioned 1:3.
geggam 7y ago
Only 15k a month ? Startup I was at burned 400k/month average sometimes when buying RI we spent 800k. Had a couple engineers route the database connections over the public load balancers for a month, that cost 20k in network alone. AWS is not cheap at scale, period.
- aasasd 7y ago
  Could you please slip me in the bills at your next place? I'm currently spending less that 500 bucks a month.
- Dunedan 7y ago
  A startup being able to spend 400k/month on AWS should also be able to employ somebody to keep an eye on the AWS cost and look for possible optimizations. If that's not the case I wouldn't blame AWS for the spent money.
  - geggam 7y ago
    No blame was on AWS. The TAMs tried to help fix things but no one was listening. Many times I was the only person meeting the 2-3 TAMs we had on the account. I have many stories about this startup. The idea is great. If the startup succeeds it will be in spite of how its being managed technically
- meritt 7y ago
  What was the monthly revenue for this startup? Those are absolutely insane numbers.
  - geggam 7y ago
    Not sure what the monthly revenue was but I do know there were several multimillion dollar deals every 6 months or so. Problems come from many areas... like the fact it was in the data management space and the 1500 or so i3's running Cassandra plus the hybrid cloud approach of front end in AWS talking to APIs in GKE which talked to backends in AWS because cool technologies you know. No architecting was done. Build this and put it in the cloud. The team from RU didn't even know about autoscaling groups and when I tried to bring them in I ran into resistance. I have many examples of "opportunities to optimize" from this place, needles to say my stress level is much lower not being there.
atmosx 7y ago
If you are profitable, why accept investor money anyway?
- cheerlessbog 7y ago
  Grow faster?
  - atmosx 7y ago
    Well then, be careful next time.
mgkimsal 7y ago
> If you make this mistake then the investors will swoop in and demand you spend more on marketing and AWS infrastructure Would there be any "conflict of interest" issues is some of the same people requiring AWS spending were also Amazon shareholders?
- oldjokes 7y ago
  Dude. A bunch of billionaires just got away with completely tanking the economy and getting bailed out for it. Zero people ever even talked about the possibility of anyone going to jail. Everyone got their bonuses. Nobody suffered any consequences. Literally nobody. If you think anyone cares about any conflict of interest among the investing class you're beyond naive, you're just delusional.
b3b0p 7y ago
> I have worked at two startups now where we made the fatal mistake of being profitable. I'm laughing because I work at a currently profitable bootstrapped startup. Luckily, we don't have investors and there are only a few of us and no plans to hire engineers at this time. I'm feeling like this comment is ripe for a top post over at /r/programmingcirclejerk. Thanks for the laugh and I would have never thought an investor would want this.
mahmoudimus 7y ago
If you're taking money from investors and you're interested in being profitable, then you're going to sacrifice growth for the sake of profitability: - Let's not hire for this idea because it eats into our burn - Hmm, let's hold off on launching this feature until we have more data Investment $ means taking risks (within reason) to maximize shareholder value. If the company you worked for was profitable, then they could've structured a leveraged debt payoff to the investors to get them off the cap-table. Unless the company took so much money that the investors owned 60%+ and they unanimously do not agree about being profitable, then this is something that can be passed as a board resolution. It sounds like the founders at your company were just inexperienced.

grogenaut 7y ago

How is Amazon supposed to know actual user numbers? That's app specific.

deleted 7y ago
[deleted]
fouc 7y ago
They probably meant page views and such.
biesnecker 7y ago
You can't get exact numbers, but for consumer-facing products you can estimate it via unique IPs hitting the load balancers, etc.
- grogenaut 7y ago
  That would require amazon gathering customer information and possibly GDPR data on the usage of all of it's customers, which is a thing amazon doesn't do, and you don't want your cloud providers doing.

tyingq 7y ago

On the other side of the spectrum, AWS's extensive cost report metrics via tagging are great for big companies. I can now show exactly which departments and dev teams are driving all the costs, and on what (CPU, storage, network). In a way that I never could for on-prem stuff.

dillonmckay 7y ago
Can you elaborate on that or do you have any best-practices for tagging and correlating costs?
- tyingq 7y ago
  We use a commercial tool. But, the most important tags are environment (dev, test, prod), application name, app version, and owning team. For some apps, perhaps a "component" or "service" tag would also be important.
joncrane 7y ago
...sure, as long as they tag their resources properly. The closest I got to an org that did this well was a big company that ran Cloud Custodian in all their AWS accounts and if you launched an EC2 instance, it would terminate it immediately with extreme prejudice if it didn't have values for three required tags, one to identify the "owner" individually and two for accounting purposes. The only problem with that is there's no mechanism to make sure that the values of the cost centers values were correct. There was a bit of a scandal when one group (who presumably just copied and pasted a bunch of CloudFormation from another group's repo) was running 5 figures a months of infrastructure under the other group's billing codes. ALSO, as many have said, bandwidth is a big part of the cost, and at this time it's nearly impossible to do showback/chargeback on bandwidth. There may be a way to do it using Flow Logs by correlating IP addresses to instances and using those tags, but I've never heard of someone doing this successfully.
- tyingq 7y ago
  Egress charges leverage tags now. You can get down to good detail. Here's an image showing it will use tags if you set them: https://blog.cloudability.com/wp-content/uploads/2017/02/data-transfer-costs-4-768x167.png https://blog.cloudability.com/wp-content/uploads/2017/02/dat... In this case, a service tag, set in some cases, not in others.

matwood 7y ago

AWS or not, investors invest for the sole purpose of spending money with the hope of finding a unicorn. If an investor just wants to sit on their money, they can do that without giving it to a company.

icxa 7y ago
Then those investors are stupid. Hope is not a strategy. But more likely is this take is wrong. Also sounds to me like GP maybe missed another consideration: architecting your application in such a manner that scaling is straightforward (not easy, straightforward, there is a difference). Using AWS as an example, at one of the businesses I worked at, we used Kinesis as an event bus. One shard handles 2MB/sec of output. This worked pretty well for thousands of messages a second, we even got up to the 100's of thousands by compressing the payload of the event message. After that, you can employ any number of strategies that work easily, such as shading and adding additional streams, and use a lambda to pipe the output of one stream to another. It scaled up to millions of messages by essentially pushing buttons in the AWS Console. Take a look at your architecture. More than likely, outside of a FB/Google/Netflix traffic scenario there is probably an easier and more straightforward architecture you can use that scales to your realistic use cases. Worry about the billions when you get there, which you yourself probably wont at that point because you would have exited, or moved into a higher role most likely by now.

pbhjpbhj 7y ago

5 billion users? That's substantially more than Facebook .. was that an error?

archgoon 7y ago
Most likely that was an exaggeration of the demands made by the investors.
manigandham 7y ago
That was a joke.

dillonmckay 7y ago

What is the rule of thumb for cloud-spending costs of production versus development/CI? Somebody mentioned 1:3.

geggam 7y ago

Only 15k a month ? Startup I was at burned 400k/month average sometimes when buying RI we spent 800k. Had a couple engineers route the database connections over the public load balancers for a month, that cost 20k in network alone. AWS is not cheap at scale, period.

aasasd 7y ago
Could you please slip me in the bills at your next place? I'm currently spending less that 500 bucks a month.
Dunedan 7y ago
A startup being able to spend 400k/month on AWS should also be able to employ somebody to keep an eye on the AWS cost and look for possible optimizations. If that's not the case I wouldn't blame AWS for the spent money.
- geggam 7y ago
  No blame was on AWS. The TAMs tried to help fix things but no one was listening. Many times I was the only person meeting the 2-3 TAMs we had on the account. I have many stories about this startup. The idea is great. If the startup succeeds it will be in spite of how its being managed technically
meritt 7y ago
What was the monthly revenue for this startup? Those are absolutely insane numbers.
- geggam 7y ago
  Not sure what the monthly revenue was but I do know there were several multimillion dollar deals every 6 months or so. Problems come from many areas... like the fact it was in the data management space and the 1500 or so i3's running Cassandra plus the hybrid cloud approach of front end in AWS talking to APIs in GKE which talked to backends in AWS because cool technologies you know. No architecting was done. Build this and put it in the cloud. The team from RU didn't even know about autoscaling groups and when I tried to bring them in I ran into resistance. I have many examples of "opportunities to optimize" from this place, needles to say my stress level is much lower not being there.

atmosx 7y ago

If you are profitable, why accept investor money anyway?

cheerlessbog 7y ago
Grow faster?
- atmosx 7y ago
  Well then, be careful next time.

mgkimsal 7y ago

> If you make this mistake then the investors will swoop in and demand you spend more on marketing and AWS infrastructure Would there be any "conflict of interest" issues is some of the same people requiring AWS spending were also Amazon shareholders?

oldjokes 7y ago
Dude. A bunch of billionaires just got away with completely tanking the economy and getting bailed out for it. Zero people ever even talked about the possibility of anyone going to jail. Everyone got their bonuses. Nobody suffered any consequences. Literally nobody. If you think anyone cares about any conflict of interest among the investing class you're beyond naive, you're just delusional.

b3b0p 7y ago

> I have worked at two startups now where we made the fatal mistake of being profitable. I'm laughing because I work at a currently profitable bootstrapped startup. Luckily, we don't have investors and there are only a few of us and no plans to hire engineers at this time. I'm feeling like this comment is ripe for a top post over at /r/programmingcirclejerk. Thanks for the laugh and I would have never thought an investor would want this.

mahmoudimus 7y ago

If you're taking money from investors and you're interested in being profitable, then you're going to sacrifice growth for the sake of profitability: - Let's not hire for this idea because it eats into our burn - Hmm, let's hold off on launching this feature until we have more data Investment $ means taking risks (within reason) to maximize shareholder value. If the company you worked for was profitable, then they could've structured a leveraged debt payoff to the investors to get them off the cap-table. Unless the company took so much money that the investors owned 60%+ and they unanimously do not agree about being profitable, then this is something that can be passed as a board resolution. It sounds like the founders at your company were just inexperienced.