7 ms·

How much of this is a problem in practice? I think in practice, S3 data is often indexed using other DBs e.g DynamoDB, Postgres, MySQL etc. Can't this index be

How much of this is a problem in practice? I think in practice, S3 data is often indexed using other DBs e.g DynamoDB, Postgres, MySQL etc. Can't this index be used to enumerate all S3 URLs? I am off-course simplifying this a lot.

Manfred 5y ago
Developers are pretty sloppy with maintaining references to object in my experience. It generally only becomes a problem when you need to clean up and that usually at time of IPO when there are petabytes of data in S3.
onion2k 5y ago
How much of this is a problem in practice? This specific issue probably isn't a very big problem. The issue of Amazon repeatedly coming up on HN as a service that will bill you when you're not unexpecting it for things that are moderately hard to understand and might refund you later probably costs them tens or even hundreds of millions in lost revenue every year from developers being cautious about deploying things to their services.
- jasonlotito 5y ago
  > probably costs them tens or even hundreds of millions in lost revenue every year from developers being cautious about deploying things to their services I see this as a good thing. They literally encourage you to be cautious with your pricing and resource usage, to the point where they put limits on what resources you can use without explicitly asking for more. Developers should be cautious and aware.
  - ghaff 5y ago
    I'm not sure it's a good thing. But open-ended charges come up a lot. AWS, like the other big public cloud providers, doubtless have self-interested reasons for letting people run up the meter. But they're probably also hesitant to add cut-offs that could end up being footguns in a production context to bring down services in complex infrastructures. People just playing around should be careful. There are ways to keep risk lower. But if absolute price caps are your priority you should probably be using a VPS of some sort.
- devnull3 5y ago
  > things that are moderately hard to understand My experience with AWS is that the pricing for each service is reasonably well documented and the calculator does descent job. The problem starts when multiple combinations of services are used and it becomes harder to reason. With the advent of cloud, cost-modelling becomes an essential skill (which can be learned). One needs to be clear about total work that gets done and "how" that work gets processed. This in turn should translate to relevant cost metric (e.g PUT requests/s for S3 or IOPS for DynamoDB, amount of data scanned for Athena, etc) This needs to be evaluated for zero load, normal load, 5x load, 20x load, etc. Zero load gives what is dead weight cost of the system i.e cost incurred when no work is being done (e.g EC2, EBS volumes, etc)

Manfred 5y ago

Developers are pretty sloppy with maintaining references to object in my experience. It generally only becomes a problem when you need to clean up and that usually at time of IPO when there are petabytes of data in S3.

onion2k 5y ago

How much of this is a problem in practice? This specific issue probably isn't a very big problem. The issue of Amazon repeatedly coming up on HN as a service that will bill you when you're not unexpecting it for things that are moderately hard to understand and might refund you later probably costs them tens or even hundreds of millions in lost revenue every year from developers being cautious about deploying things to their services.

jasonlotito 5y ago
> probably costs them tens or even hundreds of millions in lost revenue every year from developers being cautious about deploying things to their services I see this as a good thing. They literally encourage you to be cautious with your pricing and resource usage, to the point where they put limits on what resources you can use without explicitly asking for more. Developers should be cautious and aware.
- ghaff 5y ago
  I'm not sure it's a good thing. But open-ended charges come up a lot. AWS, like the other big public cloud providers, doubtless have self-interested reasons for letting people run up the meter. But they're probably also hesitant to add cut-offs that could end up being footguns in a production context to bring down services in complex infrastructures. People just playing around should be careful. There are ways to keep risk lower. But if absolute price caps are your priority you should probably be using a VPS of some sort.
devnull3 5y ago
> things that are moderately hard to understand My experience with AWS is that the pricing for each service is reasonably well documented and the calculator does descent job. The problem starts when multiple combinations of services are used and it becomes harder to reason. With the advent of cloud, cost-modelling becomes an essential skill (which can be learned). One needs to be clear about total work that gets done and "how" that work gets processed. This in turn should translate to relevant cost metric (e.g PUT requests/s for S3 or IOPS for DynamoDB, amount of data scanned for Athena, etc) This needs to be evaluated for zero load, normal load, 5x load, 20x load, etc. Zero load gives what is dead weight cost of the system i.e cost incurred when no work is being done (e.g EC2, EBS volumes, etc)