4 ms·
S3 is basically a database. You can store everything there and interact with it via SQL with Athena, for example. I could see utilizing all of Amazon's "infinit
by lambda_obrien 6y ago
S3 is basically a database. You can store everything there and interact with it via SQL with Athena, for example. I could see utilizing all of Amazon's "infinitely scalable" services to build very simple and powerful software really easily.
- barbazoo 6y agoI think Athena has to get a lot less expensive for that to become true.
- kryptk 6y agoFor small numbers of large queries, Athena has been an incredible technology stack .. I pay pennies a month to run my analytics jobs on it (aggregates across a few hundred million rows, joins with tens of millions of rows into hundreds of thousands of rows). I start my data as .csv.gz but the first step is a CTAS to extract columns and convert to compressed parquet. This step basically costs the most but gives a 10x data size reduction to downstream steps. Athena does not work at all if you perform large numbers of small indexed read queries, definitely use a traditional database for that.
- lambda_obrien 5y agoYes, it's great if you setup the data structures properly, we pay a few bucks a month to run a lot of infra on it to query telemetry data over a few hours to a few days at a time. I think an equivalent cost in just a database server instance to run the same load would be at least 25 to 50 a month. You're not really locked into athena, either, it just takes standard sql, so you could easily transition to a dedicated box later.