5 ms·

Here's my notes on DynamoDB: How to spend $100k on what would cost $10k with an sql server for a 100x worse service

boruto 4y ago
I was part of project where we moved user transaction lists from pg to dynamo. While there are pros like ease of scale and all, the biggest was to tell product and higher-ups that the out of place feature with groupbys was simply not possible and there by ending the whole discussion.
atwood22 4y ago
A master carpenter always blames his tools.
- btown 4y ago
  A master carpenter is knowledgeable enough to be critical of those who aggressively peddle tools to unsuspecting customers who are likely to be unfamiliar with just how dangerous and potentially project-destroying those tools can be when not the right tools for the job.
- otterley 4y ago
  That’s not the saying.
icedchai 4y ago
You need to use the right tool for the right job. I know people using DynamoDB for a tiny dataset that would easily fit in sqlite (or any other DB) running on a $20/month VPS. That wouldn't be serverless, of course, so it's a no-go.
- cebert 4y ago
  That same dataset cloud then be modeled and stored in DynamoDB for even leas than that, right?
  - BreakfastB0b 4y ago
    Yeah I have no idea what icedchai is talking about, DynamoDB free tier is super generous https://aws.amazon.com/dynamodb/pricing/on-demand/ https://aws.amazon.com/dynamodb/pricing/on-demand/. It's going to cost you nothing until you have enough customers to afford to pay for it. Correctly modelling single table design on the other hand ...
    - glenngillen 4y ago
      I use it for lots of stuff like this. The pay-per-use/on demand pricing makes it incredibly cheap even if I get occasional bursts of activity. With much better availability than SQLite running on a single VPS.
    - icedchai 4y ago
      1) Latency. 2) Ease of data manipulation. Using Dynamo for a small data set is overkill. You can manipulate the data way faster on a local server, where it is basically in memory (disk cache), and not have to deal with any modelling issues. I guess some people like the DynamoDB API? I find it incredibly awkward.
      - blackoil 4y ago
        You can be even faster if you store data in client. Though different use case different solutions.
- slyall 4y ago
  Not sure what you mean by "tiny dataset" by DynamboDB is great for something with 100 or a few thousand items. Especially if these are only occasionally accessed but need to be shared. Half the time it'll be in the Free Quota or perhaps $1/month. Certainly cheaper than creating an instance.
  - icedchai 4y ago
    I’m basically talking a couple gigabytes of data. Something non-trivial but also doesn’t need a massive distributed DB.
    - arinlen 4y ago
      > I’m basically talking a couple gigabytes of data. You'd be happy to learn that DynamoDB's free tier covers DBS up to 25GB. https://aws.amazon.com/dynamodb/ https://aws.amazon.com/dynamodb/
- Closi 4y ago
  > I know people using DynamoDB for a tiny dataset that would easily fit in sqlite (or any other DB) running on a $20/month VPS. Depending on the use cases, there are plenty of reasons you might want to go down a NoSQL route other than price - schemaless makes it much easier and quicker to hack together new projects for instance (and more fun too!)
- arinlen 4y ago
  > I know people using DynamoDB for a tiny dataset that would easily fit in sqlite (or any other DB) running on a $20/month VPS. I have to say your comment comes off as very ignorant. If you are a AWS customer then you either pick any of the database offerings, such as DynamoDB or Amazon RDS, or run your own database on a EC2 instance. Except running your own db in EC2 can cost around the same as running Amazon RDS, and DynamoDB has a very roomy free tier. Therefore the piece of info you somehow left out is that DynamoDB is free for "a tiny dataset", and you do not have to manage anything at all with DynamoDB too.
  - icedchai 4y ago
    I already know all that. I’ve been using AWS for over 10 years. I’ll just say I prefer the relational model when starting out and leave it there. I’ve had good luck with RDS. I’ve seen people paint themselves into a corner by screwing up their DDB keys too many times and having to export and reload all their data. If you don’t think ahead about your access patterns this is very easy to do. Nobody thinks ahead with “agile.” You’re better off starting with SQL and migrating things to Dynamo where it makes sense.
sass_muffin 4y ago
This is the video I recommend to others when working with dynamodb. The video is by Rick Houlihan about dynamodb modeling. In my experience most developers that complain about dynamodb don't fully understand it. https://www.youtube.com/watch?v=HaEPXoXVf2k https://www.youtube.com/watch?v=HaEPXoXVf2k
- SPBS 4y ago
  DynamoDB can model relational data just fine, if you're okay with setting your query access patterns in stone and never changing them again.
- newlisp 4y ago
  And many developers don't fully understand that he's a good salesman and can't see through the BS.
  - sass_muffin 4y ago
    All technologies have their pros and cons. They have use cases where they make sense and use case where they don't. The job of an engineer to decide which tool fits which use-case. To dismiss a useful technology as "BS", especially one used by companies all over the world for over a decade without any backing data seems a bit disingenuous.
    - newlisp 4y ago
      All technologies have their pros and cons. They have use cases where they make sense and use case where they don't. The job of an engineer to decide which tool fits which use-case. Exactly. But that's not how he paints it, I have seen him bashing RDBMs as been a thing of the past and his promoted way of data modeling and "new" database technology is how companies should start today or be moving to.
osigurdson 4y ago
SQL server seems like an odd choice outside of the enterprise. Suggest running Postgres on Linux.
kumarvvr 4y ago
I think it is more of an issue with not able to effectively model your data to suit the DDB paradigm. DDB absolutely shines when you have to scale. I mean, have you ever tried setting up a cluster of SQL servers. It's a nightmare. DDB is breezingly easy, as long as you know how to model your data effectively.
- hw 4y ago
  Rdms is breezingly easy, as long as you know how to operate your clusters effectively
  - fubbyy 4y ago
    I could be wrong, but at truly large scale RDMS can’t compete, right? SQL simply can’t horizontally scale in the same way?
    - fragmede 4y ago
      Given perfect knowledge of access patterns, I bet you could. Especially since it's basically all reading and not writing. Horizontally scaled with many, many read-only replicas. But then there are lies, damn lies, and benchmarks. All the big companies running huge Oracle db installations are busy running their workloads, which probably don't look like Amazon Prime day traffic. It's also impossible to have perfect knowledge of access patterns.
    - oceanplexian 4y ago
      Sure it can, and I’ve operated MySQL (Percona) at large scale for a social media company. You shard requests by user or something else, doesn’t matter if you have 50 DBs or 50,000. However in most cases you have to write the sharding mechanism yourself, and understand your workload and what such a system can and cannot do.
  - dalyons 4y ago
    even if you know how to do it, running large sharded rdbms clusters is incredibly far from "breezingly easy"
shepherdjerred 4y ago
> In 2021, during the 66-hour Amazon Prime Day shopping event, Amazon systems - including Alexa, the Amazon.com sites, and Amazon fulfill- ment centers, made trillions of API calls to DynamoDB, peak- ing at 89.2 million requests per second, while experiencing high availability with single-digit millisecond performance. Yeah, good luck beating DDB on that one.
- Demiurge 4y ago
  Even better luck building an Amazon scale business after spending 100k on dynamodb.
__alexs 4y ago
At a previous job I moved a giant Postgres server to Dynamo and it was cheaper, faster and had better resiliency. Years later some people moved it back to SQL and made it cost 2x as much... Bad engineering is possible with all technologies.

I was part of project where we moved user transaction lists from pg to dynamo. While there are pros like ease of scale and all, the biggest was to tell product and higher-ups that the out of place feature with groupbys was simply not possible and there by ending the whole discussion.

atwood22 4y ago

A master carpenter always blames his tools.

btown 4y ago
A master carpenter is knowledgeable enough to be critical of those who aggressively peddle tools to unsuspecting customers who are likely to be unfamiliar with just how dangerous and potentially project-destroying those tools can be when not the right tools for the job.
otterley 4y ago
That’s not the saying.

icedchai 4y ago

You need to use the right tool for the right job. I know people using DynamoDB for a tiny dataset that would easily fit in sqlite (or any other DB) running on a $20/month VPS. That wouldn't be serverless, of course, so it's a no-go.

cebert 4y ago
That same dataset cloud then be modeled and stored in DynamoDB for even leas than that, right?
- BreakfastB0b 4y ago
  Yeah I have no idea what icedchai is talking about, DynamoDB free tier is super generous https://aws.amazon.com/dynamodb/pricing/on-demand/ https://aws.amazon.com/dynamodb/pricing/on-demand/. It's going to cost you nothing until you have enough customers to afford to pay for it. Correctly modelling single table design on the other hand ...
  - glenngillen 4y ago
    I use it for lots of stuff like this. The pay-per-use/on demand pricing makes it incredibly cheap even if I get occasional bursts of activity. With much better availability than SQLite running on a single VPS.
  - icedchai 4y ago
    1) Latency. 2) Ease of data manipulation. Using Dynamo for a small data set is overkill. You can manipulate the data way faster on a local server, where it is basically in memory (disk cache), and not have to deal with any modelling issues. I guess some people like the DynamoDB API? I find it incredibly awkward.
    - blackoil 4y ago
      You can be even faster if you store data in client. Though different use case different solutions.
slyall 4y ago
Not sure what you mean by "tiny dataset" by DynamboDB is great for something with 100 or a few thousand items. Especially if these are only occasionally accessed but need to be shared. Half the time it'll be in the Free Quota or perhaps $1/month. Certainly cheaper than creating an instance.
- icedchai 4y ago
  I’m basically talking a couple gigabytes of data. Something non-trivial but also doesn’t need a massive distributed DB.
  - arinlen 4y ago
    > I’m basically talking a couple gigabytes of data. You'd be happy to learn that DynamoDB's free tier covers DBS up to 25GB. https://aws.amazon.com/dynamodb/ https://aws.amazon.com/dynamodb/
Closi 4y ago
> I know people using DynamoDB for a tiny dataset that would easily fit in sqlite (or any other DB) running on a $20/month VPS. Depending on the use cases, there are plenty of reasons you might want to go down a NoSQL route other than price - schemaless makes it much easier and quicker to hack together new projects for instance (and more fun too!)
arinlen 4y ago
> I know people using DynamoDB for a tiny dataset that would easily fit in sqlite (or any other DB) running on a $20/month VPS. I have to say your comment comes off as very ignorant. If you are a AWS customer then you either pick any of the database offerings, such as DynamoDB or Amazon RDS, or run your own database on a EC2 instance. Except running your own db in EC2 can cost around the same as running Amazon RDS, and DynamoDB has a very roomy free tier. Therefore the piece of info you somehow left out is that DynamoDB is free for "a tiny dataset", and you do not have to manage anything at all with DynamoDB too.
- icedchai 4y ago
  I already know all that. I’ve been using AWS for over 10 years. I’ll just say I prefer the relational model when starting out and leave it there. I’ve had good luck with RDS. I’ve seen people paint themselves into a corner by screwing up their DDB keys too many times and having to export and reload all their data. If you don’t think ahead about your access patterns this is very easy to do. Nobody thinks ahead with “agile.” You’re better off starting with SQL and migrating things to Dynamo where it makes sense.

sass_muffin 4y ago

This is the video I recommend to others when working with dynamodb. The video is by Rick Houlihan about dynamodb modeling. In my experience most developers that complain about dynamodb don't fully understand it. https://www.youtube.com/watch?v=HaEPXoXVf2k https://www.youtube.com/watch?v=HaEPXoXVf2k

SPBS 4y ago
DynamoDB can model relational data just fine, if you're okay with setting your query access patterns in stone and never changing them again.
newlisp 4y ago
And many developers don't fully understand that he's a good salesman and can't see through the BS.
- sass_muffin 4y ago
  All technologies have their pros and cons. They have use cases where they make sense and use case where they don't. The job of an engineer to decide which tool fits which use-case. To dismiss a useful technology as "BS", especially one used by companies all over the world for over a decade without any backing data seems a bit disingenuous.
  - newlisp 4y ago
    All technologies have their pros and cons. They have use cases where they make sense and use case where they don't. The job of an engineer to decide which tool fits which use-case. Exactly. But that's not how he paints it, I have seen him bashing RDBMs as been a thing of the past and his promoted way of data modeling and "new" database technology is how companies should start today or be moving to.

osigurdson 4y ago

SQL server seems like an odd choice outside of the enterprise. Suggest running Postgres on Linux.

kumarvvr 4y ago

I think it is more of an issue with not able to effectively model your data to suit the DDB paradigm. DDB absolutely shines when you have to scale. I mean, have you ever tried setting up a cluster of SQL servers. It's a nightmare. DDB is breezingly easy, as long as you know how to model your data effectively.

hw 4y ago
Rdms is breezingly easy, as long as you know how to operate your clusters effectively
- fubbyy 4y ago
  I could be wrong, but at truly large scale RDMS can’t compete, right? SQL simply can’t horizontally scale in the same way?
  - fragmede 4y ago
    Given perfect knowledge of access patterns, I bet you could. Especially since it's basically all reading and not writing. Horizontally scaled with many, many read-only replicas. But then there are lies, damn lies, and benchmarks. All the big companies running huge Oracle db installations are busy running their workloads, which probably don't look like Amazon Prime day traffic. It's also impossible to have perfect knowledge of access patterns.
  - oceanplexian 4y ago
    Sure it can, and I’ve operated MySQL (Percona) at large scale for a social media company. You shard requests by user or something else, doesn’t matter if you have 50 DBs or 50,000. However in most cases you have to write the sharding mechanism yourself, and understand your workload and what such a system can and cannot do.
- dalyons 4y ago
  even if you know how to do it, running large sharded rdbms clusters is incredibly far from "breezingly easy"

shepherdjerred 4y ago

> In 2021, during the 66-hour Amazon Prime Day shopping event, Amazon systems - including Alexa, the Amazon.com sites, and Amazon fulfill- ment centers, made trillions of API calls to DynamoDB, peak- ing at 89.2 million requests per second, while experiencing high availability with single-digit millisecond performance. Yeah, good luck beating DDB on that one.

Demiurge 4y ago
Even better luck building an Amazon scale business after spending 100k on dynamodb.

__alexs 4y ago

At a previous job I moved a giant Postgres server to Dynamo and it was cheaper, faster and had better resiliency. Years later some people moved it back to SQL and made it cost 2x as much... Bad engineering is possible with all technologies.