6 ms·
Amazon DynamoDB Transactions
- piinbinary 8y agoMy wishlist for DynamoDB is now down to: * Fast one-time data import without permanently creating a lot of shards (important if you are restoring from a backup) * Better visibility into what causes throttling (e.g. was it a hot shard? Was it a brief but large burst of traffic?) * Lower p99.9 latency. It occasionally has huge latency spikes. * Indexes of more than 2 columns * A solution for streaming out updates that is better than dynamodb streams
- monstrado 8y agoAlso, better insight into partition sizes / what's causing hot spotting. The DB abstracts a lot from the user, which isn't necessarily great, because it's still subject to the normal pitfalls of a NoSQL database.
- tejasmanohar 8y agoBigtable is a different beast, but it's new "Key Visualizer" is impressive. Has helped us quickly find anomalies https://cloud.google.com/bigtable/docs/keyvis-overview https://cloud.google.com/bigtable/docs/keyvis-overview Wish Dynamo had something similar
- darkr 8y agoNot a particularly easy solution, but you can use dynamo streams to achieve this by loading fast into a temporary table, trickle-feeding via a stream into another table. When it’s caught up, stop writes on the import table then swap over to the permanent table. A way of doing this without expending all that effort is oh my wish list too.
- Rafuino 8y agoWhat kind of p99.9 latency are you looking for?
- anentropic 8y agoand would Dax help?
- antonp 8y ago> * A solution for streaming out updates that is better than dynamodb streams What bothers you about dynamodb streams specifically?
- jedberg 8y agoThis is cool, it lifts the burden of having to bake "atomicity" into your app if you're using a key/value store like DynamoDB. I can see a nice balance of combining this with some built in error checking in the app itself.
- Thaxll 8y agoIs the heat map available to customers now or is it still a request you have to do?
- darkr 8y agoStill have to request it AFAIAA
- jchrisa 8y agoCongrats to the DynamoDB team for going beyond the traditional limits of NoSQL. There is a new breed of databases that use consensus algorithms to enable global multi-region consistency. Google Spanner and FaunaDB where I work are part of this group. I didn’t catch anything about the implementation details of DynamoDB transactions in the article. If they are using a consensus approach, expect them to add multi-region consistency soon. If they are using a traditional active/active replication approach, they’ll be limited to regional replication.
- erik_seaberg 8y agoThey warn about other regions seeing incomplete transactions (if you opt into transactions on global tables), which fits with the current "copy each new item from the stream" async replication.
- ryanworl 8y ago“DynamoDB is the only non-relational database that supports transactions across multiple partitions and tables.” Uh... this is just not true.
- otterley 8y agoCan you identify some others?
- monstrado 8y agoFoundationDB https://www.foundationdb.org/ https://www.foundationdb.org/ not only supports transactions, they are mandatory. They also go one step further and support atomic operations, which are especially killer.
- otterley 8y agoI don’t think FDB supports cross-database transactions, though.
- ryanworl 8y agoWhat do you mean by this? There is only one “database” in FoundationDB terms. You can write transactions over the entire keyspace regardless of which machine the data is stored on.
- otterley 8y agoMultiple clusters, then, or whatever you specify to FDB’s API to identify the instance when making a client connection.
- ryanworl 8y agoI'm still not sure what you mean in terms of contrasting this with DynamoDB's new features. You could implement the entire DynamoDB API, with even stronger semantics than the new features listed in the article, on top of FoundationDB. Additionally, the latency would be theoretically lower as they describe needing to do a read, write, and another read per key to verify isolation, whereas FoundationDB uses an optimistic concurrency control scheme to verify at commit time that transactions do not conflict. In the common case (where transactions don't conflict) this is faster.
- abalone 8y agoAny experience with using Aurora in place of DynamoDB? A couple years ago there was an interesting tidbit at re:Invent about customers moving from DynamoDB to Aurora to save significant costs.[1] The Aurora team made the point that DynamoDB suffers from hotspots despite your best efforts to evenly distribute keys, so you end up overprovisioning. Whereas with Aurora you just pay for I/O. And the scalability is great. Plus you get other nice stuff with Aurora like, you know, traditional SQL multi-operation transactions. It was kind of buried in a preso from the Aurora team and the high-level messaging from Amazon was still, NoSQL is the most scalable thing. Aurora was and is still seemingly positioned against other solutions within the SQL realm. I sort of get it in theory that NoSQL is still theoretically infinitely scalable whereas Aurora is bounded by 15 read replicas and one write master.. but in practice these days those limits are huge. I think one write master can handle like 100K transactions a second or something. So, I'm really curious where this has gone in the past couple years if anywhere. Is NoSQL still the best approach? [1] https://youtu.be/60QumD2QsF0?t=1021 https://youtu.be/60QumD2QsF0?t=1021
- manigandham 8y agoYes. Relational databases are very fast and using them as key/value stores is a great use-case. Using a scale-out system like Aurora makes it even better. It's slower because of SQL parsing and generally the SQL clients are not as fast, but you can get close to single-digit millisecond latency these days. We use Aurora or Postgres for key/value unless we need something specific, like multi-regional capacity or really high-end performance. For that we run ScyllaDB.
- ngrilly 8y ago> It's slower because of SQL parsing and generally the SQL clients are not as fast I'd be really surprised if the client library introduces a latency significant enough to be compared to the network latency between the app server and the database server.
- manigandham 8y agoMany libraries handle db connections poorly, or have heavy-handled pooling systems, or aren't fully async, all of which limits total throughput. The key/value clients usually have a much simpler APIs like HTTP which scale much better.
- jared2501 8y agoI'd be interested to see comparisons/benchmarks against FoundationDB. DynamoDB transactions make dynamo a serious alternative to FDB now. I can see the two manage advantages for FDB being: 1) you can deploy it on premise (which is potentially important for some B2B companies), 2) it shuffles data around so that hot-spotting of a cluster is eliminated (which dynamo appears to still suffer from).
- wahnfrieden 8y ago#2 appears to be solved now https://aws.amazon.com/blogs/database/how-amazon-dynamodb-adaptive-capacity-accommodates-uneven-data-access-patterns-or-why-what-you-know-about-dynamodb-might-be-outdated/ https://aws.amazon.com/blogs/database/how-amazon-dynamodb-ad...
- polskibus 8y agoFoundation DB is open source!
- psankar 8y agoPostgresql gets native JSON support (at least since 9.2 onwards) to store schemaless free flowing text. Dynamodb gets transaction guarantees. There is globalization and intermingling happening on technology too. On a similar thought, a few years back, C# and Java got `Any` generic types, while Python/JS got static types (via python3 typings, typescript)
- lozenge 8y agoC# doesn't have an Any generic type (Foo<?> In java parlance)
- manigandham 8y agoC# can use `object` or `dynamic`: https://stackoverflow.com/questions/2690623/what-is-the-dynamic-type-in-c-sharp-4-0-used-for https://stackoverflow.com/questions/2690623/what-is-the-dyna...
- asien 8y ago>If an item is modified outside of a transaction while the transaction is in progress, the transaction is canceled and an exception is thrown You are still responsible to implements a Queue or a Lock on the Items you want to mutate. That said this is a huge milestone for DynamoDB, we can now safely mutate multiples items while remaining ACID.
- grogers 8y agoMax 10 items per transaction, that's quite a restriction! I guess you have to plan all the transactions you would perform and make sure they meet the bounds.