7 ms·

This is a very expensive way to do leader election, at least from an infrastructure perspective. This is because you're essentially pushing the problem down to

by mgdev 2y ago

This is a very expensive way to do leader election, at least from an infrastructure perspective. This is because you're essentially pushing the problem down to S3, which does its own leader election in a way that is waaaay overbuilt for what we're trying to accomplish here. But... that doesn't mean it isn't cool. :)

otterley 2y ago
Since the user is only paying an infinitesimal fraction of the infrastructure cost, does it really matter? From the user’s perspective it’s extremely inexpensive.
- mgdev 2y ago
  If everyone started using S3 for this purpose, they would shut it down pretty quickly. Small objects (especially small hot objects) are actually problematic in S3. They cache them in the keymap instead of retrieving from block storage. But many small objects can quickly blow out a keymap shard. The keymap is designed for high-performance lookups. Because of this, it's also more expensive to scale out this class of 'storage' at the keymap layer than to scale out block storage. And you still have to do quorum writes to block storage and then cache eviction at the keymap. If you're doing this for slow-moving leader election in a small cluster, fine. But if, for example, you started using this for leader-election among end-user-defined workloads that could scale up/down, you might find yourself on the other side of a call from AWS.
  - otterley 2y ago
    > If everyone started using S3 for this purpose, they would shut it down pretty quickly I work at AWS but not on the S3 service team. (Opinions are entirely my own.) I have little doubt that the team already considered this possibility before releasing the feature. The choice to make new functionality available in a service is considered a “one-way door” that receives a tremendous amount of scrutiny. Once a feature is made available, they will do everything they can to support it as long as possible, even at the cost of convenience to themselves. And even de-emphasized services (keep the lights on) are rarely deprecated completely - existing accounts and customers already using them can frequently continue to do so.
    - mgdev 2y ago
      They introduce economic incentives. :) By way of example, a little over a decade ago a famous online streaming company used to upload all of their video masters for transcoding. This process involved a huge upload and a huge workload, followed by a significant drop in activity. The problem was that AWS had to provision for peak usage instead of average usage. This resulted in a situation where the peak-to-average ratio was very high for just one or two customers. To address this issue, the solution was to incentivize these customers to spread out their workload more evenly over time, at least until they were no longer the largest driver of peak/avg. This is also why things like Reserved Instances and Spot Instances exist.
      - otterley 2y ago
        I’m sure we agree that other means such as changing pricing and reaching out to exceptional customers are not the same thing as killing a feature.
        mgdev 2y ago
        I never said they would kill the feature. I said they would shut down the behavior. They can do that through economic incentive. (Source: I was on the S3 team. Opinions my own, etc.)
        otterley 2y ago
        You said they would “shut it down,” which can reasonably be interpreted as killing the feature. If you meant something more specific, you should have said that. Clarity is the duty of the speaker.
        mgdev 2y ago
        You must be fun at stand-ups.
        Terretta 2y ago
        >> “If everyone started using S3 for this purpose, they would shut it down pretty quickly.” > “I never said they would kill the feature. I said they would shut down the behavior.” Seems "it" is standing in for a lot of nuance here. With these three sentences side by side, still took a while to see how you might be referring to the "everyone using" behavior instead of the "S3 purpose" feature! Usually given ambiguity the nearest plausible reference wins. Since in tech "shut it down" is more often systems than humans and that was the nearest reference "it" could refer to, took some doing to see how your assertion "I said the behavior" could be accurate!
paulgb 2y ago
One advantage of this approach is that if you're already using S3 (as in the SlateDB case mentioned in the article), it's essentially “free”. And it means that a package like SlateDB just needs to be pointed at an S3 bucket, instead of making you point it to an S3 bucket _and_ a DynamoDB instance.
- mgdev 2y ago
  Totally. I'm not advocating that people don't ever use it for this. I'm just saying that from a pure resource perspective, it might be one of the least resource-efficient mechanisms for doing this. As others have pointed out, it's probably not a noticeable cost and, in fact, the fixed costs associated with setting something up yourself would far outweigh what you're paying to use S3 for this purpose. Part of me just dies inside when I think of all the stuff needlessly happening behind the scenes, given it's not actually being used for storage. I mean, it's called Simple Storage Service.
  - mgdev 2y ago
    I'll add that if a lot of people actually did start using it for this purpose, they could probably just productionize the thing they actually use for this, which is essentially their own version of etcd, but with Paxos instead of Raft.
- kortex 2y ago
  Provisioning a DynamoDB table is scarcely more effort than provisioning a bucket. And you get way nicer constructs (still not as nice as RDBMS) for locking like ConditionalUpdate and TransactWrite.

by mgdev 2y ago

otterley 2y ago

Since the user is only paying an infinitesimal fraction of the infrastructure cost, does it really matter? From the user’s perspective it’s extremely inexpensive.

mgdev 2y ago
If everyone started using S3 for this purpose, they would shut it down pretty quickly. Small objects (especially small hot objects) are actually problematic in S3. They cache them in the keymap instead of retrieving from block storage. But many small objects can quickly blow out a keymap shard. The keymap is designed for high-performance lookups. Because of this, it's also more expensive to scale out this class of 'storage' at the keymap layer than to scale out block storage. And you still have to do quorum writes to block storage and then cache eviction at the keymap. If you're doing this for slow-moving leader election in a small cluster, fine. But if, for example, you started using this for leader-election among end-user-defined workloads that could scale up/down, you might find yourself on the other side of a call from AWS.
- otterley 2y ago
  > If everyone started using S3 for this purpose, they would shut it down pretty quickly I work at AWS but not on the S3 service team. (Opinions are entirely my own.) I have little doubt that the team already considered this possibility before releasing the feature. The choice to make new functionality available in a service is considered a “one-way door” that receives a tremendous amount of scrutiny. Once a feature is made available, they will do everything they can to support it as long as possible, even at the cost of convenience to themselves. And even de-emphasized services (keep the lights on) are rarely deprecated completely - existing accounts and customers already using them can frequently continue to do so.
  - mgdev 2y ago
    They introduce economic incentives. :) By way of example, a little over a decade ago a famous online streaming company used to upload all of their video masters for transcoding. This process involved a huge upload and a huge workload, followed by a significant drop in activity. The problem was that AWS had to provision for peak usage instead of average usage. This resulted in a situation where the peak-to-average ratio was very high for just one or two customers. To address this issue, the solution was to incentivize these customers to spread out their workload more evenly over time, at least until they were no longer the largest driver of peak/avg. This is also why things like Reserved Instances and Spot Instances exist.
    - otterley 2y ago
      I’m sure we agree that other means such as changing pricing and reaching out to exceptional customers are not the same thing as killing a feature.
      - mgdev 2y ago
        I never said they would kill the feature. I said they would shut down the behavior. They can do that through economic incentive. (Source: I was on the S3 team. Opinions my own, etc.)
        otterley 2y ago
        You said they would “shut it down,” which can reasonably be interpreted as killing the feature. If you meant something more specific, you should have said that. Clarity is the duty of the speaker.
        mgdev 2y ago
        You must be fun at stand-ups.
        Terretta 2y ago
        >> “If everyone started using S3 for this purpose, they would shut it down pretty quickly.” > “I never said they would kill the feature. I said they would shut down the behavior.” Seems "it" is standing in for a lot of nuance here. With these three sentences side by side, still took a while to see how you might be referring to the "everyone using" behavior instead of the "S3 purpose" feature! Usually given ambiguity the nearest plausible reference wins. Since in tech "shut it down" is more often systems than humans and that was the nearest reference "it" could refer to, took some doing to see how your assertion "I said the behavior" could be accurate!

paulgb 2y ago

One advantage of this approach is that if you're already using S3 (as in the SlateDB case mentioned in the article), it's essentially “free”. And it means that a package like SlateDB just needs to be pointed at an S3 bucket, instead of making you point it to an S3 bucket _and_ a DynamoDB instance.

mgdev 2y ago
Totally. I'm not advocating that people don't ever use it for this. I'm just saying that from a pure resource perspective, it might be one of the least resource-efficient mechanisms for doing this. As others have pointed out, it's probably not a noticeable cost and, in fact, the fixed costs associated with setting something up yourself would far outweigh what you're paying to use S3 for this purpose. Part of me just dies inside when I think of all the stuff needlessly happening behind the scenes, given it's not actually being used for storage. I mean, it's called Simple Storage Service.
- mgdev 2y ago
  I'll add that if a lot of people actually did start using it for this purpose, they could probably just productionize the thing they actually use for this, which is essentially their own version of etcd, but with Paxos instead of Raft.
kortex 2y ago
Provisioning a DynamoDB table is scarcely more effort than provisioning a bucket. And you get way nicer constructs (still not as nice as RDBMS) for locking like ConditionalUpdate and TransactWrite.