6 ms·

As I understand it, Vitess is basically a really powerful sharding system, which goes a step further than typical sharding solutions by basically making the sha

As I understand it, Vitess is basically a really powerful sharding system, which goes a step further than typical sharding solutions by basically making the shards one or more unique databases. In the case of someone like slack, because your tenant (e.g your company slack), is completely isolated from other tenants, you can treat that basically as its own database, and have a master for just that DB, allowing much better scaling. The big limitation on Vitess is cross-shard transactions, and the fact you have to make sure your schema has a clear cut sharding key (like your tenant ID) that works nicely with your application needs. The alternative for scaling transactional SQL DBs in a multi-master fashion are the "NewSQL" DBs like yugabyte and cockroachdb which are basically document DBs with a partially implemented postgres frontend, so don't have the full feature set of your SQL engine like Vitesse does but don't require so much attention to sharding. These are oversimplifications of the actual mechanisms, but give a basic overview of the tradeoffs involved, please feel free to correct me on any inaccuracies as I'm not an expert in DBs.

kmavm 5y ago
I was Chief Architect at Slack from 2016 to 2020, and was privileged to work with the engineers who were doing the work of migrating to Vitess in that timeframe. The assumption that tenants are perfectly isolated is actually the original sin of early Slack infrastructure that we adopted Vitess to migrate away from. From some earlier features in the Enterprise product (which joins lots of "little Slacks" into a corporate-wide entity) to more post-modern features like Slack Connect (https://slack.com/help/articles/1500001422062-Start-a-direct-message-with-someone-outside-your-company https://slack.com/help/articles/1500001422062-Start-a-direct...) or Network Shared Channels (https://slack.com/blog/news/shared-channels-growth-innovation https://slack.com/blog/news/shared-channels-growth-innovatio...), the idea that each tenant is fully isolated was increasingly false. Vitess is a meta-layer on top of MySQL shards that asks, per table, which key to shard on. It then uses that information to maintain some distributed indexes of its own, and to plan the occasional scatter/gather query appropriately. In practice, simply migrating code from our application-sharded, per-tenant old way into the differently-sharded Vitess storage system was not a simple matter of pointing to a new database; we had to change data access patterns to avoid large fan-out reads and writes. The team did a great write-up about it here: https://slack.engineering/scaling-datastores-at-slack-with-vitess/ https://slack.engineering/scaling-datastores-at-slack-with-v...
- motives 5y ago
  Definitely wasn't expecting the chief architect at Slack to reply to that example, really appreciate the response, HN is such a blessing in that regard :). The scaling datastores at slack is a super interesting read aswell thanks, does make me wonder if there was a fully 100% MySQL compatible version of yugabyte/spanner etc if that would have shifted the decision.
- dcherman 5y ago
  Random aside, were you at KubeCon a couple years ago chatting with Sugu at the whole conference party in San Diegi? If so, hi! I was crazy out of my depth, but listening to folks that know this stuff better than I ever will was one of the highlights of that conference
- throwdbaaway 5y ago
  > In the fall of 2016, we were dealing with hundreds of thousands of MySQL queries per second and thousands of sharded MySQL hosts in production. > Today, we serve 2.3 million QPS at peak. 2M of those queries are reads and 300K are writes. I think the "today" QPS numbers are still doable with a properly tuned single-writer galera cluster running on machines with TBs of memory. Of course, with Slack workload, there would be too much historical data to fit into a single host, so I can see the reasons to shard into multiple clusters/hosts. Still, the numbers seem a little off. Let's say back in fall 2016 there were already 200K write QPS at peak, with 200 sharded hosts accepting write. That's just 1K write QPS at peak per host on average, and let's say 20K write QPS at peak for a particularly hot shard. What could be the bottleneck? Replication lag? Data size? I don't think any of the articles from Slack has talked about this. What Vitess provides is invaluable, especially the very solid implementation of secondary index. But sometimes I feel like it is being used/advocated as a sledgehammer ("just keep sharding") without looking at what could be done better at the lower MySQL/InnODB level, in exchange for a much more costly cloud bill.
irfansharif 5y ago
(crdb eng) I'm not sure what "document DB" means here, mind elaborating?
- aPoCoMiLogin 5y ago
  he is probably referring to the docdb document store in yugabyte: https://docs.yugabyte.com/latest/architecture/layered-architecture/#docdb https://docs.yugabyte.com/latest/architecture/layered-archit...
  - motives 5y ago
    Indeed I was referring to yugabyte, apologies for the clumsy phrasing, I havent used crdb but I guess it is a postgres frontend layered on a KV store instead of a document store?
    - qeternity 5y ago
      As opposed to? Postgres is “just” a Postgres front end on top of a kv store.
      - anarazel 5y ago
        I don't think that's really true. Which part of postgres would you describe as a KV store, compared to what cockroach does with RocksDB?
    - manigandham 5y ago
      Yugabyte uses actual Postgres code for the query layer, on top of data persistence (distribution/replication) handled by DocDB document store, which itself is a layer on top of RocksDB: https://blog.yugabyte.com/how-we-built-a-high-performance-document-store-on-rocksdb/ https://blog.yugabyte.com/how-we-built-a-high-performance-do... CockroachDB (aka CRDB) is completely custom and compatible with Postgres wire/datatype protocols, which operates directly on its own key/value store called Pebble (but originally was also RocksDB): https://www.cockroachlabs.com/blog/distributed-sql-key-value-store/ https://www.cockroachlabs.com/blog/distributed-sql-key-value... Both systems are foundationally the same SQL-on-KV but implement it very differently.
Rapzid 5y ago
Something you really hit on here; there is no free lunch with clustered databases. You have to design your application to account for the sharding or you will run into data locality related performance issues.
ddorian43 5y ago
> your SQL engine like Vitesse does but don't require so much attention to sharding You always need a lot of attention to sharding otherwise you'll have poor performance.

kmavm 5y ago

I was Chief Architect at Slack from 2016 to 2020, and was privileged to work with the engineers who were doing the work of migrating to Vitess in that timeframe. The assumption that tenants are perfectly isolated is actually the original sin of early Slack infrastructure that we adopted Vitess to migrate away from. From some earlier features in the Enterprise product (which joins lots of "little Slacks" into a corporate-wide entity) to more post-modern features like Slack Connect (https://slack.com/help/articles/1500001422062-Start-a-direct-message-with-someone-outside-your-company https://slack.com/help/articles/1500001422062-Start-a-direct...) or Network Shared Channels (https://slack.com/blog/news/shared-channels-growth-innovation https://slack.com/blog/news/shared-channels-growth-innovatio...), the idea that each tenant is fully isolated was increasingly false. Vitess is a meta-layer on top of MySQL shards that asks, per table, which key to shard on. It then uses that information to maintain some distributed indexes of its own, and to plan the occasional scatter/gather query appropriately. In practice, simply migrating code from our application-sharded, per-tenant old way into the differently-sharded Vitess storage system was not a simple matter of pointing to a new database; we had to change data access patterns to avoid large fan-out reads and writes. The team did a great write-up about it here: https://slack.engineering/scaling-datastores-at-slack-with-vitess/ https://slack.engineering/scaling-datastores-at-slack-with-v...

motives 5y ago
Definitely wasn't expecting the chief architect at Slack to reply to that example, really appreciate the response, HN is such a blessing in that regard :). The scaling datastores at slack is a super interesting read aswell thanks, does make me wonder if there was a fully 100% MySQL compatible version of yugabyte/spanner etc if that would have shifted the decision.
dcherman 5y ago
Random aside, were you at KubeCon a couple years ago chatting with Sugu at the whole conference party in San Diegi? If so, hi! I was crazy out of my depth, but listening to folks that know this stuff better than I ever will was one of the highlights of that conference
throwdbaaway 5y ago
> In the fall of 2016, we were dealing with hundreds of thousands of MySQL queries per second and thousands of sharded MySQL hosts in production. > Today, we serve 2.3 million QPS at peak. 2M of those queries are reads and 300K are writes. I think the "today" QPS numbers are still doable with a properly tuned single-writer galera cluster running on machines with TBs of memory. Of course, with Slack workload, there would be too much historical data to fit into a single host, so I can see the reasons to shard into multiple clusters/hosts. Still, the numbers seem a little off. Let's say back in fall 2016 there were already 200K write QPS at peak, with 200 sharded hosts accepting write. That's just 1K write QPS at peak per host on average, and let's say 20K write QPS at peak for a particularly hot shard. What could be the bottleneck? Replication lag? Data size? I don't think any of the articles from Slack has talked about this. What Vitess provides is invaluable, especially the very solid implementation of secondary index. But sometimes I feel like it is being used/advocated as a sledgehammer ("just keep sharding") without looking at what could be done better at the lower MySQL/InnODB level, in exchange for a much more costly cloud bill.

irfansharif 5y ago

(crdb eng) I'm not sure what "document DB" means here, mind elaborating?

aPoCoMiLogin 5y ago
he is probably referring to the docdb document store in yugabyte: https://docs.yugabyte.com/latest/architecture/layered-architecture/#docdb https://docs.yugabyte.com/latest/architecture/layered-archit...
- motives 5y ago
  Indeed I was referring to yugabyte, apologies for the clumsy phrasing, I havent used crdb but I guess it is a postgres frontend layered on a KV store instead of a document store?
  - qeternity 5y ago
    As opposed to? Postgres is “just” a Postgres front end on top of a kv store.
    - anarazel 5y ago
      I don't think that's really true. Which part of postgres would you describe as a KV store, compared to what cockroach does with RocksDB?
  - manigandham 5y ago
    Yugabyte uses actual Postgres code for the query layer, on top of data persistence (distribution/replication) handled by DocDB document store, which itself is a layer on top of RocksDB: https://blog.yugabyte.com/how-we-built-a-high-performance-document-store-on-rocksdb/ https://blog.yugabyte.com/how-we-built-a-high-performance-do... CockroachDB (aka CRDB) is completely custom and compatible with Postgres wire/datatype protocols, which operates directly on its own key/value store called Pebble (but originally was also RocksDB): https://www.cockroachlabs.com/blog/distributed-sql-key-value-store/ https://www.cockroachlabs.com/blog/distributed-sql-key-value... Both systems are foundationally the same SQL-on-KV but implement it very differently.

Rapzid 5y ago

Something you really hit on here; there is no free lunch with clustered databases. You have to design your application to account for the sharding or you will run into data locality related performance issues.

ddorian43 5y ago

> your SQL engine like Vitesse does but don't require so much attention to sharding You always need a lot of attention to sharding otherwise you'll have poor performance.