5 ms·

(Disclosure: I work on Google's Cloud Datastore.) This looks super neat, and I can't wait to learn more about it, but just for the record: I'm pretty sure this

by wsh91 10y ago

(Disclosure: I work on Google's Cloud Datastore.) This looks super neat, and I can't wait to learn more about it, but just for the record: I'm pretty sure this isn't the first serverless cloud database. Both Firebase's Realtime Database and Cloud Datastore (which powers Snapchat and Pokemon Go) are serverless; you pay only for your ops and storage. They've been publicly available for several years.

evanweaver 10y ago
Fair enough; I think it depends where you draw the line between key/value store and database. Both of those depend on other distributed storage systems under the hood, as far as I am aware? Or is Datastore an end to end system? I know Firebase was backed by MongoDB.
- elvinyung 10y ago
  If I remember correctly, Datastore is basically a thin layer on top of Megastore[1] (aka the precursor to Spanner). 1: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36971.pdf https://static.googleusercontent.com/media/research.google.c...
  - itcmcgrath 10y ago
    It's a lot more than a thin layer, but like most systems at Google is a specialized layer that utilizes built on top of more fundamental building blocks that are designed and proven to do a particular job really well. In this case Megastore provides the underly multi-region/datacenter K-V replication services. All the database features like secondary & composite indexes, query language, multi-tenancy support, PAYG model, etc, etc, are built in the Cloud Datastore layer.
    - elvinyung 10y ago
      > secondary & composite indexes Interesting, so you don't use Megastore's indexes?
      - wsh91 10y ago
        As you noted earlier, Megastore has a schema and we don't, so we have our own index implementation, yes. :)
- wsh91 10y ago
  Datastore runs on top of Megastore [1]. You can find out more about our data model here [2], but it's definitely not limited to key-value data. Our end users don't have to think much about our storage system, though, if we're doing our jobs right. :) [1] https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/ https://cloud.google.com/datastore/docs/articles/balancing-s... [2] https://cloud.google.com/datastore/docs/concepts/entities https://cloud.google.com/datastore/docs/concepts/entities
  - elvinyung 10y ago
    An interesting observation that I can't seem to "un-observe" is that Megastore is actually a lot more like MongoDB than one would expect. Both ostensibly work best when the application fits a hierarchical data model (entity groups vs. documents), and provide out-of-the-box strongly-consistent transactions for a single entity group. MongoDB feels like schemaless Megastore.
    - wsh91 10y ago
      :) Here's a paper with which you might already be familiar, but it's one of the citations for the Megastore paper: http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pdf http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pd.... You'll probably enjoy it (if you haven't already!).
jchrisa 10y ago
In technology evolution there are technologies that enable a new ecosystem, and then there are technologies that are built natively for that ecosystem. The previous generation of datastores enabled Lambda style applications, the next generation of databases assumes they are the new normal. The reasons FaunaDB fits serverless like a glove can be boiled down to a few points: pay-as-you-go, database security awareness and object level access control, hierarchical multi tenancy with quality of service management. Running on multiple clouds makes the Serverless model more acceptable for risk averse enterprises, and complements multi-cloud serverless FaaS execution environments nicely. There's more to say, check out this post on the blog: https://fauna.com/blog/serverless-cloud-database https://fauna.com/blog/serverless-cloud-database and https://fauna.com/blog/escape-the-cloud-database-trap-with-serverless https://fauna.com/blog/escape-the-cloud-database-trap-with-s...
- wsh91 10y ago
  I'm familiar with both; what disappoints me is the claim of novelty here with respect to autoscaling. That's just not true. To quote you: "A serverless system must scale dynamically per request. Current popular cloud databases do not support this level of elasticity—you have to pay for capacity you don’t use. Additionally, they often lack support for joins, indexes, authentication, and other capabilities necessary to build a rich application." That first criterion we absolutely meet, today. Cloud Datastore has been doing that for eight years now. We don't have joins, but we do have indexes, auth, multi-region replication and a whole lot more.
  - jchrisa 10y ago
    That's why it comes down to the details and the fit and finish. One neat feature is the ability to run Lambdas with a database access token corresponding to a particular user, which can then be passed through to sub-Lambdas (or it can even run with sub-permissions). Here is a blog post with quickstart instructions: https://serverless.com/blog/faunadb-serverless-authentication/ https://serverless.com/blog/faunadb-serverless-authenticatio... For instance, you could have a fire-and-forget self-service self-provisioning online shopping site builder, and bill database costs through to your customers (we give you that information in response headers). You can also use FaunaDB to do consistent coordination between FaaS execution environments running in different clouds. So if you like a processing feature Azure makes available, but want to run your user facing servers in GCE, you can use FaunaDB to coordinate between the clouds.

by wsh91 10y ago

evanweaver 10y ago

Fair enough; I think it depends where you draw the line between key/value store and database. Both of those depend on other distributed storage systems under the hood, as far as I am aware? Or is Datastore an end to end system? I know Firebase was backed by MongoDB.

elvinyung 10y ago
If I remember correctly, Datastore is basically a thin layer on top of Megastore[1] (aka the precursor to Spanner). 1: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36971.pdf https://static.googleusercontent.com/media/research.google.c...
- itcmcgrath 10y ago
  It's a lot more than a thin layer, but like most systems at Google is a specialized layer that utilizes built on top of more fundamental building blocks that are designed and proven to do a particular job really well. In this case Megastore provides the underly multi-region/datacenter K-V replication services. All the database features like secondary & composite indexes, query language, multi-tenancy support, PAYG model, etc, etc, are built in the Cloud Datastore layer.
  - elvinyung 10y ago
    > secondary & composite indexes Interesting, so you don't use Megastore's indexes?
    - wsh91 10y ago
      As you noted earlier, Megastore has a schema and we don't, so we have our own index implementation, yes. :)
wsh91 10y ago
Datastore runs on top of Megastore [1]. You can find out more about our data model here [2], but it's definitely not limited to key-value data. Our end users don't have to think much about our storage system, though, if we're doing our jobs right. :) [1] https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/ https://cloud.google.com/datastore/docs/articles/balancing-s... [2] https://cloud.google.com/datastore/docs/concepts/entities https://cloud.google.com/datastore/docs/concepts/entities
- elvinyung 10y ago
  An interesting observation that I can't seem to "un-observe" is that Megastore is actually a lot more like MongoDB than one would expect. Both ostensibly work best when the application fits a hierarchical data model (entity groups vs. documents), and provide out-of-the-box strongly-consistent transactions for a single entity group. MongoDB feels like schemaless Megastore.
  - wsh91 10y ago
    :) Here's a paper with which you might already be familiar, but it's one of the citations for the Megastore paper: http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pdf http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pd.... You'll probably enjoy it (if you haven't already!).

jchrisa 10y ago

In technology evolution there are technologies that enable a new ecosystem, and then there are technologies that are built natively for that ecosystem. The previous generation of datastores enabled Lambda style applications, the next generation of databases assumes they are the new normal. The reasons FaunaDB fits serverless like a glove can be boiled down to a few points: pay-as-you-go, database security awareness and object level access control, hierarchical multi tenancy with quality of service management. Running on multiple clouds makes the Serverless model more acceptable for risk averse enterprises, and complements multi-cloud serverless FaaS execution environments nicely. There's more to say, check out this post on the blog: https://fauna.com/blog/serverless-cloud-database https://fauna.com/blog/serverless-cloud-database and https://fauna.com/blog/escape-the-cloud-database-trap-with-serverless https://fauna.com/blog/escape-the-cloud-database-trap-with-s...

wsh91 10y ago
I'm familiar with both; what disappoints me is the claim of novelty here with respect to autoscaling. That's just not true. To quote you: "A serverless system must scale dynamically per request. Current popular cloud databases do not support this level of elasticity—you have to pay for capacity you don’t use. Additionally, they often lack support for joins, indexes, authentication, and other capabilities necessary to build a rich application." That first criterion we absolutely meet, today. Cloud Datastore has been doing that for eight years now. We don't have joins, but we do have indexes, auth, multi-region replication and a whole lot more.
- jchrisa 10y ago
  That's why it comes down to the details and the fit and finish. One neat feature is the ability to run Lambdas with a database access token corresponding to a particular user, which can then be passed through to sub-Lambdas (or it can even run with sub-permissions). Here is a blog post with quickstart instructions: https://serverless.com/blog/faunadb-serverless-authentication/ https://serverless.com/blog/faunadb-serverless-authenticatio... For instance, you could have a fire-and-forget self-service self-provisioning online shopping site builder, and bill database costs through to your customers (we give you that information in response headers). You can also use FaunaDB to do consistent coordination between FaaS execution environments running in different clouds. So if you like a processing feature Azure makes available, but want to run your user facing servers in GCE, you can use FaunaDB to coordinate between the clouds.