5 ms·
I think the comparison was Snowflake vs Databricks SQL. Databricks SQL is a PaaS service just like Snowflake. Also, it uses their Photon engine, which is a prop
by rdeboo 5y ago
I think the comparison was Snowflake vs Databricks SQL. Databricks SQL is a PaaS service just like Snowflake. Also, it uses their Photon engine, which is a proprietary engine written in C++. It is not Spark.
- hiyer 5y agoI'm aware that Databricks is a PaaS service, but what Databricks runs under the hood is Spark (with a few proprietary extensions). So your jobs/queries do require some tuning just like with OS Spark. Spark has had SQL engines (SparkSQL/Hive on Spark) for a long time. Photon is just a new, faster one. Photon tasks also run on Spark executors only, so it's not independent of Spark[1]. Also, while it's proprietary now, I wouldn't be surprised if Databricks open-sources it in the future, like they did with Delta Lake. 1. https://databricks.com/blog/2021/06/17/announcing-photon-public-preview-the-next-generation-query-engine-on-the-databricks-lakehouse-platform.html https://databricks.com/blog/2021/06/17/announcing-photon-pub...