6 ms·
Show HN: Spice.ai – materialize, accelerate, and query SQL data from any source
Hi HN, We're Luke and Phillip, and we're building Spice.ai OSS - a lightweight, portable runtime, built in Rust and powered by Apache DataFusion to locally materialize, accelerate, and query data tables sourced from any database, data warehouse or data lake.
Phillip and I first introduced Spice on Show HN in September 2021. Since then, we’ve been schooled and humbled in every way building 100TB+ data and ML systems for the https://spice.ai https://spice.ai cloud platform. Along with our customers, we struggled with getting fast, low-latency, high-concurrency SQL query within a budget, accessing and combining data from many sources, trade-offs between OLTP/OLAP compute engines, and managing datasets as code.
Today, we’re re-launching Spice, completely rebuilt from the ground up, to directly solve several of the problems we had in accessing data quickly and cost-effectively providing it to applications, dashboards, and machine learning. Spice provides federated SQL query across databases (MySQL, PostgreSQL, etc.), data warehouses (Snowflake, BigQuery, etc.) and data lakes (S3, MinIO, Databricks, etc.) with the ability to materialize remote datasets locally using in-memory Arrow, DuckDB, SQLite, or PostgreSQL. Accelerated engines run in your infrastructure giving you flexibility and control over price and performance.
You can read the full announcement blog post at https://blog.spiceai.org/posts/2024/03/28/adding-spice-the-next-generation-of-spice.ai-oss/ https://blog.spiceai.org/posts/2024/03/28/adding-spice-the-n....
We’d appreciate it if you check Spice out, give us feedback, and if you'd like to contribute, we'd love to build with you.
Thanks!
GitHub: https://github.com/spiceai/spiceai https://github.com/spiceai/spiceai
- dennispan 2y ago[flagged]
- dvdsgl 2y agoCongrats on the launch! This is exciting. The video demo is awesome: https://youtu.be/AZyrecVWnEs?si=j7JVKhhcUor1_y-f https://youtu.be/AZyrecVWnEs?si=j7JVKhhcUor1_y-f
- lukekim 2y agoThank you!
- deleted 2y ago[deleted]
- jjustin_lawson 2y agoCongrats on the launch team!
- alamb 2y agoSo great to see another project built on DataFusion @!
- phillip-spice 2y agoThanks Andrew! I'm looking forward to contributing back to DataFusion as well.
- peycke 2y ago[flagged]
- watsondoc 2y agoWow, looks promising
- leeholim 2y agoCongratulations on the launch!!
- cedrone 2y agoCongrats Luke & Phillip– exciting day!
- dwgray 2y agoThis looks great - I've been meaning to dig into Rust - seems like a solid choice for you.
- mritchie712 2y agoVery cool! One thing to keep in mind: DuckDB can directly query parquet files (and many other file types[1]), mysql, postgres[0], and SQLite. So if you're in need of something like this, DuckDB on it's own might work for your use case. 0 - https://duckdb.org/docs/extensions/postgres https://duckdb.org/docs/extensions/postgres 1 - https://twitter.com/thisritchie/status/1767922982046015840 https://twitter.com/thisritchie/status/1767922982046015840
- lukekim 2y agoYes, we're huge fans of DuckDB, Mark, Hannes and the team. What we've found is sometimes you want to materialize data in an OTLP DB, so what Spice gives you is the choice to store some datasets in DuckDB and some in something like SQLite/PostgreSQL and join them together in a single SQL query, so you can get the best of both worlds.
- riku_iki 2y agoDuckDB can both read/write to PG. What exactly usecase you are unlocking?..
- phillip-spice 2y agoDuckDB is an in-process DB similar to SQLite - so every application in your stack would need to embed it. Spice is a binary that has Flight SQL and HTTP query endpoints - so multiple applications can connect to it from any language.
- lukekim 2y agoDuckDB is awesome. As an OLAP columnar-store database it excels at certain operations, like aggregations. If your use-case is row-based lookups where an OLTP database would perform better, you now get a choice of engine, while still having a single place to access your data from your app. Originally, we only supported DuckDB in our cloud product Spice Firecache, but actually lost a customer because their use-case was optimized for an OLTP DB. Now, you can get a choice... down to the dataset level and still be able to join across them in a single query. With Spice, you can load both SQLite and DuckDB together in the same process for local materialization and acceleration. Finally, Spice OSS does more than just data query. You can read about the vision to power AI-driven applications by co-locating data with models at https://docs.spiceai.org/intelligent-applications https://docs.spiceai.org/intelligent-applications.
- alex_hirner 2y agoLooks great! Is flightsql supported over the wire too, so one could hook it up to grafana? Any plans to support iceberg?
- jeadie 2y agoYes! It can connect to FlightSQL compatible servers (see https://docs.spiceai.org/data-connectors/flightsql https://docs.spiceai.org/data-connectors/flightsql ) and its also a FlightSQL compatible server
- lukekim 2y agoWe also have a Grafana plugin we'll continue to improve to make it super easy to connect to Grafana, and Spice has a metrics endpoint and example Grafana dashboard for monitoring itself https://github.com/spiceai/spiceai/blob/trunk/monitoring/grafana-dashboard.json https://github.com/spiceai/spiceai/blob/trunk/monitoring/gra...
- jeadie 2y agoAnd yes, Iceberg is very high up on our list
- lmeyerov 2y agoAny sense of comparison to Dremio, which helped steward the Arrow ecosystem for doing this kind of thing? (The idea is great fwiw, I've been following them one-off for years, and we have to do elements of these things in how we build louie.ai and Graphistry for the GPU equivalent. Real pain point!)
- lukekim 2y agoDremio is awesome. We've followed the Dremio journey from one of Jacques' original talks a couple of years back. Dremio's idea of caching tiers and reflections is powerful for performance. Spice takes it further and provides flexibility for materialization, giving you full control over where that materialization exists (same machine, same pod, same network, same cluster, same region, etc.), what engine/processing (OLTP - SQLite/PostgreSQL, OLAP - DuckDB/Arrow) it uses and what tier (in-memory, attached NVMe, etc.) to store it down to the dataset level.
- ignoramous 2y ago> Today, we're re-launching Spice... Obtaining blockchain and smart-contract data is hard ... Spice makes it easy. http://web.archive.org/web/20220414105622/https://docs.spice.xyz/ http://web.archive.org/web/20220414105622/https://docs.spice... A slight detour from the company's original vision (https://archive.is/88IoQ https://archive.is/88IoQ)?
- lukekim 2y agoActually, we posted the original vision in Sep 2021 at https://blog.spiceai.org/posts/2021/09/07/introducing-spice.ai-open-source-time-series-ai-for-developers/ https://blog.spiceai.org/posts/2021/09/07/introducing-spice.... for AI-driven applications and discussed needing a good source of data at https://blog.spiceai.org/posts/2021/12/05/ai-needs-ai-ready-data/ https://blog.spiceai.org/posts/2021/12/05/ai-needs-ai-ready-.... We believe blockchain data is one of the most interesting time-series datasets to work in developing an AI-driven application platform, because it's continuous, well-structured, has many applications, and is open to index. Regardless of views on crypto, from a purely technical/data feed perspective, it's quite useful for testing time-series systems.
- ignoramous 2y ago> original vision ... https://blog.spiceai.org/posts/2021/09/07/introducing-spice.ai-open-source-time-series-ai-for-developers/ https://blog.spiceai.org/posts/2021/09/07/introducing-spice.... Agree. I shared the same blog link (via archive.is) in the comment you replied to. Detour into blockchain, even if warranted, might have been a distraction? Either way, congratulations on this relaunch.
- lukekim 2y agoThank you! Much appreciated.
- spxneo 2y agowow thanks for pointing this out I have a short circuit for whenever I see the B word or still pushing this smart-contract non-sense that isn't being used in serious real world projects with legal repercussions....for the 10+ years this technology has existed
- martinmao 2y agoThis looks awesome!
- neeleshs 2y agoCongratulations. Is this similar to Trino/Starburst, Drill?
- lukekim 2y agoThank you! Yes, in terms of federated queries, there are similarities, but Spice is designed to be much smaller, faster, and lightweight (single-binary, 140MB) so you can run it next to your application as a sidecar, or eventually even in the browser. Spice also gives you more options and flexibility for materialization, so you can choose where and how to store local materialized data.
- prabhatsharma 2y agoDo you support subqueries and joins?
- lukekim 2y agoSpice supports what DataFusion supports, which is generally yes but there is still work to do to push down more queries to TableProviders. For example, joins within a single source are not yet pushed down to the underlying provider. You can write a single query across many data sources which is what we show in the demo on the Git repo.
- phillip-spice 2y agoThere is an effort within DataFusion to support pushing down joins across tables from the same remote provider that we will likely contribute to as well: https://github.com/datafusion-contrib/datafusion-federation https://github.com/datafusion-contrib/datafusion-federation
- CyberDildonics 2y agoThere are eight different accounts in this thread "congratulating the launch" with their first comment. Half were created six hours ago right when this was posted. https://news.ycombinator.com/user?id=martinmao https://news.ycombinator.com/user?id=martinmao https://news.ycombinator.com/user?id=dwgray https://news.ycombinator.com/user?id=dwgray https://news.ycombinator.com/user?id=dennispan https://news.ycombinator.com/user?id=dennispan https://news.ycombinator.com/user?id=peycke https://news.ycombinator.com/user?id=peycke https://news.ycombinator.com/user?id=watsondoc https://news.ycombinator.com/user?id=watsondoc https://news.ycombinator.com/user?id=leeholim https://news.ycombinator.com/user?id=leeholim https://news.ycombinator.com/user?id=cedrone https://news.ycombinator.com/user?id=cedrone https://news.ycombinator.com/user?id=jjustin_lawson https://news.ycombinator.com/user?id=jjustin_lawson
- sneilan1 2y agoMy first thought when I saw your post pointing out the congratulations comments was one of the ending scenes from Neon Genesis Evangelion where they say congratulations repeatedly. https://youtu.be/oyFQVZ2h0V8?si=oOYSIjVmpJK6mwft https://youtu.be/oyFQVZ2h0V8?si=oOYSIjVmpJK6mwft Regardless, this is very spammy marketing.
- CyberDildonics 2y ago[flagged]
- zachmu 2y agoPeople have friends who want to support them
- deleted 2y ago[deleted]
- pvg 2y agoIf you think someone is posting abusively, email the mods. You've seen the thing about not-posting shillage insinuations in the site guidelines.
- nextworddev 2y agoHey guys - how does this compare to cube?
- phillip-spice 2y agoI'm not too familiar with https://cube.dev/ https://cube.dev/ - but my initial impression is they are focused more on providing APIs backed by SQL. They have a SQL API that emulates the PostgreSQL wire protocol, whereas Spice implements Arrow and Flight SQL natively. Their pre-aggregations are a similar concept to Spice's data accelerators. It also looks like they have their own query language, whereas Spice is native SQL as well.
- marooned4 2y agolooks great . Going to try this out
- imgdesgen 2y agoThat's awesome! I'll definitely give it a try if there's a suitable scenario.
- lukekim 2y agoThanks! Feedback and GitHub issues welcome!
- gerenuk 2y agoInteresting one. Any plans for clickhouse data connector?
- lukekim 2y agoYes, it's on the backlog and we'll prioritize as we see demand as with https://github.com/spiceai/spiceai/issues/999 https://github.com/spiceai/spiceai/issues/999.