6 ms·

IOx: InfluxData’s New Storage Engine

167 points by resizeitplz 4y ago

michael_j_ward 4y ago
Just want to say congratulations to the team! 2 years and 9,500+ commits is a hell of a feat. https://github.com/influxdata/influxdb_iox https://github.com/influxdata/influxdb_iox
PaulWaldman 4y ago
>Unbounded cardinality This has been the largest criticism of InfluxDB in the past. Kudos to the team for acknowledging and solving it! > IOx supports SQL natively and our cloud customers can connect using Postgres-compatible clients like psql, Grafana’s Postgres data source, and BI tools like PowerBI and Tableau. Initially InfluxDB had InfluxQL, a SQL like language for querying data. Then they transitioned to Flux, indicating it was superior to writing complex SQL queries over time series data. Now they are highlighting native SQL support. Since this was only announced today, hopefully there will be clear messaging on which query languages will be supported going forward. It’s also worth noting that queries can also be executed over an HTTP API that platforms like PowerBI can consume today. >First introduced in 2020 as the open source project InfluxDB IOx, the new storage engine is the product of sustained development by InfluxData and considerable contribution from the InfluxDB open source developer community. Today, the new engine based on IOx arrives first in InfluxData’s multi-tenant InfluxDB Cloud service, available to developers worldwide. Will this later be available in an OSS package for self-hosting?
- minhazm 4y ago
  The SQL support is likely because they're using DataFusion which already has pretty good SQL support, so it's sort of "free". https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html https://arrow.apache.org/datafusion/user-guide/sql/sql_statu...
  - alamb 4y ago
    Author here -- it is "free" in the sense that all the effort we put into DataFusion flows directly into IOx. But we do put a lot of effort into DataFusion
    - minhazm 4y ago
      I didn't mean to imply it's free as in no effort goes into. Just that the underlying library provides it so it's less effort on top of the already significant effort going into DataFusion itself.
      - alamb 4y ago
        Ah -- got it! This is the beauty of aligning ourselves with technologies like Arrow, Parquet and DataFusion. We can share as well as benefit from the efforts of the broader community
- pauldix 4y ago
  Hi, post author and founder of InfluxDB here. We're supporting Flux (our scripting and query language), InfluxQL (our original SQL like language), and SQL (specifically the Postgres dialect as that's what DataFusion supports). The query engine is DataFusion, which is part of the Apache Arrow project. We contribute to it significantly. So that's what's built in natively. We support Flux and InfluxQL through separate Go processes that use an API to connect to the core DB. Although we're working on native InfluxQL support (it's a Rust based InfluxQL parser that will yield DataFusion logical query plans). Right now we're focused on our cloud offering. We'll have official open source releases and documentation in the future.
okay_dude_q 4y ago
I evaluated InfluxDB with the Prometheus Kube Stack chart. It didn’t really work. I’m not stupid and I can read docs. My feeling was it’s like Elastic. Default configuration is so flawed and inscrutable, on purpose, you can forget about using it yourself. I use Thanos now. At least it fucking works. I suppose if I need fast queries, I’ll use Postgres. You guys need to focus on making stuff that works. It’s competitive out there and you don’t have the insights into people who try and wind up hating your guts for being annoying.
mrsun 4y ago
Will InfluxDB IOx eventually replace InfluxDB v2?
- mhall119 4y ago
  IOx is the data storage layer. It will replace the current TSM data storage system in InfluxDB, but it won't replace InfluxDB as a whole.
  - digerata 4y ago
    Personally, very excited to see this happening. Huge congrats! Some constructive criticism around naming... You don't have to have Flux in every single damn thing you create! InfluxDB IOx is not replacing InfluxDB v2 because... It's just a new storage engine. For querying we have Flux or InfluxQL...
_peter_ 4y ago
Isn't InfluxDB rewriting their storage engine for the nth time? It makes me have a little less faith in their project to be honest.
- mhall119 4y ago
  The original TSM engine is still used by InfluxDB v2 OSS. The InfluxDB Cloud platform uses a variation of TSM that's tailored for a distributed SaaS rather than stand-alone nodes (this was originally intended to be used in InfluxDB v2 OSS as well, but alpha-testing showed that the old engine performed better there so it ultimately was reverted for the beta release). So IOx is really the first major new storage engine in InfluxDB.
- c4wrd 4y ago
  I would argue the other way and praise them for the storage engine changes. Each iteration has had drawbacks, but based on the real-world reported usage they've made decisions to better support what customers are asking for and actually running into, as opposed to trying to iterate on the same engine over and over and making assumptions of real-world usage. Sure, there are drawbacks, but at the end of the day they're continuing to make good improvements for their customers.
- dgnorton 4y ago
  Member of the engineering team here - I would break the history into 3 phases: 1) Alpha / Beta phase where we experimented with several off-the-shelf key-value stores (RocksDB, LevelDB, & BoltDB). During this early phase, we learned from observing a wide variety of workloads / use-cases that we needed a custom built engine to achieve our early performance goals. But, using these off-the-shelf key-value stores allowed our (at the time) very small team to focus on developing a useful beta product and gathering user feedback. 2) TSM storage engine for 1.0 - Developed from scratch based on our learnings from phase 1, this was the first production storage engine that shipped with 1.0 in 2016 and carried us through 2.0. It served as the workhorse for 3 - 4 years as both the number of users and size of their workloads skyrocketed, eventually bumping into architectural limits of TSM. 3) IOx - equipped with a larger engineering team and years of experience with a wide variety of workloads and use-cases, IOx was developed to handle rapidly growing time series workloads that users need to handle.
mildbyte 4y ago
Just wanted to also give a shout out to Apache DataFusion[0] that IOx relies on a lot (and contributes to as well!). It's a framework for writing query engines in Rust that takes care of a lot of heavy lifting around parsing SQL, type casting, constructing and transforming query plans and optimizing them. It's pluggable, making it easy to write custom data sources, optimizer rules, query nodes etc. It's has very good single-node performance (there's even a way to compile it with SIMD support) and Ballista [1] extends that to build it into a distributed query engine. Plenty of other projects use it besides IOx, including VegaFusion, ROAPI, Cube.js's preaggregation store. We're heavily using it to build Seafowl [2], an analytical database that's optimized for running SQL queries directly from the user's browser (caching, CDNs, low latency, some WASM support, all that fun stuff). [0] https://github.com/apache/arrow-datafusion https://github.com/apache/arrow-datafusion [1] https://github.com/apache/arrow-ballista https://github.com/apache/arrow-ballista [2] https://github.com/splitgraph/seafowl https://github.com/splitgraph/seafowl
- pauldix 4y ago
  DataFusion is great, we're happy to be contributing to it. Also excited to see so many people around the world picking it up and contributing as well. With our development efforts on IOx, it's like a strong tailwind. But we put a ton of effort into helping manage community efforts (thanks, alamb! our developer on IOx that is also on the Arrow PMC).
  - nevi-me 4y ago
    Your initial experiments and decision to build on arrow-rs has been great for the project. Thank you and everyone involved.
  - andygrove 4y ago
    Original author of DataFusion/Ballista here. Having alamb and others from InfluxData involved has been a huge help in driving the project forward and helping build an active community behind the project. It is genuinely hard to keep up with the momentum these days!
toinbis 4y ago
Happy longtime Influxdb user here. I wanted to congratulate Paul and the team on reaching this milestone. Followed IOx development a bit - can't wait to finally test it out!
otoolep 4y ago
Congrats to the team at InfluxDB - great to see this released.
candrewlee14 4y ago
Congrats! Was always a pleasure to hear about IOx when I interned there last summer! They’re an awesome company to work for.
eskaytwo 4y ago
Will be very interesting to see this compared to Clickhouse
- zX41ZdbW 4y ago
  Here is a recent comparison: https://arxiv.org/pdf/2204.09795.pdf https://arxiv.org/pdf/2204.09795.pdf Although I'm not sure if it is using the new IOx engine or not.
  - eskaytwo 4y ago
    Thanks. It’s odd they do such in-depth analysis but don’t mention the versions used. I think that is on the old engine. It would be very interesting to see how the new engine performs - I imagine the performance gets closer to something like Vaex which was using similar Apache tooling

167 points by resizeitplz 4y ago

Just want to say congratulations to the team! 2 years and 9,500+ commits is a hell of a feat. https://github.com/influxdata/influxdb_iox https://github.com/influxdata/influxdb_iox

PaulWaldman 4y ago

>Unbounded cardinality This has been the largest criticism of InfluxDB in the past. Kudos to the team for acknowledging and solving it! > IOx supports SQL natively and our cloud customers can connect using Postgres-compatible clients like psql, Grafana’s Postgres data source, and BI tools like PowerBI and Tableau. Initially InfluxDB had InfluxQL, a SQL like language for querying data. Then they transitioned to Flux, indicating it was superior to writing complex SQL queries over time series data. Now they are highlighting native SQL support. Since this was only announced today, hopefully there will be clear messaging on which query languages will be supported going forward. It’s also worth noting that queries can also be executed over an HTTP API that platforms like PowerBI can consume today. >First introduced in 2020 as the open source project InfluxDB IOx, the new storage engine is the product of sustained development by InfluxData and considerable contribution from the InfluxDB open source developer community. Today, the new engine based on IOx arrives first in InfluxData’s multi-tenant InfluxDB Cloud service, available to developers worldwide. Will this later be available in an OSS package for self-hosting?

minhazm 4y ago
The SQL support is likely because they're using DataFusion which already has pretty good SQL support, so it's sort of "free". https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html https://arrow.apache.org/datafusion/user-guide/sql/sql_statu...
- alamb 4y ago
  Author here -- it is "free" in the sense that all the effort we put into DataFusion flows directly into IOx. But we do put a lot of effort into DataFusion
  - minhazm 4y ago
    I didn't mean to imply it's free as in no effort goes into. Just that the underlying library provides it so it's less effort on top of the already significant effort going into DataFusion itself.
    - alamb 4y ago
      Ah -- got it! This is the beauty of aligning ourselves with technologies like Arrow, Parquet and DataFusion. We can share as well as benefit from the efforts of the broader community
pauldix 4y ago
Hi, post author and founder of InfluxDB here. We're supporting Flux (our scripting and query language), InfluxQL (our original SQL like language), and SQL (specifically the Postgres dialect as that's what DataFusion supports). The query engine is DataFusion, which is part of the Apache Arrow project. We contribute to it significantly. So that's what's built in natively. We support Flux and InfluxQL through separate Go processes that use an API to connect to the core DB. Although we're working on native InfluxQL support (it's a Rust based InfluxQL parser that will yield DataFusion logical query plans). Right now we're focused on our cloud offering. We'll have official open source releases and documentation in the future.

okay_dude_q 4y ago

I evaluated InfluxDB with the Prometheus Kube Stack chart. It didn’t really work. I’m not stupid and I can read docs. My feeling was it’s like Elastic. Default configuration is so flawed and inscrutable, on purpose, you can forget about using it yourself. I use Thanos now. At least it fucking works. I suppose if I need fast queries, I’ll use Postgres. You guys need to focus on making stuff that works. It’s competitive out there and you don’t have the insights into people who try and wind up hating your guts for being annoying.

mrsun 4y ago

Will InfluxDB IOx eventually replace InfluxDB v2?

mhall119 4y ago
IOx is the data storage layer. It will replace the current TSM data storage system in InfluxDB, but it won't replace InfluxDB as a whole.
- digerata 4y ago
  Personally, very excited to see this happening. Huge congrats! Some constructive criticism around naming... You don't have to have Flux in every single damn thing you create! InfluxDB IOx is not replacing InfluxDB v2 because... It's just a new storage engine. For querying we have Flux or InfluxQL...

_peter_ 4y ago

Isn't InfluxDB rewriting their storage engine for the nth time? It makes me have a little less faith in their project to be honest.

mhall119 4y ago
The original TSM engine is still used by InfluxDB v2 OSS. The InfluxDB Cloud platform uses a variation of TSM that's tailored for a distributed SaaS rather than stand-alone nodes (this was originally intended to be used in InfluxDB v2 OSS as well, but alpha-testing showed that the old engine performed better there so it ultimately was reverted for the beta release). So IOx is really the first major new storage engine in InfluxDB.
c4wrd 4y ago
I would argue the other way and praise them for the storage engine changes. Each iteration has had drawbacks, but based on the real-world reported usage they've made decisions to better support what customers are asking for and actually running into, as opposed to trying to iterate on the same engine over and over and making assumptions of real-world usage. Sure, there are drawbacks, but at the end of the day they're continuing to make good improvements for their customers.
dgnorton 4y ago
Member of the engineering team here - I would break the history into 3 phases: 1) Alpha / Beta phase where we experimented with several off-the-shelf key-value stores (RocksDB, LevelDB, & BoltDB). During this early phase, we learned from observing a wide variety of workloads / use-cases that we needed a custom built engine to achieve our early performance goals. But, using these off-the-shelf key-value stores allowed our (at the time) very small team to focus on developing a useful beta product and gathering user feedback. 2) TSM storage engine for 1.0 - Developed from scratch based on our learnings from phase 1, this was the first production storage engine that shipped with 1.0 in 2016 and carried us through 2.0. It served as the workhorse for 3 - 4 years as both the number of users and size of their workloads skyrocketed, eventually bumping into architectural limits of TSM. 3) IOx - equipped with a larger engineering team and years of experience with a wide variety of workloads and use-cases, IOx was developed to handle rapidly growing time series workloads that users need to handle.

mildbyte 4y ago

Just wanted to also give a shout out to Apache DataFusion[0] that IOx relies on a lot (and contributes to as well!). It's a framework for writing query engines in Rust that takes care of a lot of heavy lifting around parsing SQL, type casting, constructing and transforming query plans and optimizing them. It's pluggable, making it easy to write custom data sources, optimizer rules, query nodes etc. It's has very good single-node performance (there's even a way to compile it with SIMD support) and Ballista [1] extends that to build it into a distributed query engine. Plenty of other projects use it besides IOx, including VegaFusion, ROAPI, Cube.js's preaggregation store. We're heavily using it to build Seafowl [2], an analytical database that's optimized for running SQL queries directly from the user's browser (caching, CDNs, low latency, some WASM support, all that fun stuff). [0] https://github.com/apache/arrow-datafusion https://github.com/apache/arrow-datafusion [1] https://github.com/apache/arrow-ballista https://github.com/apache/arrow-ballista [2] https://github.com/splitgraph/seafowl https://github.com/splitgraph/seafowl

pauldix 4y ago
DataFusion is great, we're happy to be contributing to it. Also excited to see so many people around the world picking it up and contributing as well. With our development efforts on IOx, it's like a strong tailwind. But we put a ton of effort into helping manage community efforts (thanks, alamb! our developer on IOx that is also on the Arrow PMC).
- nevi-me 4y ago
  Your initial experiments and decision to build on arrow-rs has been great for the project. Thank you and everyone involved.
- andygrove 4y ago
  Original author of DataFusion/Ballista here. Having alamb and others from InfluxData involved has been a huge help in driving the project forward and helping build an active community behind the project. It is genuinely hard to keep up with the momentum these days!

toinbis 4y ago

Happy longtime Influxdb user here. I wanted to congratulate Paul and the team on reaching this milestone. Followed IOx development a bit - can't wait to finally test it out!

otoolep 4y ago

Congrats to the team at InfluxDB - great to see this released.

candrewlee14 4y ago

Congrats! Was always a pleasure to hear about IOx when I interned there last summer! They’re an awesome company to work for.

eskaytwo 4y ago

Will be very interesting to see this compared to Clickhouse

zX41ZdbW 4y ago
Here is a recent comparison: https://arxiv.org/pdf/2204.09795.pdf https://arxiv.org/pdf/2204.09795.pdf Although I'm not sure if it is using the new IOx engine or not.
- eskaytwo 4y ago
  Thanks. It’s odd they do such in-depth analysis but don’t mention the versions used. I think that is on the old engine. It would be very interesting to see how the new engine performs - I imagine the performance gets closer to something like Vaex which was using similar Apache tooling