6 ms·

That article takes various concepts from typical TSDB solutions and seemingly only looks at the bad sides. Time series data has many different forms, not every

That article takes various concepts from typical TSDB solutions and seemingly only looks at the bad sides. Time series data has many different forms, not every form works for every TSDB solution. For the 3 caveats at the top, there are already two TS solutions that look promising (QuestDB, TimescaleDB). Often an operational analytics DB (Clickhouse, CrateDB) might also be a solution.

jpgvm 5y ago
If you want something like Honeycomb but scales better then maybe look at Druid.
- dikei 5y ago
  Last time I checked, Druid were not very good at ad-hoc tasks because it lacked join and SQL supports was sketchy. How is it now ?
  - jpgvm 5y ago
    Limited JOIN support. SQL is now very good. JOINs vs no JOINs isn't an adhoc vs not-adhoc thing but more of a schema thing. If you try jam a star schema into it you aren't going to have a good time. This is true for pretty all of these more optimised stores. If you have a star schema and want to do those sorts of queries (and performance or cost aren't your #1 driving factors) then the better tool is a traditional data warehouse like BigQuery. This probably won't be the case forever though, there is significant progress in the Presto/Trino world to enable push-down for stores like Druid which would allow you to source most of your fact table data from other sources and then join into your events/time-orientated data from Druid very efficiently.
camel_gopher 5y ago
Take a look at IronDB too. High scale distributed implementationd.
oconnore 5y ago
> Often an operational analytics DB (Clickhouse, CrateDB) might also be a solution This might be a bit off topic, but speaking of gaps in common observability tooling: is an OLAP database a common go-to for longer-timescale analytics (as in [1])? We're using BigQuery, but on ~600GB of log/event data I start hitting memory limits even with fairly small analytical windows. In this context I have seen other references to: Sawzall (google), Lingo (google), MapReduce/Pig/Cascading/Scalding. Are people using Spark for this sort of thing now? Perhaps a combined workflow would be ideal: filter/group/extract interesting data in Hadoop/Spark, and then load into OLAP for ad-hoc querying? [1]: https://danluu.com/metrics-analytics/ https://danluu.com/metrics-analytics/
- proddata 5y ago
  > is an OLAP database a common go-to for longer-timescale analytics (as in [1])? I would not consider Clickhouse or CrateDB "classic" OLAP DBs. I can speak for CrateDB (I work there), that it definitely would be able to handle 600GB and query across it in an ad-hoc manner. We have users ingesting Terabytes of events per day and run aggregations across 100 Terabyte.
  - nicoburns 5y ago
    What kind of hardware requirements would be needed to store and query this much data?
    - proddata 5y ago
      - Depends - Just inserting, indexing, storing and simple querying can be done with little memory (i.e. 1:500 memory-disk-ratio 0.5GB RAM per 1TB disk). Typical production clusters with high query load are in the 1:150 range i.e. 64GB RAM for 10TB disk). Otherwise typical general purpose hardware (Standard SSDs, 1:4 vCPU:memory ratios, ...)
      - claytonjy 5y ago
        Wouldn't 0.5GB RAM per 1TB disk be more like 1:2000 memory-disk-ratio? Which is even better!
alfiedotwtf 5y ago
What are the best books out there to learn about Time Series databases? (there are already a million for relational and graph, but haven't seen one for time series). Bonus on how to implement one
- dominotw 5y ago
  CMU did a timeseries series couple of years ago: https://www.youtube.com/playlist?list=PLSE8ODhjZXjY0GMWN4X8FIkYNfiu8_Wl9 https://www.youtube.com/playlist?list=PLSE8ODhjZXjY0GMWN4X8F... Things have changed a little bit now , but not much.
  - alfiedotwtf 5y ago
    Excellent, thanks
akulkarni 5y ago
(TimescaleDB co-founder) Thanks for the mention, and I completely agree :-) Personally, there is a lot in this article that is misguided. For example, it essentially defines "time-series database" as "metric store." As TimescaleDB users know, TimescaleDB handles a lot more than just metrics. In fact, we handle any of the data types that Postgres can handle, which I suspect is more than what Honeycomb's custom store supports. TSDBs are good at what they do, but high cardinality is not built into the design. The wrong tag (or simply having too many tags) leads to a combinatorial explosion of storage requirements. This is a broad generalization. Some time-series databases are better at high cardinality than others. Also, what is "high-cardinality" - 100K? 1M? 10M? (We in fact are designed for _higher cardinalities_ than most other time-series databases [0]) In contrast, our distributed column store optimizes for storing raw, high-cardinality data from which you can derive contextual traces. This design is flexible and performant enough that we can support metrics and tracing using the same backend. The same cannot be said of time series databases, though, which are hyper-specialized for a specific type of data. We just launched tracing and metrics support in the same backend - in Promscale, built on TimescaleDB [1] I do commend the folks at Honeycomb for having a good product loved by some of my colleagues (at other companies). I also commend them for attempting to write an article aimed to educate. But I wish they had done more research - because without it, this article (IMO) ends up confusing more than educating. For anyone curious on our definition of "time-series data" and "time-series databases": https://blog.timescale.com/blog/what-the-heck-is-time-series-data-and-why-do-i-need-a-time-series-database-dcf3b1b18563/ https://blog.timescale.com/blog/what-the-heck-is-time-series... [0] https://blog.timescale.com/blog/what-is-high-cardinality-how-do-time-series-databases-influxdb-timescaledb-compare/ https://blog.timescale.com/blog/what-is-high-cardinality-how... [1] https://blog.timescale.com/blog/what-are-traces-and-how-sql-yes-sql-and-opentelemetry-can-help-us-get-more-value-out-of-traces-to-build-better-software/ https://blog.timescale.com/blog/what-are-traces-and-how-sql-...
- eska 5y ago
  I had a serious case of deja vu reminding me of your article on compression in timescaledb :-D
  - 5y ago

jpgvm 5y ago

If you want something like Honeycomb but scales better then maybe look at Druid.

dikei 5y ago
Last time I checked, Druid were not very good at ad-hoc tasks because it lacked join and SQL supports was sketchy. How is it now ?
- jpgvm 5y ago
  Limited JOIN support. SQL is now very good. JOINs vs no JOINs isn't an adhoc vs not-adhoc thing but more of a schema thing. If you try jam a star schema into it you aren't going to have a good time. This is true for pretty all of these more optimised stores. If you have a star schema and want to do those sorts of queries (and performance or cost aren't your #1 driving factors) then the better tool is a traditional data warehouse like BigQuery. This probably won't be the case forever though, there is significant progress in the Presto/Trino world to enable push-down for stores like Druid which would allow you to source most of your fact table data from other sources and then join into your events/time-orientated data from Druid very efficiently.

camel_gopher 5y ago

Take a look at IronDB too. High scale distributed implementationd.

oconnore 5y ago

> Often an operational analytics DB (Clickhouse, CrateDB) might also be a solution This might be a bit off topic, but speaking of gaps in common observability tooling: is an OLAP database a common go-to for longer-timescale analytics (as in [1])? We're using BigQuery, but on ~600GB of log/event data I start hitting memory limits even with fairly small analytical windows. In this context I have seen other references to: Sawzall (google), Lingo (google), MapReduce/Pig/Cascading/Scalding. Are people using Spark for this sort of thing now? Perhaps a combined workflow would be ideal: filter/group/extract interesting data in Hadoop/Spark, and then load into OLAP for ad-hoc querying? [1]: https://danluu.com/metrics-analytics/ https://danluu.com/metrics-analytics/

proddata 5y ago
> is an OLAP database a common go-to for longer-timescale analytics (as in [1])? I would not consider Clickhouse or CrateDB "classic" OLAP DBs. I can speak for CrateDB (I work there), that it definitely would be able to handle 600GB and query across it in an ad-hoc manner. We have users ingesting Terabytes of events per day and run aggregations across 100 Terabyte.
- nicoburns 5y ago
  What kind of hardware requirements would be needed to store and query this much data?
  - proddata 5y ago
    - Depends - Just inserting, indexing, storing and simple querying can be done with little memory (i.e. 1:500 memory-disk-ratio 0.5GB RAM per 1TB disk). Typical production clusters with high query load are in the 1:150 range i.e. 64GB RAM for 10TB disk). Otherwise typical general purpose hardware (Standard SSDs, 1:4 vCPU:memory ratios, ...)
    - claytonjy 5y ago
      Wouldn't 0.5GB RAM per 1TB disk be more like 1:2000 memory-disk-ratio? Which is even better!

alfiedotwtf 5y ago

What are the best books out there to learn about Time Series databases? (there are already a million for relational and graph, but haven't seen one for time series). Bonus on how to implement one

dominotw 5y ago
CMU did a timeseries series couple of years ago: https://www.youtube.com/playlist?list=PLSE8ODhjZXjY0GMWN4X8FIkYNfiu8_Wl9 https://www.youtube.com/playlist?list=PLSE8ODhjZXjY0GMWN4X8F... Things have changed a little bit now , but not much.
- alfiedotwtf 5y ago
  Excellent, thanks

akulkarni 5y ago

(TimescaleDB co-founder) Thanks for the mention, and I completely agree :-) Personally, there is a lot in this article that is misguided. For example, it essentially defines "time-series database" as "metric store." As TimescaleDB users know, TimescaleDB handles a lot more than just metrics. In fact, we handle any of the data types that Postgres can handle, which I suspect is more than what Honeycomb's custom store supports. TSDBs are good at what they do, but high cardinality is not built into the design. The wrong tag (or simply having too many tags) leads to a combinatorial explosion of storage requirements. This is a broad generalization. Some time-series databases are better at high cardinality than others. Also, what is "high-cardinality" - 100K? 1M? 10M? (We in fact are designed for _higher cardinalities_ than most other time-series databases [0]) In contrast, our distributed column store optimizes for storing raw, high-cardinality data from which you can derive contextual traces. This design is flexible and performant enough that we can support metrics and tracing using the same backend. The same cannot be said of time series databases, though, which are hyper-specialized for a specific type of data. We just launched tracing and metrics support in the same backend - in Promscale, built on TimescaleDB [1] I do commend the folks at Honeycomb for having a good product loved by some of my colleagues (at other companies). I also commend them for attempting to write an article aimed to educate. But I wish they had done more research - because without it, this article (IMO) ends up confusing more than educating. For anyone curious on our definition of "time-series data" and "time-series databases": https://blog.timescale.com/blog/what-the-heck-is-time-series-data-and-why-do-i-need-a-time-series-database-dcf3b1b18563/ https://blog.timescale.com/blog/what-the-heck-is-time-series... [0] https://blog.timescale.com/blog/what-is-high-cardinality-how-do-time-series-databases-influxdb-timescaledb-compare/ https://blog.timescale.com/blog/what-is-high-cardinality-how... [1] https://blog.timescale.com/blog/what-are-traces-and-how-sql-yes-sql-and-opentelemetry-can-help-us-get-more-value-out-of-traces-to-build-better-software/ https://blog.timescale.com/blog/what-are-traces-and-how-sql-...

eska 5y ago
I had a serious case of deja vu reminding me of your article on compression in timescaledb :-D
- 5y ago