5 ms·
How do you use it for visualization? Do you use ClickStack? or something else?
by oulipo2 3mo ago
How do you use it for visualization? Do you use ClickStack? or something else?
- usrme 3mo agoSame question here!
- jaysh 3mo agoJust replied to that question! Let me know if you have other questions.
- jaysh 3mo agoStill via Grafana. I ran it side-by-side with Loki and despite trying to optimise Loki and using ClickHouse out of the box - it really was shocking how much faster ClickHouse was for every single query (e.g. in the last 12 hours give my the frequency of logs with a particular JSON event or even "find this log entry, then join back and find the number of times a different entry appears within the same correlation_id)
- CubsFan1060 3mo agoWhat does the layout in click house look like? Do the input logs need to have a very defined structure?
- jaysh 3mo agoNot really, ClickHouse is super forgiving so you can do something like: CREATE TABLE default.events ( `timestamp` DateTime `event` String -- e.g. 'product.updated' or empty/null `message` -- human readable message `raw` -- the raw message - this is very useful when pushing logs that aren't JSON - you just let the `event` be null and dump the entire message here ) ENGINE = MergeTree PARTITION BY toDate(timestamp) ORDER BY (timestamp, event) TTL timestamp + toIntervalMonth(6) ClickHouse is extremely performant even in the cases of e.g.: SELECT count(*) FROM `events` WHERE `raw` LIKE '%hello world%' Of course, the more columns you splat out (e.g. like correlation_id, user_id, order_id, etc) the better you can index and expect those queries to perform but in general I don't bother outside the obvious core domain ones (exampled above), the performance is so good that unindexed queries are significantly faster than indexed queries in Loki. I have reached the point where I JSON extract on-the-fly for the WHERE clause with very large queries with no meaningful performance issues.
- oulipo2 3mo agoInteresting, so you can bind a Clickhouse table as an extension to Grafana? Would you make a little Gist / post about it to show?
- jaysh 3mo agoYou only need the plugin: https://clickhouse.com/docs/observability/grafana https://clickhouse.com/docs/observability/grafana - then you get basically everything natively.
- aleks_me2 3mo agoI have used SigNoz https://signoz.io/ https://signoz.io/ for that
- jauntywundrkind 3mo agoWorth noting both hyperdx and maple too for other observability on clickhouse options. https://www.hyperdx.io/ https://www.hyperdx.io/ https://maple.dev/ https://maple.dev/
- malkia 3mo agoThere is HyperDX - search is not fastest, but it could be something that we do too - haven't checked deeply if high-cardinality is big issue with ClickHouse, but seeing some high cardinality data with what we post.
- byoj 3mo agoWe recently moved to openobserve for due to cost, but visualisations are good enough too.