8 ms·

Quickwit 0.8: Indexing and Search at Petabyte Scale

115 points by vvoyer 3y ago

deleted 2y ago
[deleted]
dracyr 2y ago
Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage. I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by embedding it in Elixir, and the API and docs have been quite pleasant to work with. Hats of to the team, looking forward to what's coming next!
- francoismassot 2y ago
  Some companies are using it with AWS Lambda to scale to 0.
- tecleandor 2y ago
  Ah Loki, I wanted to try it at my homelab bit it wasn't as simple as it says. Now I wanted to try Zincsearch or Openobserve. Have you tried that?
  - netingle 2y ago
    > it wasn't as simple as it says mind elaborating? we built loki for some pretty massive scale but I've always tried to make it work at super small scale to. what went wrong?
  - pranay01 2y ago
    You might want to have a look at SigNoz [1] as well. We have also published some perf benchmark wrt Elastic & Loki [2] and have some cool features like logs pipeline for manipulating logs before ingestion [1] https://github.com/signoz/signoz https://github.com/signoz/signoz [2] https://signoz.io/blog/logs-performance-benchmark/ https://signoz.io/blog/logs-performance-benchmark/
  - bbkane 2y ago
    I use OpenObserve and I quite enjoy it
  - mdaniel 2y ago
    in case it matters to others, https://github.com/openobserve/openobserve/tree/v0.7.0 https://github.com/openobserve/openobserve/tree/v0.7.0 is the last Apache2 licensed copy before they went AGPL with 0.7.1 https://github.com/openobserve/openobserve/blob/v0.7.0/.env.example#L6 https://github.com/openobserve/openobserve/blob/v0.7.0/.env.... is some "onoz" for me, but just recently someone submitted https://github.com/aenix-io/etcd-operator https://github.com/aenix-io/etcd-operator to the CNCF sandbox so maybe things have gotten better around keeping that PoS alive
- nextaccountic 2y ago
  Tantivity is great! Here is a postgres extension that uses it to provide full text search https://blog.paradedb.com/pages/introducing_bm25 https://blog.paradedb.com/pages/introducing_bm25 https://news.ycombinator.com/item?id=37557127 https://news.ycombinator.com/item?id=37557127
  - francoismassot 2y ago
    tantivy, not tantivity!!!!!
up2isomorphism 2y ago
13.4GB/s with 200x6 vcpus, gives 11MB/s per core, it is good but hard to say impressive.
- francoismassot 2y ago
  Building the inverted index is quite CPU-intensive, and we are also merging index files called "splits".
  - kikimora 2y ago
    I never being able to understand why log indexing has to build inverted index. Decent columnar store with partitioning by date should be enough to quickly filter gigabytes of logs.
    - nh2 2y ago
      Because you want to find all occurrences of "error abc123" over the last year, immediately?
    - fulmicoton 2y ago
      Quickwit co-founder here... I actually agree. For a few GBs, done right, columnar works fine AND is cost efficient. After all, it does not matter much if a log search query answers in 300ms or 1s. However, there are use cases where a few GB just does not cut it. The tale saying that you can always prune your dataset using timestamp and tags is simply not always valid.
      - kikimora 2y ago
        Can you share your experience of when columnar fails? It is possible to scan NVMe at a speed of multiple GB/sec, scans can be parallel and happen on multiple disks, over compressed data (10 Gb of logs ~ 1Gb to scan), data can be segmented and prefaced with Blum filters, to quickly check if a segment is worth scanning.
halvorbo 2y ago
Amazing to see how far Tantiviy has come. Remember using and making some smaller contributions to this 3 years ago - slop to phrase queries for example. Curious how the design has changed to enable large scale production usage.
- francoismassot 2y ago
  Thanks! Quickwit is the distributed engine built on top of tantivy, we basically separated compute and storage for search, I wrote this blog post to introduce the architecture: https://quickwit.io/blog/quickwit-101 https://quickwit.io/blog/quickwit-101 PS: it’s tantivy!!!
- fulmicoton 2y ago
  Very valuable contribution!
godber 2y ago
We did some experimentation with quickwit about a year ago, writing about 1m docs/second of data into it for several months. It worked well and was pretty straight forward to learn and operate. If we didn’t also manage our own S3/Ceph it might be a big win, once feature complete. It’s definitely worth a look.
- lrx 2y ago
  I think you can use quickwit with a self-hosted S3-compatible object store.
arisudesu 2y ago
musl support would be highly appreciated.
- fulmicoton 2y ago
  We used to have one. Maybe we can revive it. What is your use case?

115 points by vvoyer 3y ago

deleted 2y ago

[deleted]

Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage. I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by embedding it in Elixir, and the API and docs have been quite pleasant to work with. Hats of to the team, looking forward to what's coming next!

francoismassot 2y ago
Some companies are using it with AWS Lambda to scale to 0.
tecleandor 2y ago
Ah Loki, I wanted to try it at my homelab bit it wasn't as simple as it says. Now I wanted to try Zincsearch or Openobserve. Have you tried that?
- netingle 2y ago
  > it wasn't as simple as it says mind elaborating? we built loki for some pretty massive scale but I've always tried to make it work at super small scale to. what went wrong?
- pranay01 2y ago
  You might want to have a look at SigNoz [1] as well. We have also published some perf benchmark wrt Elastic & Loki [2] and have some cool features like logs pipeline for manipulating logs before ingestion [1] https://github.com/signoz/signoz https://github.com/signoz/signoz [2] https://signoz.io/blog/logs-performance-benchmark/ https://signoz.io/blog/logs-performance-benchmark/
- bbkane 2y ago
  I use OpenObserve and I quite enjoy it
- mdaniel 2y ago
  in case it matters to others, https://github.com/openobserve/openobserve/tree/v0.7.0 https://github.com/openobserve/openobserve/tree/v0.7.0 is the last Apache2 licensed copy before they went AGPL with 0.7.1 https://github.com/openobserve/openobserve/blob/v0.7.0/.env.example#L6 https://github.com/openobserve/openobserve/blob/v0.7.0/.env.... is some "onoz" for me, but just recently someone submitted https://github.com/aenix-io/etcd-operator https://github.com/aenix-io/etcd-operator to the CNCF sandbox so maybe things have gotten better around keeping that PoS alive
nextaccountic 2y ago
Tantivity is great! Here is a postgres extension that uses it to provide full text search https://blog.paradedb.com/pages/introducing_bm25 https://blog.paradedb.com/pages/introducing_bm25 https://news.ycombinator.com/item?id=37557127 https://news.ycombinator.com/item?id=37557127
- francoismassot 2y ago
  tantivy, not tantivity!!!!!

up2isomorphism 2y ago

13.4GB/s with 200x6 vcpus, gives 11MB/s per core, it is good but hard to say impressive.

francoismassot 2y ago
Building the inverted index is quite CPU-intensive, and we are also merging index files called "splits".
- kikimora 2y ago
  I never being able to understand why log indexing has to build inverted index. Decent columnar store with partitioning by date should be enough to quickly filter gigabytes of logs.
  - nh2 2y ago
    Because you want to find all occurrences of "error abc123" over the last year, immediately?
  - fulmicoton 2y ago
    Quickwit co-founder here... I actually agree. For a few GBs, done right, columnar works fine AND is cost efficient. After all, it does not matter much if a log search query answers in 300ms or 1s. However, there are use cases where a few GB just does not cut it. The tale saying that you can always prune your dataset using timestamp and tags is simply not always valid.
    - kikimora 2y ago
      Can you share your experience of when columnar fails? It is possible to scan NVMe at a speed of multiple GB/sec, scans can be parallel and happen on multiple disks, over compressed data (10 Gb of logs ~ 1Gb to scan), data can be segmented and prefaced with Blum filters, to quickly check if a segment is worth scanning.

halvorbo 2y ago

Amazing to see how far Tantiviy has come. Remember using and making some smaller contributions to this 3 years ago - slop to phrase queries for example. Curious how the design has changed to enable large scale production usage.

francoismassot 2y ago
Thanks! Quickwit is the distributed engine built on top of tantivy, we basically separated compute and storage for search, I wrote this blog post to introduce the architecture: https://quickwit.io/blog/quickwit-101 https://quickwit.io/blog/quickwit-101 PS: it’s tantivy!!!
fulmicoton 2y ago
Very valuable contribution!

godber 2y ago

We did some experimentation with quickwit about a year ago, writing about 1m docs/second of data into it for several months. It worked well and was pretty straight forward to learn and operate. If we didn’t also manage our own S3/Ceph it might be a big win, once feature complete. It’s definitely worth a look.

lrx 2y ago
I think you can use quickwit with a self-hosted S3-compatible object store.

arisudesu 2y ago

musl support would be highly appreciated.

fulmicoton 2y ago
We used to have one. Maybe we can revive it. What is your use case?