fulmicoton

24 ms·

1.
by fulmicoton 1y ago
One trouble I could see with your approach is that you treat the information "Doc at pos i" beats "Doc at pos j" independently from i and j. Intuitively, it is not as critical when a bad doc is at rank 9 instead of rank
2.
by fulmicoton 2y ago
This bug hit us, and yes, I hadn't thought of just switching to opendal. That's indeed a great reminder.
3.
by fulmicoton 2y ago
No. Quickwit was founded well before Warpstream and it did not inspire us. The Husky blog post was released after we released a few versions of quickwit if I recall correctly. It was not an inspiration either. As far as I know, the similari
4.
by fulmicoton 2y ago
Our seed round was 100% made of SAFE, so VCs did not have the power to force us to do anything. The sentence in the blog post is a tad misleading. I suspect François is not really talking about VCs that had already invested in quickwit, but
5.
by fulmicoton 2y ago
Developer of tantivy chiming in! (I hope that's ok) Database performance is a space where there are a lot of lies and bullshit, so you are 100% right to be suspicious. I don't know SeekStorm's team and I did not dig much into
6.
by fulmicoton 2y ago
Yes. We should shut down this demo. We reduced the hardware to cut down our costs. Right now it runs a ludicrously small amount of hardware.
7.
by fulmicoton 2y ago
Quickwit is targetting logs: - it does not do vector search. It can rank docs using BM25, but usually people just want to sort by timestamp. - its does not use an SSD cache. Quickwit reads directly into the object storage. - i
8.
by fulmicoton 2y ago
This is NOT about transaction log. This is application logs. The thing you generate via Log4j for instance. Also 100PB is measured as the input format (JSON). Internally Quickwit will have more efficient representations.
9.
by fulmicoton 2y ago
Security and customer support are the two main reasons why people want a super long retention. Medium retention (1 or 2 months) is still very appreciable if some issue in your bugtracker stay stale for this amount of time.
10.
by fulmicoton 2y ago
It is pretty much the same as Lucene. The compression ratio is very specific logs and depends on the logs themselves. (Often it is not that good)
11.
by fulmicoton 2y ago
Quickwit (like Elasticsearch/Opensearch) stores you data compressed with ZSTD in a row store, builds a full text search index, and stores some of your fields in a columnar. The "compressed size" includes all of this. The high
12.
by fulmicoton 2y ago
Again, this is application logs. The stuff you would log in your program with log4j for instance. With a microservices architecture in particular that can pile up rapidly.
13.
by fulmicoton 2y ago
Thank you for the kind word @ZeroCool2u ! :)
14.
by fulmicoton 2y ago
Building an inverted index is actually very cpu intensive. I think we are the fastest on that (if someone knows something faster than tantivy at indexing I am interested). I'd be really surprised if you can make a 10x improvement here.
15.
by fulmicoton 2y ago
If you can limit your research to GBs of logs, I kind of agree with you. It's ok if a log search request takes 100ms instead of 2s, and the "grep" approach is more flexible. Usually our users search into > 1TB. Let's
16.
by fulmicoton 2y ago
The data is just Binance's application logs for observability. Typically what a smaller business would simply send to Datadog. This log search infra is handled by two engineers who do that for the entire company. They have some standa
17.
by fulmicoton 2y ago
Quickwit is designed to do full-text search efficiently with an index stored on an object storage. There are no equivalent technology, apart maybe: - Chaossearch but it is hard to tell because they are not opensource and do not share their
18.
by fulmicoton 2y ago
This is their application logs. They need to search into it in a comfortable manner. They went for a search engine with Elasticsearch at first, and Quickwit after that because even after restriction the search on a tag and a time window &qu
19.
by fulmicoton 2y ago
Thank you @tyler!!!
20.
by fulmicoton 2y ago
Thank you so much for sharing!!!
21.
by fulmicoton 2y ago
Yes
22.
A math puzzle and a better algorithm for top-K(quickwit.io)
4 points by fulmicoton 2y ago|2 comments
23.
by fulmicoton 2y ago
A math puzzle, its relationship with the average case complexity of computing top-K using a min heap, and a simple algorithm that performs better.
24.
by fulmicoton 2y ago
We used to have one. Maybe we can revive it. What is your use case?
25.
by fulmicoton 2y ago
What we do is CPU bound and we are not just parsing JSON here. The largest work we do is building an inverted index. Oversimplified, it is equivalent to this: inverted_index = defaultdict(list) for (doc_id, doc_json) in enumerate(doc_
26.
by fulmicoton 2y ago
Quickwit co-founder here... I actually agree. For a few GBs, done right, columnar works fine AND is cost efficient. After all, it does not matter much if a log search query answers in 300ms or 1s. However, there are use cases where a few GB
27.
by fulmicoton 2y ago
Very valuable contribution!
28.
by fulmicoton 2y ago
What is your frame of reference?
29.
by fulmicoton 3y ago
Quickwit is very similar to what is described here. Unfortunately, the files are not in Parquet so even though Quickwit is opensource, it is difficult to tap into the file format. We did not pick Parquet because we want to actually be able
30.
by fulmicoton 3y ago
That looks quite promising! Thank you for crediting tantivy in the github README, that's well appreciated! Ping me if I can help with anything.

More ›