PSeitz

12 ms·

1.
by PSeitz 9mo ago
Hi, tantivy dev here. There are two recent performance improvements in tantivy, which should make term aggregations considerable faster. https://github.com/quickwit-oss/tantivy/pull/2740 https://gi
2.
by PSeitz 2y ago
Also we are working on some performance improvements based on the benchmark comparison, as they highlighted some areas we can improve in tantivy.
3.
by PSeitz 2y ago
tantivy has two dictionaries FST and SSTable. We added SSTable in tantivy because it works great with object storage, while FST does not. With some metadata we can download only the required parts and not the whole dictionary. SStable does
4.
by PSeitz 2y ago
You can just put everything in a JSON field in tantivy and set it to INDEXED and FAST
5.
by PSeitz 2y ago
They serve quite different use cases. quickwit was built to handle extremely large data volumes, you can ingest and search TB and PB of logs. meilisearches indexing doesn't scale as it will become slower the more data you have, e.g. I
6.
by PSeitz 2y ago
> Hm, I am interested, but I would love to use it as a rust lib and just have rust types instead of some json config... Yes that's how you use tantivy normally, not sure which json config you mean. tantivy-cli is more like a showcas
7.
by PSeitz 3y ago
The issue for geo search is here: https://github.com/quickwit-oss/tantivy/issues/44
8.
by PSeitz 3y ago
>Roaring bitmaps and similar data structures get their speed from decoding together consecutive groups of elements, so if you do sequential decoding or decode a large fraction of the list you get excellent performance. EF instead excels
9.
by PSeitz 3y ago
> It would definitely compress much better than roaring bitmaps. In terms of performance, it depends on the access patterns. If very sparse (large jumps), PEF would likely be faster, if dense (visit a large fraction of the bitmap), it&#x
10.
by PSeitz 3y ago
Indeed if you are colorblind that may be an issue. I'll address it If I get to it. Otherwise it assigns names to colors, no idea how you could be confused by that.
11.
Lz4_flex 0.11 Released: Faster LZ4 de/compression in Rust(flexineering.com)
7 points by PSeitz 3y ago|2 comments
12.
Tantivy 0.19: IP field, Faster indexing, Improved compression and aggregation(quickwit.io)
3 points by PSeitz 4y ago|0 comments
13.
How to investigate memory usage of your rust program(quickwit.io)
5 points by PSeitz 4y ago|0 comments
14.
by PSeitz 6y ago
file io should come with the frame format, which is not yet implemented. For the block format it's not really suited.
15.
Lz4_flex – fast LZ4 implementation in Rust(github.com)
3 points by PSeitz 6y ago|0 comments
16.
by PSeitz 6y ago
As far as I know LZ4 is much faster that most compression algorithms, with decompression speeds of over 4GB/s
17.
by PSeitz 6y ago
I ported the block format to Rust matching the C implementation in performance and ratio. https://github.com/pseitz/lz4_flex
18.
by PSeitz 7y ago
But I like coal, so coal can't be the problem either. ROLF!
19.
by PSeitz 7y ago
Or with einstein "yeah you were pretty much right. we don't have anything new"
20.
by PSeitz 8y ago
The performance comparisons between gc and non-gc languages show otherwise. Also, memory consumption eventually costs cpu cycles.
21.
by PSeitz 8y ago
Garbage collector is actually something you'd want, if it caused no performance hit.
22.
by PSeitz 8y ago
Some article about development a approach is not news. Also, hackernews is largely composed of non-news information, else I wouldn't be here ;)
23.
by PSeitz 9y ago
I use a state machine for text https://github.com/BurntSushi/fst
24.
by PSeitz 9y ago
But it does work in Tokyo, maybe it's the people
25.
by PSeitz 9y ago
Internet availability is not a problem. Just check cable availability and your up and running with 100Mbits+ in a few days.
26.
by PSeitz 9y ago
And Germans really like to complain.
27.
by PSeitz 10y ago
If your editor is your compiler, then something is seriously wrong.
28.
by PSeitz 10y ago
you always need at least compiler/interpreter to execute your code... unless write bytecode in your editor
29.
by PSeitz 10y ago
Seems to me, that apple bosses want to put their name on every patent created in their division.
30.
by PSeitz 10y ago
I heard that about Atom around 12 months ago.

More ›