wolfgarbe

6 ms·

1.
by wolfgarbe 7mo ago
Peter Norvig shows that an edit distance = 2 will cover 98.9% spelling errors. https://impythonist.wordpress.com/2014/03/18/peter-norvigs-2... That's the reason why the default maximum edit distance of S
2.
by wolfgarbe 7mo ago
Author of SymSpell here. Congrats on the launch of Lexiathan. Unfortunately, the comparison of Lexiathan vs. Symspell on your website regarding accuracy is misleading. 1. SymSpell has two parameters to control the maximum edit distance. Onc
3.
by wolfgarbe 10mo ago
The stopword list in SeekStorm is purely optional, per default it is empty. The query "to be or not to be" that you mentioned, consisting solely of stopwords, returns complete results and perform quite well in the benchmark: http
4.
by wolfgarbe 11mo ago
Can the index size exceed the RAM size (e.g., via memory mapping), or are index size and document number limited by RAM size? It would be good to mention those limitations in the README.
5.
by wolfgarbe 11mo ago
Sure, but it says "High-performance" Full Text Search Engine. Shouldn't that claim be backed up by numbers, comparing it to the state of the art?
6.
by wolfgarbe 11mo ago
Great work! Would be interesting to see how it compares to Lucene performance-wise, e.g. with a benchmark like https://github.com/quickwit-oss/search-benchmark-game
7.
by wolfgarbe 2y ago
The most widely used DHT is Kademlia from Petar Maymounkov and David Mazières. It is used in Ethereum, IPFS, I2P, Gnutella DHT, and many other applications. https://en.wikipedia.org/wiki/Kademlia https://pdo
8.
by wolfgarbe 2y ago
SeekStorm does currently not use io_uring, but it is on our roadmap. Challenges are the cross-platform compatibility. Linux (io_uring) and Windows (IoRing) use different implementations, and other OS don't support it. There is no abstr
9.
by wolfgarbe 2y ago
SeekStorm comes with an http interface. The SeekStorm server features an REST API via http: https://seekstorm.apidocumentation.com It also comes with an embedded Web UI: https://github.com/SeekStorm/SeekStor
10.
by wolfgarbe 2y ago
>> The documentation seems a bit sparse. We just released a new OpenAPI based documentation for the SeekStorm server REST API: https://seekstorm.apidocumentation.com For the library we have the standard rust doc: https:
11.
by wolfgarbe 2y ago
For the latency benchmarks we used vanilla BM25 (SimilarityType::Bm25f for a single field) for comparability, so there are no differences in terms of accuracy. For SimilarityType::Bm25fProximity which takes into account the proximity betwee
12.
by wolfgarbe 2y ago
The Seekstorm library is 9 MB, and the Seekstorm server executable is 8 MB, depending on the features selected in cargo. You add the library via 'cargo add seekstorm' to your project which you anyway have to compile. As for the se
13.
by wolfgarbe 2y ago
In SeekStorm you can choose per index whether to use Mmap or let SeekStorm fully control Ram access. There is a slight performance advantage to the latter, at the cost of higher index load time of the former. https://docs.rs&#x2F
14.
by wolfgarbe 2y ago
The benchmark should be fairly fair, as it was developed by Tantivy themselves (and Jason Wolfe). So, the choice of corpus and queries was theirs. But, of course, your mileage may vary. It is always best to benchmark it on your machine with
15.
by wolfgarbe 2y ago
Currently, you can choose between tokenizers with or without folding. But configurability per language or full customizability of the folding logic by the user is a good idea.
16.
by wolfgarbe 2y ago
The code for the distributed search cluster is not yet stable enough to be published, but it will be released as open-source as well. As for shared storage, do you mean something like NAS or, rather Amazon S3? Cloud-native support of object
17.
by wolfgarbe 2y ago
Its not just about speed. Speed reflects efficiency. Efficiency is needed to serve more queries in parallel, to search within exponentially growing data, with less expensive hardware, and fewer servers, consuming less energy. Therefore the
18.
by wolfgarbe 2y ago
PostgreSQL is an SQL database that also offers full-text search (FTS), with extensions like pg_search it also supports BM25 scoring which is essential for lexical search. SeekStorm is centered around full-text search only, it doesn't o
19.
by wolfgarbe 2y ago
Yes. We waited long for AOT compilation to become mature, to remove the need for the user to install the .Net framework. But two years ago when we decided to switch, we still couldn't just get the AOT compilation of our codebase to wor
20.
by wolfgarbe 2y ago
The 2-4 speed ratio was not meant to denounce C#, which is a great language I loved to program in for over two decades, coming from Delphi. Unfortunately, C# has not a complete SIMD support. See our request to support the SSE4.2 _mm_cmpistr
21.
by wolfgarbe 2y ago
We started with making the core search technology faster. Then we added a Unicode character folding/normalization tokenizer (diacritics, accents, umlauts, bold, italic, full-width chars...). Last week we added a tokenizer that supports
22.
by wolfgarbe 2y ago
Yes, integration in complex legacy systems is always challenging. As a small startup, we are concentrating on core search technology to make search faster and to make the most of available server infrastructure. As SeekStorm is open-source
23.
by wolfgarbe 2y ago
It depends on the application. When using SeekStorm as a server, keeping the latency per query low increases the throughput and the number of parallel queries a server can handle on top of a given hardware. An efficient search server can re
24.
Show HN: SeekStorm – open-source sub-millisecond search in Rust(github.com)
245 points by wolfgarbe 2y ago|61 comments
25.
by wolfgarbe 2y ago
Series of blog posts on search and information retrieval https://medium.com/@dtunkelang What AI Engineers Should Know about Search from Doug Turnbull https://softwaredoug.com/blog/2024/06/25&#
26.
by wolfgarbe 5y ago
This is no contradiction, you can concentrate on the algorithm also in a faster language :-) https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/
27.
by wolfgarbe 5y ago
A laudable effort. Two questions: 1. What is the rationale behind choosing Python as a implementation language? Performance and efficiency are paramount in keeping operational costs low and ensuring a good user experience even if the search
28.
by wolfgarbe 5y ago
If you click a tag on the result/preview page (in mobile Firefox), the result list is filtered by the tag (you get a new/shorter result list with an updated result number). But because you are still on the preview page you don&#x2
29.
by wolfgarbe 5y ago
The '-' operator has been fixed. 'vr -oculus' should work correctly now.
30.
by wolfgarbe 5y ago
Done. Now you can sort by newest/oldest date.

More ›