mish15

8 ms·

1.
by mish15 3y ago
Yes it's all wrong, because: a) recall is designed to measure binary relevance, but vector scores are not good relevance judgments and they aren't binary. b) most models optimise purely for distance, which makes nDCG look great, b
2.
by mish15 3y ago
Yeah basically all the vector "database" solutions in market have chosen data-dependent indexes, so you need the data upfront. Imagine if regular databases needed all data upfront before they could build indexes. It's kind of
3.
by mish15 4y ago
On hybrid indexes with full text and vector support?
4.
by mish15 4y ago
Last I heard Pinecone doesn’t even support full text search, let alone hybrid indexes, what do you think you are disproving exactly? Real-time upserts on hybrid and vector indexes is very unusual, please link to how you do this.
5.
by mish15 5y ago
You pay a decent cost to do the hash, it’s a compression algorithm of sorts. But the data is a fraction of the size and comparison is way faster. If you do many of these or compare the same ones more than once you amortise the cost very qui
6.
by mish15 5y ago
Yes and yes. Network based hashing is great to maximise information quality of the hash (compared to other LSH methods). It works to compress existing vectors super efficiently. Very soon things like language embeddings will skip the vector
7.
by mish15 5y ago
We will add neural hash based ANN to that repo when we get time. I expect HNSW to get pushed out in time for many reasons.
8.
by mish15 5y ago
I was part of the above article. Happy to answer questions. In terms of accuracy, it totally depends on the resolution needed. We can get >99% accuracy of L2 waaaaay faster with 1/10 of the memory overhead. For what we are doing tha
9.
by mish15 5y ago
Yes. Storage also. You can get >99% ordering quality of exhaustive cosine with a tiny fraction of memory usage
10.
by mish15 5y ago
Interesting. Hadn’t seen it actually. BOW is problematic as a starting point, but this is neat and a long time ago too.
11.
by mish15 5y ago
This is easily the most fun thing I’ve been involved with for years. Can’t wait to see it ship.
12.
by mish15 6y ago
You just mmap memory as a slice. You can grow and shrink it, flush to disk if backed by a file, etc. In code it’s just a slice. Tough part is you need to allocate, control bounds, binary encode/decode, struct align, defrag, etc. It’s h
13.
by mish15 6y ago
You can do this in pure Go though, you just need to manage the allocator yourself. That is a significant technical undertaking, but once done the advantages are huge. No GC overhead, no cgo. Done properly you can even have multiple readers
14.
by mish15 7y ago
What search tech are you building it on out of interest?
15.
by mish15 7y ago
Totally agree. We looked at providing tech in this space and it means you have to plug into literally every system. It’s an endless problem. Slack search is awful and they have a full team on it. Jira is a mess. Gmail search is pretty good
16.
by mish15 7y ago
We have an early access product for personalized ecommerce search @Sajari if you are interested. One early access company is on track to generate $30 million in additional revenue from switching (over 10% search conversion increase). That i
17.
Machine assisted search ranking(medium.com)
3 points by mish15 9y ago|0 comments
18.
by mish15 10y ago
Unlikely, the GSA product is also discontinued. This is a push to a google cloud alternative i would guess.
19.
by mish15 10y ago
Their search appliance (GSA) is being replaced with a cloud search product. I would guess once that is released it will make the GSS product a bit redundant, so it makes sense to discontinue, the timing is a bit strange though...
20.
Spell checker in Golang, 40k words / sec at 60% accuracy(github.com)
2 points by mish15 12y ago|0 comments
21.
by mish15 15y ago
I'm amazed so many people and companies are focussed on trying to manage/limit peoples exposure to tracking technologies. Everyone is looking at this problem backwards. I say this because the vast majority of these technologies are javascri