5 ms·

You don't. You use a full-text indexer and normal search tools. A chatbot is only going to decrease the integrity of query results.

vikramkr 2y ago
You wouldn't use a chatbot for the same query you'd use normal search tools for (and on a side note your answer would be much more useful with an example of what those tools would be, it's not really actionable). A vague natural language question over data whose structure you haven't fully understood using terms that might be inexact is not as likely to provide good results with normal search tools as with an llm based tool.
- skydhash 2y ago
  > your answer would be much more useful with an example of what those tools would be Paperless, DevonThink, even Calibre (the ebook manager) can do it. You only need a day or two to categorize the documents. No need for huge amounts of RAM, or privacy concerns, or hallucinated answers.
  - dotancohen 2y ago
    > You only need a day or two For some of us, for some types of data, huge amounts of RAM, or even privacy concerns, or even the occasional hallucinated answer, is an easier pill to swallow. A recent example, maybe not the best example but recent, was the query "What do the three headed dog from the Harry Potter books and the cat from Alien have in common"
    - brudgers 2y ago
      They are fictional.
  - xeromal 2y ago
    I never want to categorize stuff. I want it done for me.
- ajsnigrutin 2y ago
  Another (ugly but works nice): https://www.recoll.org/pics/index.html https://www.recoll.org/pics/index.html opensource, local, yada yada, almost zero configuration (just add folders, run indexer, wait).
andai 2y ago
I found that grep actually outperformed vector search for many queries. The only thing I was missing was when I didn't know how exactly to phrase something (the exact keyword to use). Do keyword search systems have workarounds for this? My own idea was for each keyword to generate a list of neighbor keywords in semantic space. I figured with such a dataset, I'd get something approximating vector search for free. I made some attempts at that (found neighbors by their proximity in text), but I ended up with a lot of noise (words that often go together without having the same meaning). So I'd probably have to use actual embeddings instead. More generally, any suggestions for full-text indexing? Elasticsearch seems like overkill. I built my own keyword search in Python (simple tf-idf) which was surprisingly easy. (Long-term project is to have an offline copy of a useful/interesting subset of the internet. Acquiring the datasets is also an open question. Common Crawl is mostly random blogs and forum arguments...)
- skydhash 2y ago
  > The only thing I was missing was when I didn't know how exactly to phrase something (the exact keyword to use). I think that's the only things GUI (or TUI) directories have over CLI. I remember having Wikipedia locally (english texts, back in 2010) and the portals were surprisingly useful. They act like the semantic space in case you can't find an article for your exact word. So Literature > Fiction > Fantasy > Epic Fantasy will probably land you somewhere close to "The Lord of The Rings".
- SkyPuncher 2y ago
  Most developers are going to outperform vector search. We “get” how computers do lookups so we build our queries appropriately. Vector search is amazing for using layman concepts.
- j0hnyl 2y ago
  The point of vector search is to support semantic search. It makes sense that grep will outperform if you're just looking for verbatim occurrences of a string.
  - 3abiton 2y ago
yreg 2y ago
> decrease the integrity of query results What does that even mean. When you know the exact keywords then you use full-text. When you don't know them then other tools can be helpful.
- eviks 2y ago
  It means you'd use the same tool since it's more convenient and get worse results in one tool vs. the other
  - Capricorn2481 2y ago
    Because they're two different tools for two different tasks. If you expect to always know the exact phrase than, yes, grep will be better. But if you search a semantically similar phrase you will get nothing
rahimnathwani 2y ago
Paperless-ngx set up using docker compose is good for this use case.
barrenko 2y ago
Hi bastien, Could you expand on the answer? Thanks!

vikramkr 2y ago

You wouldn't use a chatbot for the same query you'd use normal search tools for (and on a side note your answer would be much more useful with an example of what those tools would be, it's not really actionable). A vague natural language question over data whose structure you haven't fully understood using terms that might be inexact is not as likely to provide good results with normal search tools as with an llm based tool.

skydhash 2y ago
> your answer would be much more useful with an example of what those tools would be Paperless, DevonThink, even Calibre (the ebook manager) can do it. You only need a day or two to categorize the documents. No need for huge amounts of RAM, or privacy concerns, or hallucinated answers.
- dotancohen 2y ago
  > You only need a day or two For some of us, for some types of data, huge amounts of RAM, or even privacy concerns, or even the occasional hallucinated answer, is an easier pill to swallow. A recent example, maybe not the best example but recent, was the query "What do the three headed dog from the Harry Potter books and the cat from Alien have in common"
  - brudgers 2y ago
    They are fictional.
- xeromal 2y ago
  I never want to categorize stuff. I want it done for me.
ajsnigrutin 2y ago
Another (ugly but works nice): https://www.recoll.org/pics/index.html https://www.recoll.org/pics/index.html opensource, local, yada yada, almost zero configuration (just add folders, run indexer, wait).

andai 2y ago

I found that grep actually outperformed vector search for many queries. The only thing I was missing was when I didn't know how exactly to phrase something (the exact keyword to use). Do keyword search systems have workarounds for this? My own idea was for each keyword to generate a list of neighbor keywords in semantic space. I figured with such a dataset, I'd get something approximating vector search for free. I made some attempts at that (found neighbors by their proximity in text), but I ended up with a lot of noise (words that often go together without having the same meaning). So I'd probably have to use actual embeddings instead. More generally, any suggestions for full-text indexing? Elasticsearch seems like overkill. I built my own keyword search in Python (simple tf-idf) which was surprisingly easy. (Long-term project is to have an offline copy of a useful/interesting subset of the internet. Acquiring the datasets is also an open question. Common Crawl is mostly random blogs and forum arguments...)

skydhash 2y ago
> The only thing I was missing was when I didn't know how exactly to phrase something (the exact keyword to use). I think that's the only things GUI (or TUI) directories have over CLI. I remember having Wikipedia locally (english texts, back in 2010) and the portals were surprisingly useful. They act like the semantic space in case you can't find an article for your exact word. So Literature > Fiction > Fantasy > Epic Fantasy will probably land you somewhere close to "The Lord of The Rings".
SkyPuncher 2y ago
Most developers are going to outperform vector search. We “get” how computers do lookups so we build our queries appropriately. Vector search is amazing for using layman concepts.
j0hnyl 2y ago
The point of vector search is to support semantic search. It makes sense that grep will outperform if you're just looking for verbatim occurrences of a string.
- 3abiton 2y ago

yreg 2y ago

> decrease the integrity of query results What does that even mean. When you know the exact keywords then you use full-text. When you don't know them then other tools can be helpful.

eviks 2y ago
It means you'd use the same tool since it's more convenient and get worse results in one tool vs. the other
- Capricorn2481 2y ago
  Because they're two different tools for two different tasks. If you expect to always know the exact phrase than, yes, grep will be better. But if you search a semantically similar phrase you will get nothing

rahimnathwani 2y ago

Paperless-ngx set up using docker compose is good for this use case.

barrenko 2y ago

Hi bastien, Could you expand on the answer? Thanks!