8 ms·
Ask HN: I have many PDFs – what is the best local way to leverage AI for search?
As the title says, I have many PDFs - mostly scans via Scansnap - but also non-scans. These are sensitive in nature, e.g. bills, documents, etc. I would like a local-first AI solution that allows me to say things like: "show me all tax documents for August 2023" or "show my home title". Ideally it is Mac software that can access iCloud too, since that where I store it all. I would prefer to not do any tagging. I would like to optimize on recall over precision, so False Positives in the search results are ok. What are modern approaches to do this, without hacking one up on my own?
- adyashakti 2y agogetcody.ai
- borg16 2y agothe op wanted a local method, and this does not seem to be local
- dudus 2y agoI tried Google's NotebookLM for this use case and was very pleased with the experience. If you trust Google that is.
- hobo_mark 2y agoNotebookLM is currently US only, limited to 20 documents (sorry, 'sources') per notebook, and only works with Google Drive.
- bendsawyer 2y agoNot offline. I do not trust anyone with some data, because I have contractually promised not to do so.
- gibsonf1 2y agohttps://graphmetrix.com/trinpod-server https://graphmetrix.com/trinpod-server
- finack 2y agoOCR and pattern matching on text are computationally cheap and incredibly easy to do. For example, tax documents often bear the name of your government's tax authority, which presumably you are familiar with and can search for. They also tend to have years on them.
- hiq 2y agoThis. I wanted to convert some equations from some maths textbook back into latex, and I found that taking a screenshot and feeding the image into some LLM service supporting images was a good way to do that.
- m0shen 2y agoPaperless supports OCR + full text indexing: https://docs.paperless-ngx.com/ https://docs.paperless-ngx.com/ As far as AI goes, not sure.
- whynotmaybe 2y agoYou can use Gpt4all with localdocs to analyze the folder where you store the output of paperless-ngx
- elrostelperien 2y agoFor macOS, there's this: https://pdfsearch.app/ https://pdfsearch.app/ Without AI, but searching the PDF content, I use Recoll (https://www.recoll.org/ https://www.recoll.org/) or ripgrep-all (https://github.com/phiresky/ripgrep-all https://github.com/phiresky/ripgrep-all)
- gumboshoes 2y agoThe best indexer for macOS, bar none is Foxtrot Professional. https://foxtrot-search.com/foxtrot-professional.html https://foxtrot-search.com/foxtrot-professional.html Very sophisticated searching, including regex, its own query language, and proximity searches - x within z words of y - which for me is the biggest win. I have 2TB of files indexed with this.
- 1123581321 2y agoDevonthink would do this with a tiny model to translate your natural length search prompts into its syntax and your folder/tag tree. If you're okay with some false positives, Devonthink would work as is, actually.
- bendsawyer 2y agoI used to use this, but the LLM approach allows for much deeper interactions. Not "find all times I've typed X" but "act as an expert in Y, looking across all times I've typed X, summarize my changing position over thee years, and suggest other terms that have a similar pattern of change, in a list." The kind of thing I used to give to an intern over a month, with results that are not far off what that intern produced...
- 1123581321 2y agoI’d love to see that built in as well. The devonthink crab-bucket community is hostile to any use of LLMs but I don’t think they understand how the app would structure and augment the input and output to keep it from returning fanciful output.
- bendsawyer 2y agoI looked into this for sensitive material recently. In the end I got a purpose-built local system built and am having it remotely maintained. Cost: around 5k a year. I used http://www.skunkwerx.ai http://www.skunkwerx.ai, who are US based. The result is a huge step up from 'full text search' solutions, for my use case. I can have conversations with decades of documents, and it's incredibly helpful. The support scheme keeps my original documents unconnected from the machine, which I own, while updates are done over a remote link. It's great, and I feel safe. Things change so fast in this space that there did not seem to be a cheap, stable, local alternative. I honestly doubt one is coming. This is not a on-size-fits-all problem.
- Kikawala 2y agoQuivr: https://github.com/QuivrHQ/quivr https://github.com/QuivrHQ/quivr SecureAI-Tools: https://github.com/SecureAI-Tools/SecureAI-Tools https://github.com/SecureAI-Tools/SecureAI-Tools
- pierre 2y agoRAG cli from llamaindex, allow you to do it 100% locally when used with ollama or llamacpp instead of OpenAI. https://docs.llamaindex.ai/en/stable/getting_started/starter_tools/rag_cli/ https://docs.llamaindex.ai/en/stable/getting_started/starter...
- homarp 2y agoand at some point (https://github.com/ggerganov/llama.cpp/issues/7444 https://github.com/ggerganov/llama.cpp/issues/7444) you will be able to use Phi-3-vision https://huggingface.co/microsoft/Phi-3-vision-128k-instruct https://huggingface.co/microsoft/Phi-3-vision-128k-instruct but for now you will have to use python. You can try it here https://ai.azure.com/explore/models/Phi-3-vision-128k-instruct/version/1/registry/azureml https://ai.azure.com/explore/models/Phi-3-vision-128k-instru... to get an idea of its OCR + QA abilities
- ekianjo 2y agollamaindex has an horrible API, very poor docs and is constantly changing. I do not recommend it.
- papichulo2023 2y agoAny alternative?
- hm-nah 2y agoVanilla python
- dmd 2y agoSo your solution to “I don’t like flying [specific airline]” would be “how about a big pile of aluminum and some jet fuel”?
- vladsanchez 2y agoLOL `papichulo`? Que tigre!?
- yousnail 2y agoPrivateGPT is a great starting point for using a local model and RAG. Text-generation-ui, oogabooga, using superbooga V2 is very nice and more customizable. I’ve used both for sensitive internal SOPs, and both work quite well. Private gpt excels at ingesting many separate documents, the other excels at customization. Both are totally offline, and can use mostly whatever models you want.
- edgyquant 2y agoUsing python to dump the PDF to text then use llama3 (8B) to parse
- nl 2y agoThe "Using python to dump the PDF to text" dramatically underestimates how hard this is. Tables and especially multi-column PDFs often need one-off handling and - worse - you don't know when one is being misparsed until you start getting weird search results. At that point you need to debug your entire search pipeline, which isn't fun!
- bastien2 2y agoYou don't. You use a full-text indexer and normal search tools. A chatbot is only going to decrease the integrity of query results.
- vikramkr 2y agoYou wouldn't use a chatbot for the same query you'd use normal search tools for (and on a side note your answer would be much more useful with an example of what those tools would be, it's not really actionable). A vague natural language question over data whose structure you haven't fully understood using terms that might be inexact is not as likely to provide good results with normal search tools as with an llm based tool.
- skydhash 2y ago> your answer would be much more useful with an example of what those tools would be Paperless, DevonThink, even Calibre (the ebook manager) can do it. You only need a day or two to categorize the documents. No need for huge amounts of RAM, or privacy concerns, or hallucinated answers.
- dotancohen 2y ago> You only need a day or two For some of us, for some types of data, huge amounts of RAM, or even privacy concerns, or even the occasional hallucinated answer, is an easier pill to swallow. A recent example, maybe not the best example but recent, was the query "What do the three headed dog from the Harry Potter books and the cat from Alien have in common"
- brudgers 2y agoThey are fictional.
- xeromal 2y agoI never want to categorize stuff. I want it done for me.
- sciencesama 2y agoYou can tabulate the info 90% of your info will be from single source. There are online tools that sort costco and walmart bills !!
- gandalfthepink 2y agoI use Curiosity AI. Good interface.
- brailsafe 2y agoLike many others have suggested, local indexing is what I use for this, although some more natural interface may be better for structured search and querying. What I haven't seen suggested though, is the built-in spotlight. Press CMD+Space, type some unique words that might appear in the document, and spotlight will search it. This also works surprisingly well for non-OCRd images of text, anything inside a zip file, an email, etc..
- westcort 2y agoUse Recoll on Linux or File Locator Lite on Windows to do RegEx searches. Design the RegEx searches with GPT or llama running locally (or write them yourself).
- SoftTalker 2y agoIf you haven’t given some serious thought to getting rid of most of the documents then consider it. There is very little need to keep most routine documents for more than a few years. If you think you need your electric bill for March 2006 at your fingertips, why?
- datpiff 2y agoI was hoping someone would make this point. A lot of digital archiving is just delaying tossing things - a hard drive is easier to deal with than boxes of paper. The contents can still be useless. When it comes to a search solution - what kind of searches have you done in the past? What kind of problems did you come across? If the answer to either is "none" you are planning on building a useless system.
- temp3000 2y agoYou never know when you will need a 10 year old doc. Audits and disputes for example. In addition I suspect keeping all the docs uses 1% of the spaces of photos people back up anyway. I agree that search is overkill - just drudge manually or use grep when the time comes to dig.
- treetalker 2y agoOn MacOS, use HoudahSpot. It’s awesome. Not AI, but as others have said, you likely want plain text search, not “AI” or a chatbot, for something like this. If you’re having trouble thinking of search terms to plug into HoudahSpot (or grep etc.) then I suppose you could ask a chatbot to assist your brainstorming, and then plug those terms into HoudahSpot/grep/etc.
- pixelmonkey 2y agorga, aka ripgrep-all, is my go-to for this. I suppose grep is a form of AI -- or, at least, an advanced intelligence that's wiser than it looks. ;) https://github.com/phiresky/ripgrep-all https://github.com/phiresky/ripgrep-all
- gyrovagueGeist 2y ago+1 for this. I use rga all the time. it's a "simple" solution but often enough for what I actually needed.
- deleted 2y ago[deleted]
- constantinum 2y agoThe primary challenge is not just about harnessing AI for search; it's about preparing complex documents of various formats, structures, designs, scans, multi-layout tables, and even poorly captured images for LLM consumption. This is a crucial issue. There is a 20 min read on why parsing PDFs is hell: https://unstract.com/blog/pdf-hell-and-practical-rag-applications/ https://unstract.com/blog/pdf-hell-and-practical-rag-applica... To parse PDFs for RAG applications, you'll need tools like LLMwhisperer[1] or unstructured.io[2]. Now back to your problem: This solution might be an overkill for your requirement, but you can try the following: To set things up quickly, try Unstract[3], an open-source document processing tool. You can set this up and bring your own LLM models; it also supports local models. It has a GUI to write prompts to get insights from your documents.[4] [1] https://unstract.com/llmwhisperer/ https://unstract.com/llmwhisperer/ [2] https://unstructured.io/ https://unstructured.io/ [3] https://github.com/Zipstack/unstract https://github.com/Zipstack/unstract [4] https://github.com/Zipstack/unstract/blob/main/docs/assets/prompt_studio.png https://github.com/Zipstack/unstract/blob/main/docs/assets/p...
- fooker 2y agoModern LLMs are good enough at treating pdfs as images and groking the context. Well, Claude and GPT-4 seem to be.
- jszymborski 2y agoApache Tika could help extract the relevant bits of PDFs, couldnt it? https://tika.apache.org/ https://tika.apache.org/
- ilaksh 2y agoIf you want to run locally you can look into this https://github.com/PaddlePaddle/PaddleOCR https://github.com/PaddlePaddle/PaddleOCR https://andrejusb.blogspot.com/2024/03/optimizing-receipt-processing-with.html https://andrejusb.blogspot.com/2024/03/optimizing-receipt-pr... But I suggest that you just skip that and use gpt-4o. They aren't actually going to steal your data. Sort through it to find anything with a credit card number or anything ahead time. Or you could look into InternVL.. Or a combination of PaddleOCR first and then use a strong LLM via API, like gpt-4o or llama3 70b via together.ai If you truly must do it locally, then if you have two 3090s or 4090s it might work out. Otherwise it the LLMs may not be smart enough to give good results. Leaving out the details of your hardware makes it impossible to give good advice about running locally. Other than, it's not really necessary.
- gnicholas 2y ago> But I suggest that you just skip that and use gpt-4o. They aren't actually going to steal your data. Why do you have this confidence? Is it based on reading their TOS, and assuming they'll follow it?
- hulitu 2y ago> Ask HN: I have many PDFs – what is the best local way to leverage AI for search? Adobe Reader can search all PDFs in a directory. They hide this function though.
- kkfx 2y agoHonestly? ocrmypdf + ripgrep-all, recoll (GUI+XLI xapian wrapper) if you prefer an indexed version, for mere full-text search, currently nothing gives better results. The semantic search it's still not there, Paperless-ngx, tagspaces and so on demand way too much time for adding just a single document to be useful at a certain scale. My own personal version is org-mode, I keep all my stuff org-attached, so instead of searching the pdfs I search my notes linking them, a kind of metadata-rich, taggable, quick, full-text search however even if org-ql is there I almost not use it, just org-roam-node-find and counsel-rg on notes. Once done this allow for quick enough manual and variously automated archiving, do it on a large home directory it's a very long and tedious manual work. For me it's worth done since I keep adding documents and using them, but it took more than an year to be "almost done enough" and it's still unfinished after 4 years.
- jesterson 2y agoThe best tool I found for myself for similar goal was Devonthink. Using it for many years since and quite happy with it. There is no AI or any other modern fad, but fulltext search (including OCR for image files inside PDFs) works great
- skapa_flow 2y agoGoogle Drive. It doesn't fullfill the "local" criteria, but it works for us (small engineering firm). We synchronize our local file server with GD nighly and use it only for searching. Google is just good when it comes to search.
- Tylast 2y agoYou can try https://gpt4all.io/index.html https://gpt4all.io/index.html
- epirogov 2y agoCheap but full featured solution for batch AI processing of PDF documents on your local is an Aspose.PDF ChatGPT plugin https://products.aspose.org/pdf/net/chat-gpt/ https://products.aspose.org/pdf/net/chat-gpt/
- pawelduda 2y agoTry https://github.com/phiresky/ripgrep-all https://github.com/phiresky/ripgrep-all before going down the rabbit hole of AI and advanced indexers. Quick to set up and undo if that's not what you want, but I'm pretty sure you'll be surprised how far can this get you
- theolivenbaum 2y agoIf you're looking for something local, we develop an app for macOS and Windows that let's you search and talk to local files and data from cloud apps: https://curiosity.ai https://curiosity.ai For the AI features, you can use OpenAI or local models (the app uses llama.cpp in the background, it ships with llama3 and a few other models, and we're soon going to let you use any .gguf model)
- hm-nah 2y agoYou have the find a good OCR tool that you can run locally on your hardware. RAG depends on your doc processing pipeline. It’s not local, but the Azure Document Intelligence OCR service has a number of prebuilt models. The “prebuilt-read” model is $1.50/1k pages. Once you OCR your docs, you’ll have a JSON of all the text AND you get breakdowns by page/word/paragraph/tables/figures/alllll with bouding-boxes. Forget the Lang/Llama/Chain-theory. You can do it all in vanilla Python.
- jeffreyq 2y agoTangentially related, but you can try https://macro.com/ https://macro.com/ for reading your PDFs.
- Ey7NFZ3P0nzAe 2y agoI am a medical students with thousands and thousands of PDF and was unsatisfied with RAG tools so I made my own. It can consume basically any type of content (pdf, epub, youtube playlist, anki database, mp3, you name it) and does a multi step RAG by first using embedding then filtering using a smaller LLM then answering using by feeding each remaining document to the strong LLM then combine those answers. It supports virtually all LLMs and embeddings, including local LLMs and local embedding It scales surprisingly well and I have tons of improvements to come, when I have some free time or procrastinate. Don't hesitate to ask for features! Here's the link: https://github.com/thiswillbeyourgithub/DocToolsLLM/ https://github.com/thiswillbeyourgithub/DocToolsLLM/
- samspenc 2y agoNvidia's 'Chat with RTX' can do this as well https://www.nvidia.com/en-us/ai-on-rtx/chatrtx/ https://www.nvidia.com/en-us/ai-on-rtx/chatrtx/ You do need a beefy GPU to run the local LLM, but I think it's a similar requirement for running any LLM on your machine.
- Ey7NFZ3P0nzAe 2y agoI am deeply unsatisfied with how most RAG systems handle questions, chunking, embeddings, storage, and even those used for summaries are usually rubbish. That's why I created my own tool. Check it out I updated it a lot! It supports ollama too for private use.
- vrighter 2y agoyou use a tool intended for accurately searihing. Which is not ai based.
- hypefi 2y agocheck out my app "Chofane" this is something that does it, local batch OCR scan for PDFs and PNG files, I am just launching it, you can export results to json and csv, do some text based search on results https://chofane-landing.pages.dev/ https://chofane-landing.pages.dev/
- phodo 2y agoThank you all for the comments. Got a lot of good input and ways to think thru the tried and true tools (enjoying ripgrep-all + fzf) plus the standard ai/rag-style tools. I do think there is room for a bridge or an integrated way to pipe in similarity / embedding into the ripgreps of the world. Maybe something close to fzf’s piping model. Will explore if I have some time.
- Kikobeats 2y agoYou can use Microlink to turn PDF into HTML, and combine it with other service for processing the text data. Here an example turning a arxiv paper into real text: https://api.microlink.io/?data.html.selector=html&embed=html&meta=false&url=https://arxiv.org/pdf/2104.12871 https://api.microlink.io/?data.html.selector=html&embed=html... It looks like PDF, but it you open devtools you can see it's actually a very precise HTML representation.
- ssahoo 2y agoThis could be a humor or real hack. Get a copilot PC with recall enabled and quickly scan through the documents by opening in Adobe acrobat reader. Voillla! You will have an sqlite dB that has your index. Few days later, Adobe could have your data in their llm.