8 ms·

PrivateGPT

520 points by antouank 3y ago

rolisz 3y ago
For some reason, downloading the model they suggest keeps failing. I tried to download it in Firefox and Edge. I'm using Windows, if that matters. Anyone else seeing similar issues?
bohlenlabs 3y ago
So many good links here, thanks to the OP for sharing, and to all commenters as well!
Wronnay 3y ago
Wow. I keep a personal Wiki, Journal and use plain text accounting... This project could help me create a personal AI which answers any questions to my life, finances or knowledge...
- moffkalast 3y ago
  Well maybe it works on Obsidian vaults for note taking heh, but with llama models' 2k input token range it'd get a tenth of the way before starting to drop context. Likely useless without something like an 100k model.
  - rolisz 3y ago
    Well you wouldn't input the whole Vault to the model, you would use embeddings to find the content that is most relevant to the question being asked.
    - moffkalast 3y ago
      Is that actually a thing yet? Proper vector DB integration? I sure would like to see some demos of that, as it's been hyped up a lot but I haven't really seen anyone deploy anything proper with it yet.
      - rolisz 3y ago
        Even PrivateGPT does that, using Chroma as vector DB
lysp 3y ago
Quick how-to/demo: https://www.youtube.com/watch?v=A3F5riM5BNE https://www.youtube.com/watch?v=A3F5riM5BNE Also has a suggestion of a few alternative models to use.
seydor 3y ago
does this only work with llamaCPP ? I.e. can't use GPU models with this?
- joebiden2 3y ago
  llama supports GPU by now: https://old.reddit.com/r/LocalLLaMA/comments/13gok03/llamacpp_now_officially_supports_gpu_acceleration/ https://old.reddit.com/r/LocalLLaMA/comments/13gok03/llamacp...
  - seydor 3y ago
    How do i enable GPU in privateGPT w llamaCpp ? It turns my CPU to a vacuum cleaner
divan 3y ago
This will still hallucinate, right? Projects like this for using with your documents datasets are invaluable, but everything I've tried so far is hallucinating, so not practical. What's the state of the art of the LLM without hallucination at the moment?
- XCSme 3y ago
  How do you define hallucination?
  - divan 3y ago
    factually incorrect / nonsensical output
    - TeMPOraL 3y ago
      It will still talk like a human blurting out their train of thought out loud, yes.
    - XCSme 3y ago
      I assume this is only possible if the training data contains only a "right answer". If the training data contains two contradicting answers A and B, then, from the AIs perspective, there is no correct answer. I assume that for questions like "What year was Bill Gates born in?", it should never return a wrong answer, if the answer was in the training data. If it was not, it should respond that it doesn't know.
- anu7df 3y ago
  Not exactly sure if this would qualify as an LLM in the GPT4 sense. But for no hallucination this seems good: https://www.thirdai.com/pocketllm/ https://www.thirdai.com/pocketllm/ Full disclosure. I know the founder, but not really associated with the company in any way.
- Art9681 3y ago
  Like many others, I’m also building my own platform to accomplish this. What I’ve learned is the document preparation is key in getting the LLM to answer correctly. The text splitting portion is a crucial step here. Picking the correct splitter and parameters for your use case is important. At first I was getting incorrect or made up answers. Setting up a proper prompt template and text splitting parameters fixed the issue for the most part and now I have 99% success. Also, the local model used makes a big difference. Right now wizard-mega and manticore are the best ones to use. I run the 16b ggml versions in an M2 Pro and it takes about 30 seconds to “warm up” and produce some quality responses.
amelius 3y ago
With so many LLM options out there, how do we keep track of which ones are good?
- superbiome 3y ago
  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... is one place.
amelius 3y ago
How many tokens/second on an average machine?
davidy123 3y ago
Granted I'm not coming from the python world, but I have tried many of these projects, and very few of them install out of the box. They usually end with some incompatibility, and files scattered all over the place, leading to future nightmares. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. sentry-sdk 1.22.2 requires urllib3<2.0.0, but you have urllib3 2.0.2 which is incompatible Just for fun, here's the result of python -m pip install -r ./requirements.txt for tortoise-tts; …many many lines raise ValueError("%r is not a directory" % (package_path,)) ValueError: 'build/py3k/scipy' is not a directory Converting to Python3 via 2to3... … /tmp/pip-install-hkb_4lh7/scipy_088b20410aca4f0cbcddeac86ac7b7b1/build/py3k/scipy/signal/fir_filter_design.py [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed I'm not asking for support, just saying if people really want to make something 'easy' they'd use docker. I gather there are better python package managers, but I gather that's a bit of a mess too. Someone is thinking "this is part of learning the language," but I think it's just bad design.
- oefrha 3y ago
  You don’t need Docker, you just need a virtual env for each random thing you try instead of making them all conflict with each other. Maybe some day pip will add a switch to automatically create one, but until then, python3 -m venv venv . venv/bin/activate before you try something random. Also, `python` is usually Python 2.7. If it is, I advise removing it from your system unless you have a strong reason to keep it.
  - davidy123 3y ago
    Sorry, but (base) vid@kk:~/D/ai/tortoise-tts$ python3 -m venv venv (base) vid@kk:~/D/ai/tortoise-tts$ . venv/bin/activate (venv) (base) vid@kk:~/D/ai/tortoise-tts$ python -m pip install -r ./requirements.txt Collecting tqdm Using cached tqdm-4.65.0-py3-none-any.whl (77 kB) Collecting rotary_embedding_torch Using cached rotary_embedding_torch-0.2.3-py3-none-any.whl (4.5 kB) … × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [8 lines of output] Traceback (most recent call last): File "<string>", line 2, in <module> File "<pip-setuptools-caller>", line 34, in <module> File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 196, in <module> setup_package() File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 147, in setup_package from numpy.distutils.core import setup ModuleNotFoundError: No module named 'numpy' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. [notice] A new release of pip available: 22.3.1 -> 23.1.2 [notice] To update, run: pip install --upgrade pip (venv) (base) vid@kk:~/D/ai/tortoise-tts$ I'm sure you could eventually help get this working, which is kind of you, but the point is the "supposed tos" don't work either. It needs to be comprehensively fixed if python really wants to be approachable. Maybe it doesn't. It's also just not a good citizen when it comes to heterogeneous apps on the same system. This isn't the first time venv didn't work for me, then there's anaconda, miniconda, and a bunch of other things that add env and directories. I don't really know what any of them do, and -I don't want to- I'm not an expert on every app on my system, but I can use nearly all of them without pain. (remember this is about ease of use) Oh yeah, and python 2 vs python 3. <rolls eyes> It's very much the "works for me" experience from the old days. There's no good learning from it, except dependencies suck and python systems aren't good at them. I think when releasing anything that includes dependencies that span the operating system, it's just good engineering to use a container approach. Otherwise you're just causing a lot of discomfort in the world for no good reason. It's funny because chatgpt would give me an answer to this in a few moments, but I'm locked out for a week because it can't figure out where I am. Now I'm spending my Sunday morning setting up a dockerfile for tortoise-tts. At least I will learn something reusable from that. I guess I will create a PR for it, though it seems the author isn't tending the repo anymore.
j_shi 3y ago
Self-hosted + self-trained LLMs are probably the future for enterprise. While consumers are happy to get their data mined to avoid paying, businesses are the opposite: willing to pay a lot to avoid feeding data to MSFT/GOOG/META. They may give assurances on data protection (even here GitHub copilot TOS has sketchy language around saving down derived data), but can’t get around fundamental problem that their products need user interactions to work well. So it seems with BigTechLLM there’s inherent tension between product competitiveness and data privacy, which makes them incompatible with enterprise. Biz ideas along these lines: - Help enterprises set up, train, maintain own customized LLMs - Security, compliance, monitoring tools - Help AI startups get compliant with enterprise security - Fine tuning service
- cddotdotslash 3y ago
  I suspect the major cloud providers will also each offer their own “enterprise friendly” LLM services (Azure already offers a version of OpenAI’s API). If they have the right data guarantees, that’ll probably be sufficient for companies that are already using their IaaS offerings.
  - quaintdev 3y ago
    Enterprises should work on an open source LLM and run it on their own. This also helps people like you and me to run LLM at home. It has worked before like in case of Linux and can work again.
    - cubefox 3y ago
      Powerful LLMs are so large that they can only be trained by the major AI companies. Even LLaMA 65B (where the open release was less than intended) can't compete with GPT-3.5, let alone GPT-4. And the price for the most powerful models will only increase now, as we have effectively an arms race between OpenAI/Microsoft and Google. Few, if anyone, will be able to keep up. Linux is different. It doesn't require huge investments in server farms.
      - 3y ago
superbiome 3y ago
Does something like this exist for local code repos? (Excuse my ignorance since the space is moving faster than light.)
- arbol 3y ago
  Seems to me that this could be used for exactly that. Just fork the repo and change the filetypes and loaders for your code source files.
- arbol 3y ago
  https://python.langchain.com/en/latest/modules/indexes/document_loaders.html https://python.langchain.com/en/latest/modules/indexes/docum...
monkeydust 3y ago
Got this working locally - badly needs GPU support (have a 3090 so come on!) there is some workaround but expect it will come pretty soon. This video was a useful walkthough esp on using different model and upping the CPU threads. https://www.youtube.com/watch?v=A3F5riM5BNE https://www.youtube.com/watch?v=A3F5riM5BNE
Havoc 3y ago
Would this work better with something like llama or a instruction following model like alpaca?
debbiedowner 3y ago
This is a shortcut/workaround to transforming the private docs to a prompt:answer dataset and fine tuning right? What would be the difference in user experience or information retrieval performance between the two? My impression is it saves work on the dataset transformation and compute for fine tuning, so it must be less performant. Is there a reason to prefer the strategy here other than ease of setup?
hodanli 3y ago
These are the similar projects I've come across: - [GitHub - e-johnstonn/BriefGPT: Locally hosted tool that connects documents to LLMs for summarization and querying, with a simple GUI.](https://github.com/e-johnstonn/BriefGPT https://github.com/e-johnstonn/BriefGPT) - [GitHub - go-skynet/LocalAI: Self-hosted, community-driven, local OpenAI-compatible API. Drop-in replacement for OpenAI running LLMs on consumer-grade hardware. No GPU required. LocalAI is a RESTful API to run ggml compatible models: llama.cpp, alpaca.cpp, gpt4all.cpp, rwkv.cpp, whisper.cpp, vicuna, koala, gpt4all-j, cerebras and many others!](https://github.com/go-skynet/LocalAI https://github.com/go-skynet/LocalAI) - [GitHub - paulpierre/RasaGPT: RasaGPT is the first headless LLM chatbot platform built on top of Rasa and Langchain. Built w/ Rasa, FastAPI, Langchain, LlamaIndex, SQLModel, pgvector, ngrok, telegram](https://github.com/paulpierre/RasaGPT https://github.com/paulpierre/RasaGPT) - [GitHub - imartinez/privateGPT: Interact privately with your documents using the power of GPT, 100% privately, no data leaks](https://github.com/imartinez/privateGPT https://github.com/imartinez/privateGPT) - [GitHub - reworkd/AgentGPT: Assemble, configure, and deploy autonomous AI Agents in your browser.](https://github.com/reworkd/AgentGPT https://github.com/reworkd/AgentGPT) - [GitHub - deepset-ai/haystack: Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-4, ChatGPT and alike). Haystack offers production-ready tools to quickly build complex question answering, semantic search, text generation applications, and more.](https://github.com/deepset-ai/haystack https://github.com/deepset-ai/haystack) - [PocketLLM « ThirdAi](https://www.thirdai.com/pocketllm/ https://www.thirdai.com/pocketllm/) - [GitHub - imClumsyPanda/langchain-ChatGLM: langchain-ChatGLM, local knowledge based ChatGLM with langchain ｜基于本地知识库的 ChatGLM 问答](https://github.com/imClumsyPanda/langchain-ChatGLM https://github.com/imClumsyPanda/langchain-ChatGLM)
zestyping 3y ago
Would someone do me the kindness of explaining (a little more) how this works? It looks like you can ask a question and the model will use its combined knowledge of all your documents to figure out the answer. It looks like it isn't fine-tuned or trained on all the documents, is that right? How is each document turned into an embedding, and then how does the model figure out which documents to consult to answer the question?
simonw 3y ago
I'm always interested in seeing the prompt that drives these kinds of tools. In this case it appears to be using RetrievalQA from LangChain, which I think is this prompt here: https://github.com/hwchase17/langchain/blob/v0.0.176/langchain/chains/retrieval_qa/prompt.py https://github.com/hwchase17/langchain/blob/v0.0.176/langcha... Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. {context} Question: {question} Helpful Answer:
- seydor 3y ago
  The problem is, when does it know that it does not know.
- zora_goron 3y ago
  On a related note, in case it's of interest to anyone else -- I pulled out all the default prompts from LangChain and put them up here: https://github.com/samrawal/langchain-prompts/blob/main/README.md https://github.com/samrawal/langchain-prompts/blob/main/READ...
- jstarfish 3y ago
  Do such fail-early conditions save processing time?
  - mabbo 3y ago
    If you mean the "If you don't know" part, oh no, they have a much bigger problem they're solving. The LLM will absolutely lie if it doesn't know and you haven't made it perfectly clear that you'd rather it did not do that. LLMs seem to be trying to give answers that make you happy. A good lie will make you happy. Unless it understands that you will not be happy with a lie. Is this anthropomorphizing? Yep. But that's the best way I've found to reason about them.
    - TeMPOraL 3y ago
      > Is this anthropomorphizing? Yep. But that's the best way I've found to reason about them. My take? It's like a high-schooler being asked a question by the teacher and having to answer on the spot. If they studied the material well, they'll give a good and correct answer. If they (like me, more often than I'd care to admit) only half-listened to the lectures and maaaaybe skimmed some cliff's notes before class, they will give an answer too - one strung together out of few remembered (or misremembered) facts, an overall feel for the problem space (e.g. writing style, historical period, how people behave), with lots and lots of interpolation in between. Delivered confidently, it has more chance of avoiding a bad mark (or even scoring a good one) than flat-out saying, "I don't know". Add to that some usual mistakes out of carelessness and... whatever it is that makes you forget a minus sign and realize it half a page of equations later - and you get GPT-4. It's giving answers like a person who just blurts out whatever thoughts pop into their head, without making a conscious attempt at shaping or interrogating them.
kordlessagain 3y ago
Working on something similar that uses keyterm extraction for traversal of topics and fragments, without using Langchain. It's not designed to be private, however: https://github.com/FeatureBaseDB/DocGPT/tree/main https://github.com/FeatureBaseDB/DocGPT/tree/main
behnamoh 3y ago
When you split a document into chunks, doesn't some crucial information get cut in half? In that case, you'd probably lose that information in the context if that information was immediately followed by an irrelevant information that reduces the cosine similarity. Is there a "smarter" way to feed documents as context to LLMs?
- haolez 3y ago
  Don't know if there is a smarter way, but these libraries usually offer an overlap parameter that allows you to repeat the last N characters of a chunk in the first N of the next chunk.
aldarisbm 3y ago
One quick plug I want to have the memory part of langchain down, vector store + local database + client to chat with an LLM (gpt4all model can be swapped with OpenAI api just switching the base URL) https://github.com/aldarisbm/memory https://github.com/aldarisbm/memory It's still got ways to go, if someone wants to help let me know :)
- santiagobasulto 3y ago
  Sorry for my ignorance. But memory refers to the process of using embeddings for QA right? The process roughly is: Ingestion: - Process embeddings for your documents (from text to array of numbers) - Store your documents in a Vector DB Query time: - Process embeddings for the query - Find documents similar to the query using distance from other docs in the Vector db - Construct prompt with format: """ Answer question using this context: {DOCUMENTS RETRIEVED} Question: {question} Answer: """ Is that correct? Now, my question is, can the models be swapped easily? Or that requires a complete recalculation of the embedding (and new ingestion)?
  - bigfudge 3y ago
    The embeddings can be based on a different model to the one you pass them as context to. So you could upgrade the summmariser model without upgrading the embeddings.
    - santiagobasulto 3y ago
      But you'd need to keep both models in parallel, right? Using M1 to keep computing embeddings and using M2 for completions.
skykooler 3y ago
"System requirements" section should really mention what amount of RAM or VRAM is needed for inference.
- awestroke 3y ago
  That depends on which model you use it with. It's "bring your own model"
  - rain1 3y ago
    so list a few known to work models and their requirements
    - seydor 3y ago
      https://old.reddit.com/r/LocalLLaMA/wiki/models https://old.reddit.com/r/LocalLLaMA/wiki/models
deleted 3y ago
[deleted]
thefourthchime 3y ago
I tried this on my M2 Macbook with 16gb of RAM but got: "ggml_new_tensor_impl: not enough space in the context's memory pool (needed 18296202768, available 18217606000)"
- soferio 3y ago
  Anyone got it working on an M1 with 8gb?
  - kseudo 3y ago
    I got it working on an M1 with 16gb. Quite slow but it trains and returns responses.
    - brianjking 3y ago
      What did you do to make it work? I'm getting an illegal hardware instruction when I try to run python privateGPT.py Using embedded DuckDB with persistence: data will be stored in: db [1] 8281 illegal hardware instruction python privateGPT.py
LaurenceRitchie 3y ago
[dead]
- weikju 3y ago
  Is it private if it's using ChatGPT?
  - unstatusthequo 3y ago
    They will say using API means you data isn’t used for training. True if you believe OpenAI t&c’s. But that’s different than being able to say you didn’t send data to any third party.
    - weikju 3y ago
      Indeed, it's still a far cry from being private if the data is leaving my device for any reason.
jaimehrubiks 3y ago
If you select a gpt4all model like GPT-J can this be used commercially or is there other dependency that limits the license?
yosito 3y ago
> Put any and all your files into the source_documents directory Why? Why can't I define any directory (my existing Obsidian vault, for example) as the source directory?
- carlio 3y ago
  You can by setting an environment variable - https://github.com/imartinez/privateGPT/blob/main/ingest.py#L35 https://github.com/imartinez/privateGPT/blob/main/ingest.py#...
keeptrying 3y ago
This is the future.
daitangio 3y ago
Hi, very interesting... what are the memory/disk requirements to run it? 16GB of RAM would be enough? I suggest to add these requirements to the README
- moffkalast 3y ago
  Well I'm not sure which models specifically work, but it runs on llama.cpp, which would mean lama derivative ones. Here's a little table for quantized CPU (GGML) versions and the RAM they require as a general rule of thumb: > Name Quant method Bits Size RAM required Use case WizardLM-7B.GGML.q4_0.bin q4_0 4bit 4.2GB 6GB 4bit. WizardLM-7B.GGML.q4_1.bin q4_0 4bit 4.63GB 6GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. WizardLM-7B.GGML.q5_0.bin q5_0 5bit 4.63GB 7GB 5-bit. Higher accuracy, higher resource usage and slower inference. WizardLM-7B.GGML.q5_1.bin q5_1 5bit 5.0GB 7GB 5-bit. Even higher accuracy, and higher resource usage and slower inference. WizardLM-7B.GGML.q8_0.bin q8_0 8bit 8GB 10GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. > Name Quant method Bits Size RAM required Use case wizard-vicuna-13B.ggmlv3.q4_0.bin q4_0 4bit 8.14GB 10.5GB 4-bit. wizard-vicuna-13B.ggmlv3.q4_1.bin q4_1 4bit 8.95GB 11.0GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. wizard-vicuna-13B.ggmlv3.q5_0.bin q5_0 5bit 8.95GB 11.0GB 5-bit. Higher accuracy, higher resource usage and slower inference. wizard-vicuna-13B.ggmlv3.q5_1.bin q5_1 5bit 9.76GB 12.25GB 5-bit. Even higher accuracy, and higher resource usage and slower inference. wizard-vicuna-13B.ggmlv3.q8_0.bin q5_1 5bit 16GB 18GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. > Name Quant method Bits Size RAM required Use case VicUnlocked-30B-LoRA.ggmlv3.q4_0.bin q4_0 4bit 20.3GB 23GB 4-bit. VicUnlocked-30B-LoRA.ggmlv3.q4_1.bin q4_1 5bit 24.4GB 27GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. VicUnlocked-30B-LoRA.ggmlv3.q5_0.bin q5_0 5bit 22.4GB 25GB 5-bit. Higher accuracy, higher resource usage and slower inference. VicUnlocked-30B-LoRA.ggmlv3.q5_1.bin q5_1 5bit 24.4GB 27GB 5-bit. Even higher accuracy, and higher resource usage and slower inference. VicUnlocked-30B-LoRA.ggmlv3.q8_0.bin q8_0 8bit 36.6GB 39GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. Copied of some of The-Bloke's model descriptions on huggingface. With 16G you can run practically all 7B and 13B versions. With shared GPU+CPU inference, one can also offload some layers onto a GPU (not sure if that makes the initial RAM requirement smaller), but you do need CUDA of course.
sinandrei91 3y ago
Is there a benchmark for retrieval from multiple ft documents? I tried the LangchainQA with Pinecone and wasn't impressed with the search result when using it on my Zotero library.
ChocoluvH 3y ago
Always wondering pros/cons of Chroma and Qdrant. Can someone tell me?
- kacperlukawski 3y ago
  Chroma doesn't seem to be a real DB, it's rather a wrapper around tools like hnswlib, DuckDB or Clickhouse. Qdrant is way more mature - it has its own HNSW implementation with some tweaks to incorporate filtering directly during the vector search phase, supports horizontal and vertical scaling, as well as provides its own managed cloud offering. In general, Qdrant is a real DB, not a library and that's a huge difference.
  - ChocoluvH 3y ago
    What does Chroma lack? Their APIs seem pretty much the same to me.
    - jeadie 3y ago
      I've tried both Chroma and Qdrant. I don't think Chroma lacks that much. Definitely newer, but is also a great product. I think cloud support coming Q3 2023
udev4096 3y ago
I posted it 9 days ago and somehow this one gets the attention. The same freaking post. Unbelievable https://news.ycombinator.com/item?id=35914810 https://news.ycombinator.com/item?id=35914810
- dutchbrit 3y ago
  Day and time when you post something matters a lot unfortunately.
  - udev4096 3y ago
    And now I am getting downvoted for it. Perfect
    - andruby 3y ago
      possibly because the tone of your post and that it doesn't actually add to the conversation. weekday and time of day have an impact. Thousands of entries are posted each day (see https://news.ycombinator.com/newest https://news.ycombinator.com/newest) most never get a comment or upvote.
      - udev4096 3y ago
        People get very easily offended these days. What's wrong with that tone? I was just simply stating a fact
        Mystery-Machine 3y ago
        You are right. People get easily offended. But your tone does seem a bit _freaking_ upset. Over what? The fact that someone posted the same link like you did and got more clicks from random users browsing the internet? Is that what upset you? What do you get if you got billions of clicks and upvotes? They say that people who need external validation don't have their own values and need to seek approval from others...

520 points by antouank 3y ago

For some reason, downloading the model they suggest keeps failing. I tried to download it in Firefox and Edge. I'm using Windows, if that matters. Anyone else seeing similar issues?

bohlenlabs 3y ago

So many good links here, thanks to the OP for sharing, and to all commenters as well!

Wronnay 3y ago

Wow. I keep a personal Wiki, Journal and use plain text accounting... This project could help me create a personal AI which answers any questions to my life, finances or knowledge...

moffkalast 3y ago
Well maybe it works on Obsidian vaults for note taking heh, but with llama models' 2k input token range it'd get a tenth of the way before starting to drop context. Likely useless without something like an 100k model.
- rolisz 3y ago
  Well you wouldn't input the whole Vault to the model, you would use embeddings to find the content that is most relevant to the question being asked.
  - moffkalast 3y ago
    Is that actually a thing yet? Proper vector DB integration? I sure would like to see some demos of that, as it's been hyped up a lot but I haven't really seen anyone deploy anything proper with it yet.
    - rolisz 3y ago
      Even PrivateGPT does that, using Chroma as vector DB

lysp 3y ago

Quick how-to/demo: https://www.youtube.com/watch?v=A3F5riM5BNE https://www.youtube.com/watch?v=A3F5riM5BNE Also has a suggestion of a few alternative models to use.

seydor 3y ago

does this only work with llamaCPP ? I.e. can't use GPU models with this?

joebiden2 3y ago
llama supports GPU by now: https://old.reddit.com/r/LocalLLaMA/comments/13gok03/llamacpp_now_officially_supports_gpu_acceleration/ https://old.reddit.com/r/LocalLLaMA/comments/13gok03/llamacp...
- seydor 3y ago
  How do i enable GPU in privateGPT w llamaCpp ? It turns my CPU to a vacuum cleaner

divan 3y ago

This will still hallucinate, right? Projects like this for using with your documents datasets are invaluable, but everything I've tried so far is hallucinating, so not practical. What's the state of the art of the LLM without hallucination at the moment?

XCSme 3y ago
How do you define hallucination?
- divan 3y ago
  factually incorrect / nonsensical output
  - TeMPOraL 3y ago
    It will still talk like a human blurting out their train of thought out loud, yes.
  - XCSme 3y ago
    I assume this is only possible if the training data contains only a "right answer". If the training data contains two contradicting answers A and B, then, from the AIs perspective, there is no correct answer. I assume that for questions like "What year was Bill Gates born in?", it should never return a wrong answer, if the answer was in the training data. If it was not, it should respond that it doesn't know.
anu7df 3y ago
Not exactly sure if this would qualify as an LLM in the GPT4 sense. But for no hallucination this seems good: https://www.thirdai.com/pocketllm/ https://www.thirdai.com/pocketllm/ Full disclosure. I know the founder, but not really associated with the company in any way.
Art9681 3y ago
Like many others, I’m also building my own platform to accomplish this. What I’ve learned is the document preparation is key in getting the LLM to answer correctly. The text splitting portion is a crucial step here. Picking the correct splitter and parameters for your use case is important. At first I was getting incorrect or made up answers. Setting up a proper prompt template and text splitting parameters fixed the issue for the most part and now I have 99% success. Also, the local model used makes a big difference. Right now wizard-mega and manticore are the best ones to use. I run the 16b ggml versions in an M2 Pro and it takes about 30 seconds to “warm up” and produce some quality responses.

amelius 3y ago

With so many LLM options out there, how do we keep track of which ones are good?

superbiome 3y ago
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... is one place.

amelius 3y ago

How many tokens/second on an average machine?

davidy123 3y ago

Granted I'm not coming from the python world, but I have tried many of these projects, and very few of them install out of the box. They usually end with some incompatibility, and files scattered all over the place, leading to future nightmares. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. sentry-sdk 1.22.2 requires urllib3<2.0.0, but you have urllib3 2.0.2 which is incompatible Just for fun, here's the result of python -m pip install -r ./requirements.txt for tortoise-tts; …many many lines raise ValueError("%r is not a directory" % (package_path,)) ValueError: 'build/py3k/scipy' is not a directory Converting to Python3 via 2to3... … /tmp/pip-install-hkb_4lh7/scipy_088b20410aca4f0cbcddeac86ac7b7b1/build/py3k/scipy/signal/fir_filter_design.py [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed I'm not asking for support, just saying if people really want to make something 'easy' they'd use docker. I gather there are better python package managers, but I gather that's a bit of a mess too. Someone is thinking "this is part of learning the language," but I think it's just bad design.

oefrha 3y ago
You don’t need Docker, you just need a virtual env for each random thing you try instead of making them all conflict with each other. Maybe some day pip will add a switch to automatically create one, but until then, python3 -m venv venv . venv/bin/activate before you try something random. Also, `python` is usually Python 2.7. If it is, I advise removing it from your system unless you have a strong reason to keep it.
- davidy123 3y ago
  Sorry, but (base) vid@kk:~/D/ai/tortoise-tts$ python3 -m venv venv (base) vid@kk:~/D/ai/tortoise-tts$ . venv/bin/activate (venv) (base) vid@kk:~/D/ai/tortoise-tts$ python -m pip install -r ./requirements.txt Collecting tqdm Using cached tqdm-4.65.0-py3-none-any.whl (77 kB) Collecting rotary_embedding_torch Using cached rotary_embedding_torch-0.2.3-py3-none-any.whl (4.5 kB) … × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [8 lines of output] Traceback (most recent call last): File "<string>", line 2, in <module> File "<pip-setuptools-caller>", line 34, in <module> File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 196, in <module> setup_package() File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 147, in setup_package from numpy.distutils.core import setup ModuleNotFoundError: No module named 'numpy' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. [notice] A new release of pip available: 22.3.1 -> 23.1.2 [notice] To update, run: pip install --upgrade pip (venv) (base) vid@kk:~/D/ai/tortoise-tts$ I'm sure you could eventually help get this working, which is kind of you, but the point is the "supposed tos" don't work either. It needs to be comprehensively fixed if python really wants to be approachable. Maybe it doesn't. It's also just not a good citizen when it comes to heterogeneous apps on the same system. This isn't the first time venv didn't work for me, then there's anaconda, miniconda, and a bunch of other things that add env and directories. I don't really know what any of them do, and -I don't want to- I'm not an expert on every app on my system, but I can use nearly all of them without pain. (remember this is about ease of use) Oh yeah, and python 2 vs python 3. <rolls eyes> It's very much the "works for me" experience from the old days. There's no good learning from it, except dependencies suck and python systems aren't good at them. I think when releasing anything that includes dependencies that span the operating system, it's just good engineering to use a container approach. Otherwise you're just causing a lot of discomfort in the world for no good reason. It's funny because chatgpt would give me an answer to this in a few moments, but I'm locked out for a week because it can't figure out where I am. Now I'm spending my Sunday morning setting up a dockerfile for tortoise-tts. At least I will learn something reusable from that. I guess I will create a PR for it, though it seems the author isn't tending the repo anymore.

j_shi 3y ago

Self-hosted + self-trained LLMs are probably the future for enterprise. While consumers are happy to get their data mined to avoid paying, businesses are the opposite: willing to pay a lot to avoid feeding data to MSFT/GOOG/META. They may give assurances on data protection (even here GitHub copilot TOS has sketchy language around saving down derived data), but can’t get around fundamental problem that their products need user interactions to work well. So it seems with BigTechLLM there’s inherent tension between product competitiveness and data privacy, which makes them incompatible with enterprise. Biz ideas along these lines: - Help enterprises set up, train, maintain own customized LLMs - Security, compliance, monitoring tools - Help AI startups get compliant with enterprise security - Fine tuning service

cddotdotslash 3y ago
I suspect the major cloud providers will also each offer their own “enterprise friendly” LLM services (Azure already offers a version of OpenAI’s API). If they have the right data guarantees, that’ll probably be sufficient for companies that are already using their IaaS offerings.
- quaintdev 3y ago
  Enterprises should work on an open source LLM and run it on their own. This also helps people like you and me to run LLM at home. It has worked before like in case of Linux and can work again.
  - cubefox 3y ago
    Powerful LLMs are so large that they can only be trained by the major AI companies. Even LLaMA 65B (where the open release was less than intended) can't compete with GPT-3.5, let alone GPT-4. And the price for the most powerful models will only increase now, as we have effectively an arms race between OpenAI/Microsoft and Google. Few, if anyone, will be able to keep up. Linux is different. It doesn't require huge investments in server farms.
    - 3y ago

superbiome 3y ago

Does something like this exist for local code repos? (Excuse my ignorance since the space is moving faster than light.)

arbol 3y ago
Seems to me that this could be used for exactly that. Just fork the repo and change the filetypes and loaders for your code source files.
arbol 3y ago
https://python.langchain.com/en/latest/modules/indexes/document_loaders.html https://python.langchain.com/en/latest/modules/indexes/docum...

monkeydust 3y ago

Got this working locally - badly needs GPU support (have a 3090 so come on!) there is some workaround but expect it will come pretty soon. This video was a useful walkthough esp on using different model and upping the CPU threads. https://www.youtube.com/watch?v=A3F5riM5BNE https://www.youtube.com/watch?v=A3F5riM5BNE

Havoc 3y ago

Would this work better with something like llama or a instruction following model like alpaca?

debbiedowner 3y ago

This is a shortcut/workaround to transforming the private docs to a prompt:answer dataset and fine tuning right? What would be the difference in user experience or information retrieval performance between the two? My impression is it saves work on the dataset transformation and compute for fine tuning, so it must be less performant. Is there a reason to prefer the strategy here other than ease of setup?

hodanli 3y ago

These are the similar projects I've come across: - [GitHub - e-johnstonn/BriefGPT: Locally hosted tool that connects documents to LLMs for summarization and querying, with a simple GUI.](https://github.com/e-johnstonn/BriefGPT https://github.com/e-johnstonn/BriefGPT) - [GitHub - go-skynet/LocalAI: Self-hosted, community-driven, local OpenAI-compatible API. Drop-in replacement for OpenAI running LLMs on consumer-grade hardware. No GPU required. LocalAI is a RESTful API to run ggml compatible models: llama.cpp, alpaca.cpp, gpt4all.cpp, rwkv.cpp, whisper.cpp, vicuna, koala, gpt4all-j, cerebras and many others!](https://github.com/go-skynet/LocalAI https://github.com/go-skynet/LocalAI) - [GitHub - paulpierre/RasaGPT: RasaGPT is the first headless LLM chatbot platform built on top of Rasa and Langchain. Built w/ Rasa, FastAPI, Langchain, LlamaIndex, SQLModel, pgvector, ngrok, telegram](https://github.com/paulpierre/RasaGPT https://github.com/paulpierre/RasaGPT) - [GitHub - imartinez/privateGPT: Interact privately with your documents using the power of GPT, 100% privately, no data leaks](https://github.com/imartinez/privateGPT https://github.com/imartinez/privateGPT) - [GitHub - reworkd/AgentGPT: Assemble, configure, and deploy autonomous AI Agents in your browser.](https://github.com/reworkd/AgentGPT https://github.com/reworkd/AgentGPT) - [GitHub - deepset-ai/haystack: Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-4, ChatGPT and alike). Haystack offers production-ready tools to quickly build complex question answering, semantic search, text generation applications, and more.](https://github.com/deepset-ai/haystack https://github.com/deepset-ai/haystack) - [PocketLLM « ThirdAi](https://www.thirdai.com/pocketllm/ https://www.thirdai.com/pocketllm/) - [GitHub - imClumsyPanda/langchain-ChatGLM: langchain-ChatGLM, local knowledge based ChatGLM with langchain ｜基于本地知识库的 ChatGLM 问答](https://github.com/imClumsyPanda/langchain-ChatGLM https://github.com/imClumsyPanda/langchain-ChatGLM)

zestyping 3y ago

Would someone do me the kindness of explaining (a little more) how this works? It looks like you can ask a question and the model will use its combined knowledge of all your documents to figure out the answer. It looks like it isn't fine-tuned or trained on all the documents, is that right? How is each document turned into an embedding, and then how does the model figure out which documents to consult to answer the question?

simonw 3y ago

I'm always interested in seeing the prompt that drives these kinds of tools. In this case it appears to be using RetrievalQA from LangChain, which I think is this prompt here: https://github.com/hwchase17/langchain/blob/v0.0.176/langchain/chains/retrieval_qa/prompt.py https://github.com/hwchase17/langchain/blob/v0.0.176/langcha... Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. {context} Question: {question} Helpful Answer:

seydor 3y ago
The problem is, when does it know that it does not know.
zora_goron 3y ago
On a related note, in case it's of interest to anyone else -- I pulled out all the default prompts from LangChain and put them up here: https://github.com/samrawal/langchain-prompts/blob/main/README.md https://github.com/samrawal/langchain-prompts/blob/main/READ...
jstarfish 3y ago
Do such fail-early conditions save processing time?
- mabbo 3y ago
  If you mean the "If you don't know" part, oh no, they have a much bigger problem they're solving. The LLM will absolutely lie if it doesn't know and you haven't made it perfectly clear that you'd rather it did not do that. LLMs seem to be trying to give answers that make you happy. A good lie will make you happy. Unless it understands that you will not be happy with a lie. Is this anthropomorphizing? Yep. But that's the best way I've found to reason about them.
  - TeMPOraL 3y ago
    > Is this anthropomorphizing? Yep. But that's the best way I've found to reason about them. My take? It's like a high-schooler being asked a question by the teacher and having to answer on the spot. If they studied the material well, they'll give a good and correct answer. If they (like me, more often than I'd care to admit) only half-listened to the lectures and maaaaybe skimmed some cliff's notes before class, they will give an answer too - one strung together out of few remembered (or misremembered) facts, an overall feel for the problem space (e.g. writing style, historical period, how people behave), with lots and lots of interpolation in between. Delivered confidently, it has more chance of avoiding a bad mark (or even scoring a good one) than flat-out saying, "I don't know". Add to that some usual mistakes out of carelessness and... whatever it is that makes you forget a minus sign and realize it half a page of equations later - and you get GPT-4. It's giving answers like a person who just blurts out whatever thoughts pop into their head, without making a conscious attempt at shaping or interrogating them.

kordlessagain 3y ago

Working on something similar that uses keyterm extraction for traversal of topics and fragments, without using Langchain. It's not designed to be private, however: https://github.com/FeatureBaseDB/DocGPT/tree/main https://github.com/FeatureBaseDB/DocGPT/tree/main

behnamoh 3y ago

When you split a document into chunks, doesn't some crucial information get cut in half? In that case, you'd probably lose that information in the context if that information was immediately followed by an irrelevant information that reduces the cosine similarity. Is there a "smarter" way to feed documents as context to LLMs?

haolez 3y ago
Don't know if there is a smarter way, but these libraries usually offer an overlap parameter that allows you to repeat the last N characters of a chunk in the first N of the next chunk.

aldarisbm 3y ago

One quick plug I want to have the memory part of langchain down, vector store + local database + client to chat with an LLM (gpt4all model can be swapped with OpenAI api just switching the base URL) https://github.com/aldarisbm/memory https://github.com/aldarisbm/memory It's still got ways to go, if someone wants to help let me know :)

santiagobasulto 3y ago
Sorry for my ignorance. But memory refers to the process of using embeddings for QA right? The process roughly is: Ingestion: - Process embeddings for your documents (from text to array of numbers) - Store your documents in a Vector DB Query time: - Process embeddings for the query - Find documents similar to the query using distance from other docs in the Vector db - Construct prompt with format: """ Answer question using this context: {DOCUMENTS RETRIEVED} Question: {question} Answer: """ Is that correct? Now, my question is, can the models be swapped easily? Or that requires a complete recalculation of the embedding (and new ingestion)?
- bigfudge 3y ago
  The embeddings can be based on a different model to the one you pass them as context to. So you could upgrade the summmariser model without upgrading the embeddings.
  - santiagobasulto 3y ago
    But you'd need to keep both models in parallel, right? Using M1 to keep computing embeddings and using M2 for completions.

skykooler 3y ago

"System requirements" section should really mention what amount of RAM or VRAM is needed for inference.

awestroke 3y ago
That depends on which model you use it with. It's "bring your own model"
- rain1 3y ago
  so list a few known to work models and their requirements
  - seydor 3y ago
    https://old.reddit.com/r/LocalLLaMA/wiki/models https://old.reddit.com/r/LocalLLaMA/wiki/models

deleted 3y ago

[deleted]

thefourthchime 3y ago

I tried this on my M2 Macbook with 16gb of RAM but got: "ggml_new_tensor_impl: not enough space in the context's memory pool (needed 18296202768, available 18217606000)"

soferio 3y ago
Anyone got it working on an M1 with 8gb?
- kseudo 3y ago
  I got it working on an M1 with 16gb. Quite slow but it trains and returns responses.
  - brianjking 3y ago
    What did you do to make it work? I'm getting an illegal hardware instruction when I try to run python privateGPT.py Using embedded DuckDB with persistence: data will be stored in: db [1] 8281 illegal hardware instruction python privateGPT.py

LaurenceRitchie 3y ago

[dead]

weikju 3y ago
Is it private if it's using ChatGPT?
- unstatusthequo 3y ago
  They will say using API means you data isn’t used for training. True if you believe OpenAI t&c’s. But that’s different than being able to say you didn’t send data to any third party.
  - weikju 3y ago
    Indeed, it's still a far cry from being private if the data is leaving my device for any reason.

jaimehrubiks 3y ago

If you select a gpt4all model like GPT-J can this be used commercially or is there other dependency that limits the license?

yosito 3y ago

> Put any and all your files into the source_documents directory Why? Why can't I define any directory (my existing Obsidian vault, for example) as the source directory?

carlio 3y ago
You can by setting an environment variable - https://github.com/imartinez/privateGPT/blob/main/ingest.py#L35 https://github.com/imartinez/privateGPT/blob/main/ingest.py#...

keeptrying 3y ago

This is the future.

daitangio 3y ago

Hi, very interesting... what are the memory/disk requirements to run it? 16GB of RAM would be enough? I suggest to add these requirements to the README

moffkalast 3y ago
Well I'm not sure which models specifically work, but it runs on llama.cpp, which would mean lama derivative ones. Here's a little table for quantized CPU (GGML) versions and the RAM they require as a general rule of thumb: > Name Quant method Bits Size RAM required Use case WizardLM-7B.GGML.q4_0.bin q4_0 4bit 4.2GB 6GB 4bit. WizardLM-7B.GGML.q4_1.bin q4_0 4bit 4.63GB 6GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. WizardLM-7B.GGML.q5_0.bin q5_0 5bit 4.63GB 7GB 5-bit. Higher accuracy, higher resource usage and slower inference. WizardLM-7B.GGML.q5_1.bin q5_1 5bit 5.0GB 7GB 5-bit. Even higher accuracy, and higher resource usage and slower inference. WizardLM-7B.GGML.q8_0.bin q8_0 8bit 8GB 10GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. > Name Quant method Bits Size RAM required Use case wizard-vicuna-13B.ggmlv3.q4_0.bin q4_0 4bit 8.14GB 10.5GB 4-bit. wizard-vicuna-13B.ggmlv3.q4_1.bin q4_1 4bit 8.95GB 11.0GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. wizard-vicuna-13B.ggmlv3.q5_0.bin q5_0 5bit 8.95GB 11.0GB 5-bit. Higher accuracy, higher resource usage and slower inference. wizard-vicuna-13B.ggmlv3.q5_1.bin q5_1 5bit 9.76GB 12.25GB 5-bit. Even higher accuracy, and higher resource usage and slower inference. wizard-vicuna-13B.ggmlv3.q8_0.bin q5_1 5bit 16GB 18GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. > Name Quant method Bits Size RAM required Use case VicUnlocked-30B-LoRA.ggmlv3.q4_0.bin q4_0 4bit 20.3GB 23GB 4-bit. VicUnlocked-30B-LoRA.ggmlv3.q4_1.bin q4_1 5bit 24.4GB 27GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. VicUnlocked-30B-LoRA.ggmlv3.q5_0.bin q5_0 5bit 22.4GB 25GB 5-bit. Higher accuracy, higher resource usage and slower inference. VicUnlocked-30B-LoRA.ggmlv3.q5_1.bin q5_1 5bit 24.4GB 27GB 5-bit. Even higher accuracy, and higher resource usage and slower inference. VicUnlocked-30B-LoRA.ggmlv3.q8_0.bin q8_0 8bit 36.6GB 39GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. Copied of some of The-Bloke's model descriptions on huggingface. With 16G you can run practically all 7B and 13B versions. With shared GPU+CPU inference, one can also offload some layers onto a GPU (not sure if that makes the initial RAM requirement smaller), but you do need CUDA of course.

sinandrei91 3y ago

Is there a benchmark for retrieval from multiple ft documents? I tried the LangchainQA with Pinecone and wasn't impressed with the search result when using it on my Zotero library.

ChocoluvH 3y ago

Always wondering pros/cons of Chroma and Qdrant. Can someone tell me?

kacperlukawski 3y ago
Chroma doesn't seem to be a real DB, it's rather a wrapper around tools like hnswlib, DuckDB or Clickhouse. Qdrant is way more mature - it has its own HNSW implementation with some tweaks to incorporate filtering directly during the vector search phase, supports horizontal and vertical scaling, as well as provides its own managed cloud offering. In general, Qdrant is a real DB, not a library and that's a huge difference.
- ChocoluvH 3y ago
  What does Chroma lack? Their APIs seem pretty much the same to me.
  - jeadie 3y ago
    I've tried both Chroma and Qdrant. I don't think Chroma lacks that much. Definitely newer, but is also a great product. I think cloud support coming Q3 2023

udev4096 3y ago

I posted it 9 days ago and somehow this one gets the attention. The same freaking post. Unbelievable https://news.ycombinator.com/item?id=35914810 https://news.ycombinator.com/item?id=35914810

dutchbrit 3y ago
Day and time when you post something matters a lot unfortunately.
- udev4096 3y ago
  And now I am getting downvoted for it. Perfect
  - andruby 3y ago
    possibly because the tone of your post and that it doesn't actually add to the conversation. weekday and time of day have an impact. Thousands of entries are posted each day (see https://news.ycombinator.com/newest https://news.ycombinator.com/newest) most never get a comment or upvote.
    - udev4096 3y ago
      People get very easily offended these days. What's wrong with that tone? I was just simply stating a fact
      - Mystery-Machine 3y ago
        You are right. People get easily offended. But your tone does seem a bit _freaking_ upset. Over what? The fact that someone posted the same link like you did and got more clicks from random users browsing the internet? Is that what upset you? What do you get if you got billions of clicks and upvotes? They say that people who need external validation don't have their own values and need to seek approval from others...