7 ms·
If I understand correctly what you are looking for, Ollama might be a solution (https://ollama.com/ https://ollama.com/)?. I have no affiliation, but I lazily u
by Kydlaw 2y ago
If I understand correctly what you are looking for, Ollama might be a solution (https://ollama.com/ https://ollama.com/)?. I have no affiliation, but I lazily use this solution when I want to run a quick model locally.
- TechDebtDevin 2y agoBetter yet install Open Web GUI and ollama at the same time via docker. Most people will want a familiar GUI rather than the terminal. https://github.com/open-webui/open-webui https://github.com/open-webui/open-webui This will install ollama and open web GUI: For GPU support run: docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama Use for CPU only support: docker run -d -p 3000:8080 -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama
- Der_Einzige 2y agoWhy do people recommend this instead of the much better oobabooga text-gen-webui? https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui It's like you hate settings, features, and access to many backends!
- TechDebtDevin 2y agoTo each their own, how are you using these extra features? I personally am not looking to spend a bunch on API credits and don't have the hardware to run models larger than 7-8b parameters. I use local llms almost exclusively for formatting notes and as a reading assistant/summarizer and therefor don't need these features.