4 ms·

Ask HN: What is your local LLM setup?

10 points by anditherobot 2y ago

talldayo 2y ago
RTX 3070
lysace 2y ago
Is anyone doing a local Copilot? What's your setup? Is it competitive with Github Copilot? I just that realized my 32 GB Mac M2 Max Studio is pretty good at running relatively large models using Ollama. And there's the Continue.dev VS Code plugin that can use it, but I feel that the suggested defaults aren't very optimal for this config.
- kingkongjaffa 2y ago
  You can connect a local ollama instance into the zed editor to chat with your open files and get inline prompting.
ActorNightly 2y ago
At work, we have access to AWS bedrock, so we use that. At home, I did the math, and its cheaper for me to buy credits for openai and use gpt4 than investing in graphics cards.I use maybe 5 dollars a month max
p1esk 2y ago
8xA6000
sandwichsphinx 2y ago
For local large language models, my current setup is Ollama running on my M1 Mac Mini with 8GB of RAM, using whatever SOTA 8B model comes out. I used to have a more powerful workstation I built in 2016 with three GTX 1070s, but the capacitors were falling off, and I could not justify replacing it when Claude and ChatGPT subscriptions are more than enough for me. I plan on building a new dedicated workstation as soon as the first-mover disadvantage comes down. Today's hardware is still too early and too expensive to warrant any significant personal investment, in my opinion.
roosgit 2y ago
I have a separate PC that I access through SSH. I recently bought a GPU for it, before that I was running it on CPU alone. - B550MH motherboard - Ryzen 3 4100 CPU - 32GB (2x16) RAM cranked up to 3200MHz (prompt generation in memory bound) - 256GB M.2 NVMe (helps with loading models faster) - Nvidia 3060 12GB Software-wise, I use llamafile because on the CPU it's faster by 10-20% for prompt processing than llama.cpp. Performance "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf": CPU-only: 23.47 t/s (processing), 8.73 t/s (generation) GPU: 941.5 t/s (processing), 29.4 t/s (generation)

10 points by anditherobot 2y ago

RTX 3070

Is anyone doing a local Copilot? What's your setup? Is it competitive with Github Copilot? I just that realized my 32 GB Mac M2 Max Studio is pretty good at running relatively large models using Ollama. And there's the Continue.dev VS Code plugin that can use it, but I feel that the suggested defaults aren't very optimal for this config.

kingkongjaffa 2y ago
You can connect a local ollama instance into the zed editor to chat with your open files and get inline prompting.

ActorNightly 2y ago

At work, we have access to AWS bedrock, so we use that. At home, I did the math, and its cheaper for me to buy credits for openai and use gpt4 than investing in graphics cards.I use maybe 5 dollars a month max

p1esk 2y ago

8xA6000

sandwichsphinx 2y ago

For local large language models, my current setup is Ollama running on my M1 Mac Mini with 8GB of RAM, using whatever SOTA 8B model comes out. I used to have a more powerful workstation I built in 2016 with three GTX 1070s, but the capacitors were falling off, and I could not justify replacing it when Claude and ChatGPT subscriptions are more than enough for me. I plan on building a new dedicated workstation as soon as the first-mover disadvantage comes down. Today's hardware is still too early and too expensive to warrant any significant personal investment, in my opinion.

roosgit 2y ago

I have a separate PC that I access through SSH. I recently bought a GPU for it, before that I was running it on CPU alone. - B550MH motherboard - Ryzen 3 4100 CPU - 32GB (2x16) RAM cranked up to 3200MHz (prompt generation in memory bound) - 256GB M.2 NVMe (helps with loading models faster) - Nvidia 3060 12GB Software-wise, I use llamafile because on the CPU it's faster by 10-20% for prompt processing than llama.cpp. Performance "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf": CPU-only: 23.47 t/s (processing), 8.73 t/s (generation) GPU: 941.5 t/s (processing), 29.4 t/s (generation)