5 ms·
I ran the 7B Vicuna (ggml-vic7b-q4_0.bin) on a 2017 MacBook Air (8GB RAM) with llama.cpp. Worked OK for me with the default context size. 2048, like you see in
by mozillas 3y ago
I ran the 7B Vicuna (ggml-vic7b-q4_0.bin) on a 2017 MacBook Air (8GB RAM) with llama.cpp.
Worked OK for me with the default context size. 2048, like you see in most examples was too slow for my taste.
- koheripbal 3y agoGiven the current price (mostly free) off public llms I'm not sure what the use case of running out at home are yet. OpenAIs paid GPT4 has few restrictions and is still cheap. ... Not to mention GPT4 with browsing feature is vastly superior to any home of the models you can run at home.
- 2devnull 3y agoMany would be users can’t send their data data to openAI. Think HIPPA and other laws restricting data sharing. Federation or distribution of the models for local training is the other solution to that problem.
- toxik 3y agoThe point for me personally is the same as why I find it so powerful to self host SMTP, IMAP, HTTP. It’s in my hands, I know where it all begins and ends. I answer to no one. For LLMs this means I am allowed their full potential. I can generate smut, filth, illegal content of any kind for any reason. It’s for me to decide. It’s empowering, it’s the hacker mindset.
- sagarm 3y agoI think it's mostly useful if you want to do your own fine tuning, or the data you are working with can't be sent to a third party for contractual, legal, or paranoid reasons.
- sroussey 3y agoI’m working on an app to index your life, and having it local is a huge plus for the people I have using it.
- theaussiestew 3y agoSounds interesting, got a link?
- sroussey 3y agoNot yet…
- int_19h 3y agoGPT4 API is still not universally available, for starters.