JohnTheNerd

15 ms·

1.
by JohnTheNerd 6mo ago
or if you have an Apple Silicon Mac, this: https://github.com/TaterTotterson/microWakeWord-Trainer-Appl... I used it personally, did a lot of research (including asking questions to the creator of microWakeWord), and
2.
by JohnTheNerd 2y ago
that's a great idea! I've been looking into that (I'm merely logging all prompts in a JSON file for now, so that I can analyze them later). skipping the LLM would be tough because there are so many devices in my house, not to
3.
Making my local LLM voice assistant faster and more scalable with RAG(johnthenerd.com)
122 points by JohnTheNerd 2y ago|16 comments
4.
by JohnTheNerd 2y ago
If you are scared of messing with electricity like I am, instead of using power monitoring, another viable but less reliable option is to use vibration sensors. I picked up a simple Zigbee vibration sensor for less than 20$, taped it on top
5.
by JohnTheNerd 3y ago
I think you have a valid point, but the risk of this feels exaggerated. I already had a few entities I didn't really need it using (not for security reasons, but to shorten the system prompt). I simply excluded them within the Jinja te
6.
by JohnTheNerd 3y ago
"successful" is a very optimistic way of looking at it. it has several downsides but largely works for my needs: - read access is mostly available for sensors, even if access wasn't granted. - some integrations (especially cu
7.
by JohnTheNerd 3y ago
I recommend opening the original link if possible, because the archive link is missing the demo video and a few important updates to the jinja templates!
8.
by JohnTheNerd 3y ago
that is correct! the less I rely on external companies and/or servers, the happier I am with my setup. I actually greatly simplified my infrastructure in the blog... there's a LOT going on behind those network switches. it took qu
9.
by JohnTheNerd 3y ago
I do it, but I'm completely insane: - I actually stay on top of all patches, including HomeAssistant itself - I run it behind a WAF and IPS. lots of VLANs around. even if you breach a service, you'll probably trip something up in
10.
by JohnTheNerd 3y ago
it is almost entirely the LLM. I can see this in action by typing a response on my computer instead of using my phone/watch, which bypasses Whisper and Piper entirely. your approach would work, but I really like the creativity of havin
11.
by JohnTheNerd 3y ago
it actually works really well when I use it, but is slow because of the 4060Ti's (~8 seconds) and there is slight overfitting to the examples provided. none of it seemed to affect the actions taken, just the commentary. I don't ha
12.
by JohnTheNerd 3y ago
alternatively, could we not simply split by common characters such as newlines and periods, to split it within sentences? it would be fragile with special handling required for numbers with decimal points and probably various other edge cas
13.
by JohnTheNerd 3y ago
thank you for building an amazing product! I suspect cloning OpenAI's API is done for compatibility reasons. most AI-based software already support the GPT-4 API, and OpenAI's official client allows you to override the base URL ve
14.
by JohnTheNerd 3y ago
power consumption. I am running multiple GPUs somewhere residential. the 4060Ti only draws 180W at max load (which it almost never reaches). 3090 is about double for 1.5x the VRAM, and it's notorious for briefly consuming much more tha
15.
by JohnTheNerd 3y ago
that is true, but consider two things: - motherboards and CPUs have a limited number of PCIe lanes available. I went with a second-hand Threadripper 2920x to be able to have 4 GPU's in the future. since you can only fit so many GPUs, y
16.
by JohnTheNerd 3y ago
I never tried it, but I think it would go very poorly without a wake word of sorts. HomeAssistant seems to natively support wake words, but I haven't looked into it yet. I simply use my smartwatch (Wear OS supports replacing Google Ass
17.
by JohnTheNerd 3y ago
yes, they are the 16GB models. beware that the memory bus limits you quite a bit. however, buying brand new, they are the best VRAM per dollar in the NVIDIA world as far as I could see. I use 4-bit GPTQ quants. I use tensor parallelism (vL
18.
by JohnTheNerd 3y ago
that is correct! I would much rather run everything in-house, where I know the quality won't be degraded over time (see the Google Assistant announcement from yesterday) and I am in full control of my data. using a cloud service is muc
19.
by JohnTheNerd 3y ago
that is correct, however I am already using all of my VRAM. it would mean I have to degrade my model quality. I instead decided that I would rather have one solid model, and have all my use cases tied to that one model. using RAM instead
20.
by JohnTheNerd 3y ago
I would strongly advise using a GPU for inference. the reason behind this is not mere tokens-per-second performance, but that there is a dramatic difference in how long you have to wait before seeing the first token output . this scales ve
21.
Building a fully local LLM voice assistant to control my smart home(johnthenerd.com)
699 points by JohnTheNerd 3y ago|186 comments