5 ms·
What kind of machine do you have running that? My attempts at local have always resulted in a very hot lap
by ncphillips 1mo ago
What kind of machine do you have running that? My attempts at local have always resulted in a very hot lap
- AbsurdCensor 1mo agoStrix Halo for me. If I am running something on my laptop, it's a much smaller usually around 12b model, but those are a bit less functional. I mean I think there is a a ROG FLow Z that has the Strix Halo setup, but that thing was super expensive.
- nozzlegear 1mo agoI host the models on my Mac Studio, an M1 Ultra with 64gb ram (I bought it when it came out, just happens to be good at LLMs). So when I work on my laptop, I have my oh-my-pi setup configured to use the models on my Mac over my local "bonjour" network or whatever Apple calls it. That way I have a nice cool lap, while using models that my M4 MacBook Air with its 16gb ram couldn't possibly run.
- ncphillips 1mo agoCool yeah. I got a M3 with 32GB ram and it’s a little iffy. I’ve considered getting a MacMini to act as an in-house