5 ms·
If you want a massive MacBook anyway then it's great. They are decent for local LLMs, awesome for local image models and it's a MacBook so AppleCare+ has your b
by jpgvm 3mo ago
If you want a massive MacBook anyway then it's great. They are decent for local LLMs, awesome for local image models and it's a MacBook so AppleCare+ has your back. IMO it's a no brainer if you wanted a MacBook anyway but it's a poor choice if your reason to buy it is to run LLMs.
- mzubairtahir 3mo agoare you saying because of speed or it just cant run them?
- zepearl 3mo agoI agree. To run an acceptable model (e.g. Qwen/Qwen3.6-27B or google/gemma-4-31B) with a good quantization (minimum Q5) with a good context size (min 64k) you could buy 2 or even 3 GTX 5060 16GiB VRAM for ~550$ each. Fyi the much faster MoE models were useless for my usecases - e.g not able to correctly identify me/I/you, endless thinking loops, etc. I'm currently running those models using an RTX 5070 12GiB + RTX 5060 16GiB + RTX 3060 12GiB with a 96k context size with MTP/speculative decoding and I'm quite happy (the 5070 is about 4x faster than the 3060, the 5060 is inbetween them so about 2x faster than a 3060).
- nubg 3mo agohow many tokens per second do you get?
- cybertim 3mo agoI bought two RTX3080s with 20GB during my holiday in china (set me back 700euros) I'm getting 800-1000 input tps and 60-100tps output with Qwen 3.6 27b Q8 (MTP, P2P, 200k context) this feels like opus4.5 level while coding (pi harness). Also easy to just host your own openai compatible api from home this way and still use your MacBook as dev station.
- usagisushi 3mo agoNot the OP, but their setup must be faster than my 4060 16GB + 3060 12GB setup. Here are my numbers (typical values, N=1): Model pp (t/s) tg (t/s) Qwen 3.6 27B 900 29 Qwen 3.6 35B-A3B 2100 85 Gemma 4 31B 750 28 Gemma 4 26B-A4B 2500 90 - All models: UD-Q4 w/ MTP. Context size: ~100k (MoE) / ~70k (Dense). - Layer splitting used. Tensor splitting is ~1.2x faster in TG, but power spikes from 150W to 380W.
- zepearl 3mo agoIn my case it somehow depends a lot on the task being performed... . E.g. when doing text transcription/OCR from images (Qwen 3.6 27B Q4_K_M by Bartowski) with a context size of ~50k I get a pp of ~460 tokens per second and a generation ranging from 35 to 45 tokens per second (using "--spec-type draft-mtp --spec-draft-n-max 2" currently with llama.cpp b6548). On the other hand when handling code (Qwen 3.6 27B Q5_K_M by Bartowski) with a context size of 128k I get a pp ranging between 500 to 1500 tokens per second and a generation between 25 and 40 tokens per second (using in this case as well "--spec-type draft-mtp --spec-draft-n-max 2" currently with llama.cpp b6548). Anyway in theory with "--split-mode layer" I think that it's anyway the slowest card that drives the overall performance (I do see in "nvtop" that usually the 5070 is ~25% active, the 5060 ~50% and the 3060 ~75%).
- eklavya 3mo agoHow are you running these together, splitting the model somehow or did you mean different models on any one card at a time?
- zepearl 3mo agoI always run just one model at once, I switch between them depending on what I do (e.g. Qwen3.6 27B Q5_K_M by Bartowski when programming with "OpenHands" and when doing OCR text transcription and transformation, Gemma4 31B-it Q5_K_M by Bartowski when chatting in Open WebUI doing general tasks).