Argonautlabs

6 ms·

1.
Show HN: GLM-5.3 744B at 4 tok/s on a MacBook Pro, experts streamed from 4 SSDs(github.com)
2 points by Argonautlabs 2h ago|0 comments
2.
Deep Seek v4.1 M5 Max at 17 tokens/s(github.com)
13 points by Argonautlabs 1d ago|2 comments
3.
DeepSeek v4.1 Flash (518GB, 4-bit) on a 128GB MacBook: 2.7x prefill, 17 tok/s(github.com)
1 points by Argonautlabs 2d ago|1 comments
4.
by Argonautlabs 8d ago
Often Deep Seek V4 flash or Qwen should be enough. I wanted to see whether Kimi runs at all on one machine with the full record published, and for long multi-table finance reasoning I wanted the strongest model I could keep on the machine.
5.
by Argonautlabs 9d ago
OWC Express 1M2 (Thunderbolt 5, single M.2 NVMe) — three of them, two on the Mac's own ports and one behind an OWC Thunderbolt 5 hub since the machine has three ports. Each enclosure tops out at about 7.1 GB/s on whole-file reads
6.
by Argonautlabs 9d ago
In principle yes, and the upstream engine already has a CUDA path with expert streaming and residency (that's theirs, not ours — we only measured on this Mac). Two things carry over: the experts are read from disk per token either way,
7.
by Argonautlabs 9d ago
Just internal 2Tb Macbook M5 Max drive it came in at roughly half the four-drive speed (0.535 vs 1.038 tok/s at 128 tokens), since one fast drive still has to serve all 16 reads per layer while four drives split the load https:/&
8.
by Argonautlabs 9d ago
Memory, not the model. The KV cache on this engine grows about 2.8 MiB per token of context, and the machine's 128 GB is already holding the 50.7 GiB resident trunk plus reserves the speculative-verify path needs (we found the hard way
9.
by Argonautlabs 9d ago
Thank you. Put a five-line TL;DR at the top of the README — what it is, the number, the honest limit, the two findings, credits — with the detail below for anyone who wants it.
10.
by Argonautlabs 9d ago
Fair, and we didn't measure it. Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand. The current configuration admit
11.
by Argonautlabs 9d ago
That's the one workload this setup is worst at today, unfortunately: output tokens are cheap at 1/s but input isn't — a 512-token prompt takes ~6 minutes before the first token, because prefill currently reads each layer&#x27
12.
by Argonautlabs 9d ago
Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s is ~2.5 ms, which is what we measure at queue depth 1 on the SN8100s. W
13.
by Argonautlabs 9d ago
Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum. Going fro
14.
by Argonautlabs 9d ago
SSDs are connected via Thunderbolt 5 enclosures. I have one Gen4 and three Gen5 ssds inside enclosures. You can see specs here https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
15.
by Argonautlabs 9d ago
Thank you! Here is the short version: Kimi K3, 2.78T parameters, ~1.45 TB of MXFP4 experts streamed from four SSDs on an M5 Max / 128 GB. 1.00 tok/s steady over 512 tokens, 1.13 over 128, ~6.3 min to first token on a 512-token
16.
by Argonautlabs 9d ago
It actully does the job. Example: every morning it takes 30-40 minutes to generate reports automatically and these reports are being sent as a pdf to read to Telegram.
17.
by Argonautlabs 9d ago
Not useful for chat, agreed — and I wouldn't pretend otherwise. It's useful for the other kind of work: scheduled, unattended jobs where nobody is waiting on the cursor. My use is day/week/month end review — go through t
18.
by Argonautlabs 9d ago
Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NO
19.
Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs(github.com)
277 points by Argonautlabs 9d ago|155 comments