anemll

5 ms·

1.
by anemll 4d ago
Not all systems affected M1 and M5MAX are OK https://x.com/anemll/status/2098454204478366132?s=20
2.
by anemll 6mo ago
Check it out, you might be able to speed it up using this https://github.com/Anemll/anemll-flash-mlx https://x.com/anemll/status/2038684375425200360
3.
by anemll 6mo ago
17B includes 10 expert plus one shared. So actual size of the expert is much smaller
4.
by anemll 6mo ago
Check my repo, I had added some support for GUFF/untloth, Q3,Q5/Q8 https://github.com/Anemll/flash-moe/blob/iOS-App/docs/gguf-h...
5.
by anemll 6mo ago
Thanks for posting this, that's how I first found out about Dan's experiment! SSD speed doubled in the M5P/M generation, that makes it usable! I think one paper under the radar is "KV Prediction for Improved Time to Firs
6.
by anemll 6mo ago
SSD streaming to compute units is new. M4 max can do 15 t/s with its 15GB/s drives
7.
by anemll 6mo ago
Yes, SSD speed is critical though. The repo has macOS builds for CLI and Desktop. It's early stages though. M4 Max gets 10-15 TPS on 400B depending on quantization. Compute is an issue too; a lot of code is PoC level.
8.
by anemll 6mo ago
multiple NAND, and apple already used it in Mac Studio. Plus better cooling
9.
by anemll 6mo ago
both, tbh
10.
by anemll 6mo ago
Probably 2x speed for Mac Studio this year if they do double NAND ( or quad?)
11.
iPhone 17 Pro Demonstrated Running a 400B LLM(twitter.com)
713 points by anemll 6mo ago|326 comments
12.
by anemll 9mo ago
Tensor Parallel test with RDMA last week https://x.com/anemll/status/1996349871260107102 Note fast sync workaround
13.
RDMA over Thunderbolt 5 on Apple Silicon – 14µs latency(twitter.com)
6 points by anemll 10mo ago|1 comments
14.
by anemll 10mo ago
In macOS 26.2 (Tahoe) beta, Apple introduced a low-latency Thunderbolt 5 RDMA driver, enabling up to 80 Gb/s bidirectional bandwidth for Mac clustering—ideal for distributed ML on Apple Silicon. It's optimized for low latency, del
15.
by anemll 1y ago
It’s also supported in Apple Neural Engine https://github.com/Anemll/Anemll
16.
Anemll adds Qwen3 support for Apple neural engine(twitter.com)
4 points by anemll 1y ago|0 comments
17.
by anemll 1y ago
We can ran 2000 or 4000 context with ANE
18.
by anemll 1y ago
M4 max should work at 120GB for ANE and 500+ for GPU. So GPU will be 3-4 times faster for anything over 1-3B. ANE is likely as fast for prefill due to higher FLOPs
19.
by anemll 1y ago
Right.I was thinking about it, you still need batch refill, however, Apple Core ML tools were failing for attention activations quantization. Long context, pre-fill is still compute bound.
20.
by anemll 1y ago
Yes for GPU, however ANE only supports FP16 plus integers. M4/A17 added accelerated int8 that is twice faster than FP16
21.
by anemll 1y ago
Memory bandwidth is the main bottleneck. It got better with M3/M4. ANE is really fast in FLOPS but low in memory bandwidth.
22.
by anemll 1y ago
What hardware are you on? Most models are memory bandwidth limited. ANE was limited to 64GB/s prior to M3 Max or M4 pro. If you are on M1, GPU will be significantly faster for 3-8B models due to memory bandwidth rather then ANE capabil