alexandercheema

6 ms·

1.
by alexandercheema 6mo ago
lol bro there is already a big OSS project called exo: https://github.com/exo-explore/exo
2.
by alexandercheema 7mo ago
Isn't Claude Code for Infrastructure just...Claude Code?
3.
by alexandercheema 8mo ago
Is there one for Kimi K2.5?
4.
Kimi K2.5 on 2 Mac Studios (<$20K) running at 24 TPS(twitter.com)
6 points by alexandercheema 8mo ago|4 comments
5.
by alexandercheema 11mo ago
Appreciate you checking back so often. We have some exciting plans. Keep checking and it won't be long before something pops up :)
6.
by alexandercheema 11mo ago
Yes, these models are mostly compute-bound so benefit even more from the compute on the DGX Spark.
7.
by alexandercheema 11mo ago
Blog author here. Actually, no. The model can be streamed into the DGX Spark, so we can run prefill of models much larger than 128GB (e.g. DeepSeek R1) on the DGX Spark. This feature is coming to EXO 1.0 which will be open-sourced soonTM.
8.
Clustering Nvidia DGX Spark and M3 Ultra Mac Studio for 4x Faster LLM Inference(twitter.com)
8 points by alexandercheema 11mo ago|1 comments
9.
Clustering Nvidia DGX Spark and M3 Ultra Mac Studio for 4x Faster LLM Inference(blog.exolabs.net)
5 points by alexandercheema 11mo ago|0 comments
10.
by alexandercheema 2y ago
exo maintainer here. tgtweak is correct. This looks like potentially some promising research that I'm looking into reproducing now. We want to lower the barrier to running large models as much as possible so if this works, it would be
11.
Meta AI: "The Future of AI Is Open Source and Decentralized"(twitter.com)
214 points by alexandercheema 2y ago|111 comments
12.
Two MacBooks is All You Need: Running Llama 3 405B using Apple MLX and Exo(twitter.com)
11 points by alexandercheema 2y ago|2 comments
13.
by alexandercheema 2y ago
Not yet, should I make an issue for it?
14.
by alexandercheema 2y ago
This is fixed now, with these commits: - https://github.com/exo-explore/exo/commit/dbbc7be57fb1871d2b... - https://github.com/exo-explore/exo/commit/ce46f000591d8d59c1... Pleas
15.
by alexandercheema 2y ago
Do you mean with Apple Intelligence? You can already query models you host from Apple using exo or even just local on-device inference.
16.
by alexandercheema 2y ago
Try it out - don't trust me! The way this works is that each device holds a partition of the model (for now a continuous set of layers). E.g. let's say you have 3 devices and the model is 32 layers. Device 1 could hold layers 1-10
17.
by alexandercheema 2y ago
Repo maintainer here. It supports any device tinygrad does, which is a lot. We didn’t expect it to blow up so soon - the repo is still experimental. Internally we’ve mostly been testing on MacBooks and Mac Minis, and that’s where dev is hap
18.
by alexandercheema 2y ago
Thank you for the support! I agree on the cost point, and personally I don’t want to live in a world where all AI runs on H100s in a giant datacenter controlled by one company.
19.
by alexandercheema 2y ago
You’re right. The assumption right now is that you’re running on trusted devices on your own local network. I will add a section in the README.
20.
by alexandercheema 2y ago
That’s where we want to get eventually. There’s a lot of work that needs to be done but I’m confident we’ll get there. Give us 3 months and it’ll be as simple as running Dropbox.
21.
by alexandercheema 2y ago
Could you create a GitHub issue? There's a lot of work we'd like to do to improve this.
22.
by alexandercheema 2y ago
Thanks for pointing out that. Fixed https://github.com/exo-explore/exo/blob/main/LICENSE
23.
by alexandercheema 2y ago
Fascinating, will check this out! I wanted to focus on Python first to build this quickly, test out ideas and iterate. This seems like a good option for a switch. Do you know if any of these can run on Apple/Android devices?
24.
by alexandercheema 2y ago
Repo author here. This sounds interesting. Could you elaborate on the benefits of Bumblebee / Axon?
25.
by alexandercheema 2y ago
Issue link: https://github.com/exo-explore/exo/issues/11
26.
by alexandercheema 2y ago
It *should* but I haven't tried it. I will try it. Updated in this issue: We could also try raspberry pi + coral usb tpu ( https://coral.ai/products/ ) - that might be a killer combo for super cheap home ai cluster.
27.
by alexandercheema 2y ago
"Transformers are memory bandwidth bound" - this is the precise reason why this makes sense. If a model doesn't fit into memory on a single device, it needs to be incrementally loaded into memory (offloading), which is bottle
28.
by alexandercheema 2y ago
We didn't expect this to blow up so quickly. A lot of work needs to be done on getting different setups working. I have made an issue here: https://github.com/exo-explore/exo/issues/11
29.
by alexandercheema 2y ago
Yeah, unfortunately the autoregressive nature of these models slows it down significantly with added device<->device latency. However, you can still max out on throughput with pipeline parallelism, where you overlap execution. See: h
30.
by alexandercheema 2y ago
Repo author here. That's correct. The embeddings for Llama-3-8B are around 8KB-10KB. For Llama-3-70B they're around 32KB. These are small enough to send around between devices on a local network. For a SETI@home style network, lat

More ›