13 ms·
Mesh LLM: distributed AI computing on iroh
- turtleyacht 2mo agoIt sounds like iroh enables distributed compute without having to finangle custom hardware.
- jmercouris 2mo agoI thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.
- i386 2mo agoThat isn’t true. llama RPC is incredibly slow but staged splits in skippy are orders of magnitude faster.
- stymaar 2mo agoThroughput is not a problem as you just share relatively small vectors (a few kilobytes in size), the key issue is network latency.
- jmercouris 2mo agoI didn't mean the throughput of the network, I meant the throughput of the calculations. Yes, the latency is the issue.
- nttylock 2mo ago[flagged]
- darkpicnic 2mo agococompute.ai is already doing this really well.
- SwellJoe 2mo agoIs it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).
- darkpicnic 2mo agoGood point. I know cocompute is working on splitting, but it’s not there yet; I was referring to the round-robin delegation within a trusted pool. Mesh LLM looks great too!
- dnoberon 2mo agoCool, always good to have more in the ecosystem. I love Iroh and hope this continues to succeed.
- SwellJoe 2mo agoI note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
- woadwarrior01 2mo agoPerf should be fairly straightforward to ballpark. You'll need to transfer roughly 2 . hidden_size . num_shards bytes over the network per token during autoregressive decoding. And divide that number by chunk size during prefill.
- i386 2mo agoThis was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN. The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others. I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.
- zdw 2mo agoWhat hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?
- i386 2mo agoThe lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet. We use a customized Q2 quantization that preserves sensitive tensors at Q8. To reduce compute time per layer, we are developing a custom GLM DSA Metal graph. While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2. Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.
- darkpicnic 2mo agoDoes Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?
- tekacs 2mo agoI'm not affiliated, but yes – the main 'point' of iroh is that it's 'dial-a-key', QUIC with encryption based on the keys of the endpoints.
- metadat 2mo agoJust wondering, why do you care about encryption in this context?
- darkpicnic 2mo agoIf payloads to LLMs are being passed around to various nodes, even trusted ones (like friends and family), it gets awkward if you send something very personal. Think sending a medical question to medgemma:27b.
- oezi 2mo agoEven if transport is encrypted, the LLM computations will always be clear text, right?
- embedding-shape 2mo agoIndeed, it's in-transit-encrypted so snoopers won't be able to see it, but it's not E2E encrypted nor in-process encrypted, the one's doing the inference could obviously see the input/output.
- i386 2mo agoI’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
- cromka 2mo agoWould this benefit from integrating with the Colibri project announced here just days ago?
- deleted 2mo ago[deleted]
- iotapi322 2mo agoThis is super impressive, We have a lab with lots of different epycs and different models - to bring them together this way is amazing. Well done!
- i386 2mo agoThank you! AMD is a weak spot in our testing right now. If you’re willing to contribute or let us borrow some compute time, drop in on the Discord.
- Lerc 2mo agoI have never really delved into kv cache implementation, do they run effectively separate caches per layer? If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries. Is anyone doing best-of-n with a n stage pipeline running each query offset by one?
- i386 2mo agoEach stage has its own KV for the layers it hosts. You are on the money there, when one stage is waiting it's free for more parallelism. I am planning on exploiting this for more token verification through ngram spec decoding.
- 2mo ago
- tlongwell-block 2mo ago[dead]
- _superposition_ 2mo agoI just wish I had the hardware to try it out!
- jing09928 2mo ago[dead]
- dwoosley 2mo agoI’ve been curious what a polymorphic botnet that runs one (or multiple) distributed LLMs would be capable of doing. The idea would be to evolve the botnet delivery and payload using the clustered compute of all hosts in the botnet to run LLMs that guides the evolution of various botnet clusters. Bad cluster morphs get caught and cleaned off and bad delivery methods never spread, but the best versions survive to continue to grow. What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.
- nullc 2mo agoDoes this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.
- michaelneale 2mo ago@null not as yet, but did a tiny bit of research before. If i understand it this would help mainly with batching (ie concurrent sessions/users/turns etc) to keep the pipeline more busy (vs a batch of a fixed size going together), which is interesting, but most work so far has been on latency hiding for straight line performance to start. We did have a version of things which used expert islands for MoE parallel alternative (I think that has been mostly scrubbed from the code). It showed early promise by having trunk+hot experts together, but as models got larger, it made both that very large on its own but also didn't seem to work as well (or we just weren't good at grouping experts), the idea being to really side step latency and route sessions to those islands. It felt like as models scaled it didn't stay smart (as diverse experts activated more than I thought they would). Could be our mistake (was an exciting possibility though - if you don't mind accepting some loss). tldr; yes this seems a very nice enhancement for smarter batching/keeping things busy and seems like most larger models we look at are MoE!
- whatjustin 2mo agoThe real test is throughput. I'd like to see tokens/sec at higher concurrency and with uneven hardware.
- Abishek_Muthian 2mo agoI'm more interested in running distributed inference for purpose built small language models than these coding LLMs. Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output. Nicely done OP.
- unrvl22 2mo agoSomething like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.
- Abishek_Muthian 2mo agoTrue, I've been telling that coding LLMs have been detrimental to local AI scene. What happened to those nifty ML projects which can run on raspberry pi zero.
- sig_kill 2mo agoThis is actually how I develop and use the mesh at home. Rather than splitting models, I aggregate disparate compute behind one endpoint, without having separate inference providers on each host and a gateway like LiteLLM
- downrightmike 2mo agodifference between this and Exo?
- sig_kill 2mo agoWrite up here: https://meshllm.cloud/docs/pages/exo-comparison/ https://meshllm.cloud/docs/pages/exo-comparison/
- deleted 2mo ago[deleted]
- MattPerry 2mo agoThe first picture "gpu rig", "laptop", "server", "cloud node, etc made me realize how little compute I have. I don't have a laptop with 24GB VRAM or a workstation with 96GB. I think if I convinced all of my friends to run LLMs on their gaming PCs, I don't I would have the total VRAM in the picture. As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.
- kennywinker 2mo agohttps://public.meshllm.cloud/ https://public.meshllm.cloud/
- whs 2mo agoI've been looking for similar distributed computing style LLM, and I found AI Horde and a few other smaller efforts like one from Aphrodite people and distributed training from Nous Research. AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models. I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.
- SubiculumCode 2mo agoAll these ASICS being designed and specialized for AI but none seem to be being built for consumers. Reason?
- kennywinker 2mo agoI spent a while trying to get mesh-llm running, but none of the installable llama.cpp builds worked with my older gpu. It looks like it should be able to be used to proxy an external llama.cpp service, but I had no luck setting that up either. Seems very cool, but definitely some rough edges.
- i386 2mo agoI’d love a bug report - we can get it working for you!
- bubbi 2mo ago[flagged]
- josefrichter 2mo agoIs there a catch? If not, this would be super useful.
- stymaar 2mo agoThe catch is that the token generation speed is going to be limited by network latency, making it unbearably slow to run over the internet. It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).
- luciana1u 2mo ago[flagged]
- jkwang 2mo ago[dead]
- deleted 2mo ago[deleted]
- vigsterkr 2mo agothe https://query.mt/ https://query.mt/ project has been using iroh based mesh for a while. maybe give it a go, especially if you wanna use your mesh models on your mobile phone as well.
- derdi 2mo agoThey might want to make their docs less confusing. https://query.mt/showcase/gpu-sharing/ https://query.mt/showcase/gpu-sharing/ says "no API key needed", and talks about running llama.cpp on a friend's GPU. No API keys are needed for this. But then the page goes on about who "holds" API keys, and where API keys "stay", where API keys are "locked", etc.
- tangsoupgallery 2mo ago[flagged]
- dana321 2mo agoI knew this was possible, i asked chatgpt about a year ago and it said no the latency would be too big of a problem. I spent the best part of a year learning libp2p and was looking for a project to do with it at the time.
- roger_ 2mo agoDoes this support Qwen 3.6 (e.g. 27B) and the myriad of llama.cpp options (batch sizes, quantization, etc.)? I'd love to see some performance data.
- sig_kill 2mo agoYes! I’ve worked on the settings interface between our runtime and llamacpp, these are documented and available via our config.toml file
- hydrogenbon007 2mo ago[dead]
- Onavo 2mo agoIs this truly more secure though? The host can still see your data.
- maccam912 2mo agoI have a macbook pro, figured I'd see how easy it was to contribute some vram... And I can't overstate how easy it was. The swarm page thing had a little "join" button and said to run "mesh-llm --auto". And I did. And it worked first try. That is such an uncommon experience I had to report back. It handled picking a model to serve, downloading it from peers, and to test it I chatted with the model I was hosting, I could see the GPU doing work, etc. It might be more of an endorsement for iroh than mesh-llm, although I'm sure getting it to all work seamlessly took work on both sides. But to whoever spent the time and energy trying to make it seamless, consider the effort recognized!
- jjheuaao 2mo ago[flagged]
- postpress 2mo agoThis got me thinking about experiments with models talking to each other over WebRTC: https://xt-ml.github.io/shadow-claw/ https://xt-ml.github.io/shadow-claw/ Its sort of a "P2P mesh" :-) Watch four instances of the harness running together and collaborating on checking the weather: https://www.youtube.com/watch?v=h1les1A3gcg https://www.youtube.com/watch?v=h1les1A3gcg
- lenerdenator 2mo agoLet's say I have a bunch of computers (Raspberry Pi 3+, MBP from 2017, Lenovo T420) sitting around that are on my local network. All are from the last 15 years and my main driver is a M2 MBP with 32 gb of RAM. Would this let me utilize that spare hardware to do anything meaningful with a self-hosted coding-focused LLM at a somewhat-decent token generating speed, or is this still a pipe dream?
- s4saif 2mo agoSo, how is this different then exo ? it does the same job as well
- i386 2mo agoexo is Mac only.
- throw1234567891 2mo agono it isn't
- brcmthrowaway 2mo agoHow does this differ to darkbloom?
- enbarca 2mo agoI'm looking at adopting a mesh network architecture for self-sovereign inference across longitudinal health records.
- guillaume_code 2mo ago[dead]