8 ms·
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
by i386 2mo ago
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
- cromka 2mo agoWould this benefit from integrating with the Colibri project announced here just days ago?
- deleted 2mo ago[deleted]
- iotapi322 2mo agoThis is super impressive, We have a lab with lots of different epycs and different models - to bring them together this way is amazing. Well done!
- i386 2mo agoThank you! AMD is a weak spot in our testing right now. If you’re willing to contribute or let us borrow some compute time, drop in on the Discord.
- Lerc 2mo agoI have never really delved into kv cache implementation, do they run effectively separate caches per layer? If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries. Is anyone doing best-of-n with a n stage pipeline running each query offset by one?
- i386 2mo agoEach stage has its own KV for the layers it hosts. You are on the money there, when one stage is waiting it's free for more parallelism. I am planning on exploiting this for more token verification through ngram spec decoding.
- DerivativeBS 2mo agoCurious about: does it have fault tolerance if one of the machines goes down mid-inference? Can it dynamically reroute, or does it just retry?
- i386 2mo agoIt can dynamically route. If a machine drops out of split, the topology is recalculated and the request is automatically retried.
- Creamsicle47 2mo agoHey, this is a super cool project. It's great to see a lot of the IPFS stuff resurfacing again. A few questions: 1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed. 2.) Any safeguards against malicious actors poisoning model activations?
- i386 2mo agoTo be honest, both are very tough problems we don't have a good answer for yet. If that is something that concerns you, look into building a private mesh with trusted peers.
- Creamsicle47 2mo agoThanks for answering, that makes sense. Also - your setup seems like it could greatly benefit from speculative decoding. Have you guys given any thought to how that might work in this system? P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.
- zmmmmm 2mo agoThe obvious burning question is how performance looks over different network conditions on some standard models. Have you done much benchmarking? Is it mainly latency affected or is overall throughput less than the capacity of the GPUs due to being distributed?
- sig_kill 2mo agoYeah, this is one area we’re struggling with due to the sheer volume of variations and conditions, but I’ve been thinking of collecting some real-time statistics around latency, prefill/decode, and model distribution… that way we can update some kind of live + aggregated performance numbers for interested parties.
- whs 2mo agoI wonder how security is done in this engine, since it's accepting input from anyone. llama.cpp's RPC layer seems to says that you shouldn't run it in public (I assume because it is lower level and may result in RCE on your GPU)
- maxgashkov 2mo agoWhat is the incentive for me to join the public mesh? Do you have any fairness guarantees, e.g. if I contribute 1/8th of the VRAM required to run a particular model, do I get at least 1/16th of the inference share, or anything similar to this?
- sig_kill 2mo agoGreat question! We’ve had a lot of discussions about the direction we want to take this, and how to best generate some kind of incentive / fairness reward. And we’ve found ourselves hesitating on a direction because, at least for now, the primary use case in a useable env is private hosts you own in a mesh; which makes any sort of reward/ incentive structure somewhat unnecessary (for this setup). When the public mesh becomes large enough and we get around to extending the existing “mesh governance” features is likely when we’ll add something like you’ve described. For now, the public mesh is totally open with no restrictions or limitations.
- keynha 2mo ago[flagged]
- stymaar 2mo agoIs it a fully custom inference engine or are you reusing parts of an existing stack? (llama.CPP, vLLM, etc.)
- i386 2mo agoOur skippy library is a patch queue on top of llama that allows us to access internal information, such as activations, and filter tensors on model load.
- potluri 2mo agoHow does this differ from exo?
- sig_kill 2mo agoThere’s a breakdown on our docs page you can check out! https://meshllm.cloud/docs/pages/exo-comparison/ https://meshllm.cloud/docs/pages/exo-comparison/
- throw1234567891 2mo ago> A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline. Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34? Is there a document explaining all constraints of this implementation?
- sig_kill 2mo agoIt depends on the size / structure of the model… but generally you can check out the MeshLLM huggingface org for pre-computed model splits. We have a job that will take popular models, and dice them up and post on HF. When that specific model is served, the layers will be shovelled around the available nodes on the mesh.
- throw1234567891 2mo agoDo you have instructions how to dice models myself? Every model in your org is a measly 4bit. I kinda hoped it is as simple as taking a GGUF and throwing at Mesh LLM but I fear it's not going to be that straightforward :)
- dsegovia 2mo agoThis is surprisingly similar to what I'm trying to build (at least at the network level), down to the use of iroh. I have not attempted model splitting though, and the angle is slightly different. Major props for this, it seems like a great project. The more we can push local inference, the better for everyone. Interested to see if there's collaboration opportunities. https://github.com/dsegovia90/cocompute https://github.com/dsegovia90/cocompute