6 ms·
I did a quick scan of the repo and didn't see any reference to Ray. Would this indicate that llm-d lacks support for pipeline parallelism?
by dzr0001 1y ago
I did a quick scan of the repo and didn't see any reference to Ray. Would this indicate that llm-d lacks support for pipeline parallelism?
- qntty 1y agoI believe this is a question you should ask about vLLM, not llm-d. It looks like vLLM does support pipeline parallelism via Ray: https://docs.vllm.ai/en/latest/serving/distributed_serving.html#running-vllm-on-a-single-node https://docs.vllm.ai/en/latest/serving/distributed_serving.h... This project appears to make use of both vLLM and Inference Gateway (an official Kubernetes extension to the Gateway resource). The contributions of llm-d itself seems to mostly be a scheduling algorithm for load balancing across vLLM instances.
- smarterclayton 1y agoWe inherit any multi-host support from vLLM, so https://docs.vllm.ai/en/latest/serving/distributed_serving.html#running-vllm-on-multiple-nodes https://docs.vllm.ai/en/latest/serving/distributed_serving.h... would be the expected path. We plan to publish examples of multi-host inference that leverages LeaderWorkerSets - https://github.com/kubernetes-sigs/lws https://github.com/kubernetes-sigs/lws - which helps run ranked serving workloads across hosts. LeaderWorkerSet is how Google supports both TPU and GPU multi-host deployments - see https://github.com/kubernetes-sigs/lws/blob/main/config/samples/leaderworkerset_tpu.yaml https://github.com/kubernetes-sigs/lws/blob/main/config/samp... for an example. Edit: Here is an example Kubernetes configuration running DeepSeek-R1 on vLLM multi-host using LeaderWorkerSet https://github.com/kubernetes-sigs/wg-serving/blob/main/serving-catalog/core/lws/vllm/deepseek-r1/gke/README.md https://github.com/kubernetes-sigs/wg-serving/blob/main/serv.... This work would be integrated into llm-d.