8 ms·
> Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregat
by inferencecoder 2mo ago
> Not to mention no one serious is serving this on 8xB200 instead of multiple nodes: the vast majority of Moonshot's inference work is focused on PD-disaggregation
The GPU price discourse is absurd, but many are serving models on single node setups when the model fits
- BoorishBears 1mo agoYou can't beat current API pricing running Kimi K3 a single node, so as I said anyone serious is not doing that for Kimi K3. Not sure how you're getting "no one serves models on a single node" out of that.
- inferencecoder 1mo agoI meant serious players often do serve on a single node. They can beat API pricing as well. Multi-node can add gain, but also adds a lot of deployment complexity so its just not always possible or optimal.