6 ms·

Alibaba Cloud claims to reduce Nvidia GPU used for serving unpopular models by 82% (emphasis mine) > 17.7 per cent of GPUs allocated to serve only 1.35 per cen

Alibaba Cloud claims to reduce Nvidia GPU used for serving unpopular models by 82% (emphasis mine) > 17.7 per cent of GPUs allocated to serve only 1.35 per cent of requests in Alibaba Cloud’s marketplace, the researchers found Instead of 1192 GPUs they now use 213 for serving those requests.

yorwba 11mo ago
Not really, Figure 1(a) of the paper says that the 17.7% are relative to a total of 30k GPUs (i.e. 5310 GPUs for handling those 1.35% of requests) and the reduction is measured in a smaller beta deployment with only 47 different models (vs. the 733 "cold" models overall.) Naïve extrapolation by model count suggests they would need 3321 GPUs to serve all cold models, a 37.5% reduction to before. (Or 6.6% reduction of the full 30k-GPU cluster.)
- somerandomdude2 11mo ago
  Really: "A paper presented at SOSP 2025 details how token-level scheduling helped one GPU serve multiple LLMs, reducing demand from 1,192 to 213 H20s." Which, if you scale it, matches the GPs statement.
  - yorwba 11mo ago
    From the SCMP article you might get the impression that the various figures all refer to the same GPU cluster, but in the paper itself it's very clear that this is not the case, i.e. the 213 GPUs in the smaller cluster are not serving 1.35% of the requests in the larger cluster. Then if you want to scale it, you have a choice of different numbers you could scale, and each would get different results. Since they're constrained by the limited number of different models a single GPU can serve, I think scaling by the number of models is the most realistic option.
MangoCoffee 11mo ago
In the past, software and computer engineers would tackle problems head-on, designing algorithms and finding creative solutions. thanks to the US restrictions on semiconductor industry (Chinese), Chinese engineers are being forced to innovate and find their own ways to overcome challenges like the old school engineers (What Silicon Valley used to be)
- _heimdall 11mo ago
  If you're one who sees progress as an end goal unto itself, what you describe is a good thing. When one party is attempting novel solutions to outcompete the competition we will be faster to whatever the next change is. That said, I'm not sure what the US policies specifically have to do with this. Countries are always in competition with one another, and if one industry or technology is considered a national security threat they will guard it.
  - coliveira 11mo ago
    If AI is a threat to other nations, why is anyone even supporting this? Are we really trying to annihilate the planet as quickly as possible?
    - _heimdall 11mo ago
      For the same reason we built nuclear bombs, "if I don't my enemies will"
      - coliveira 11mo ago
        AI is a bomb that has already been released.
      - coliveira 11mo ago
        Are we creating enemies so quickly like this?
bee_rider 11mo ago
I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed? (I’m not an AI guy, though—actually I’m used to asking SLURM for new nodes with every run I do!)
- make3 11mo ago
  the models are huge, so not a single (latest gen) one can fit on a single GPU. It's likely that these are small unpopular (non flagship) models, or that they only pack eg one layer of each model.
  - svachalek 11mo ago
    Per the very short article, the solution was to pack multiple models per GPU.
    - make3 11mo ago
      yes but that could mean a layer per model
- smallnix 11mo ago
  > I guess I’d assumed this sort of thing would be allocated dynamically At the scale of a hyperscaler I think Alibaba is the one that would be doing that. AWS, Azure and I assume Alibaba do lease/rent data centers, but someone has to own the servers / GPU racks. I know there are specialized companies like nscale (and more further down the chain) in the mix, but I always assumed they only lease out fixed capacity.
- yorwba 11mo ago
  The paper is about techniques to do that dynamic allocation to maximize utilization without incurring unacceptable latencies. If you let a GPU sit idle for several minutes after serving a single request, you're setting money on fire. So they reuse it for a different model as soon as possible, starting even before the first request is finished, because: If you don't have a dedicated GPU for a model, are you going to wait for a multi-gigabyte transfer before each request? So they have a dedicated GPU (or two, one for prefill, one for decode) for a group of models that are processed in an interleaved fashion, scheduled such that they stay within the latency budget.
hinkley 11mo ago
So 82% of 17.7%? 14.5% is worth a raise at least. But it’s still misleading.
- abejfehr 11mo ago
  I don't think that's what this is saying, isn't it that 100 - ~82 = 17.7% ?
  - hinkley 11mo ago
    That is a confusing coincidence, but no. > Reserving full GPU instances for these models leads to allocating 17.7% of our GPUs to serve only 1.35% of requests > Deployment results show that Aegaeon reduces the number of GPUs required for serving these models from 1,192 to 213, highlighting an 82% GPU resource saving. 82% of their CPUs were serving 98.6% of all traffic. If they reduced the cluster size, they got it to 96.2% of their CPUs serving 98.6% of their traffic. If they reallocated those, which is more likely, then 96.8% of their CPUs are serving 98.6% of all requests, or around 17% more capacity for popular requests on the same hardware.
xor1101 11mo ago
Doesnt sound right

yorwba 11mo ago

Not really, Figure 1(a) of the paper says that the 17.7% are relative to a total of 30k GPUs (i.e. 5310 GPUs for handling those 1.35% of requests) and the reduction is measured in a smaller beta deployment with only 47 different models (vs. the 733 "cold" models overall.) Naïve extrapolation by model count suggests they would need 3321 GPUs to serve all cold models, a 37.5% reduction to before. (Or 6.6% reduction of the full 30k-GPU cluster.)

somerandomdude2 11mo ago
Really: "A paper presented at SOSP 2025 details how token-level scheduling helped one GPU serve multiple LLMs, reducing demand from 1,192 to 213 H20s." Which, if you scale it, matches the GPs statement.
- yorwba 11mo ago
  From the SCMP article you might get the impression that the various figures all refer to the same GPU cluster, but in the paper itself it's very clear that this is not the case, i.e. the 213 GPUs in the smaller cluster are not serving 1.35% of the requests in the larger cluster. Then if you want to scale it, you have a choice of different numbers you could scale, and each would get different results. Since they're constrained by the limited number of different models a single GPU can serve, I think scaling by the number of models is the most realistic option.

MangoCoffee 11mo ago

In the past, software and computer engineers would tackle problems head-on, designing algorithms and finding creative solutions. thanks to the US restrictions on semiconductor industry (Chinese), Chinese engineers are being forced to innovate and find their own ways to overcome challenges like the old school engineers (What Silicon Valley used to be)

_heimdall 11mo ago
If you're one who sees progress as an end goal unto itself, what you describe is a good thing. When one party is attempting novel solutions to outcompete the competition we will be faster to whatever the next change is. That said, I'm not sure what the US policies specifically have to do with this. Countries are always in competition with one another, and if one industry or technology is considered a national security threat they will guard it.
- coliveira 11mo ago
  If AI is a threat to other nations, why is anyone even supporting this? Are we really trying to annihilate the planet as quickly as possible?
  - _heimdall 11mo ago
    For the same reason we built nuclear bombs, "if I don't my enemies will"
    - coliveira 11mo ago
      AI is a bomb that has already been released.
    - coliveira 11mo ago
      Are we creating enemies so quickly like this?

bee_rider 11mo ago

I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use? I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed? (I’m not an AI guy, though—actually I’m used to asking SLURM for new nodes with every run I do!)

make3 11mo ago
the models are huge, so not a single (latest gen) one can fit on a single GPU. It's likely that these are small unpopular (non flagship) models, or that they only pack eg one layer of each model.
- svachalek 11mo ago
  Per the very short article, the solution was to pack multiple models per GPU.
  - make3 11mo ago
    yes but that could mean a layer per model
smallnix 11mo ago
> I guess I’d assumed this sort of thing would be allocated dynamically At the scale of a hyperscaler I think Alibaba is the one that would be doing that. AWS, Azure and I assume Alibaba do lease/rent data centers, but someone has to own the servers / GPU racks. I know there are specialized companies like nscale (and more further down the chain) in the mix, but I always assumed they only lease out fixed capacity.
yorwba 11mo ago
The paper is about techniques to do that dynamic allocation to maximize utilization without incurring unacceptable latencies. If you let a GPU sit idle for several minutes after serving a single request, you're setting money on fire. So they reuse it for a different model as soon as possible, starting even before the first request is finished, because: If you don't have a dedicated GPU for a model, are you going to wait for a multi-gigabyte transfer before each request? So they have a dedicated GPU (or two, one for prefill, one for decode) for a group of models that are processed in an interleaved fashion, scheduled such that they stay within the latency budget.

hinkley 11mo ago

So 82% of 17.7%? 14.5% is worth a raise at least. But it’s still misleading.

abejfehr 11mo ago
I don't think that's what this is saying, isn't it that 100 - ~82 = 17.7% ?
- hinkley 11mo ago
  That is a confusing coincidence, but no. > Reserving full GPU instances for these models leads to allocating 17.7% of our GPUs to serve only 1.35% of requests > Deployment results show that Aegaeon reduces the number of GPUs required for serving these models from 1,192 to 213, highlighting an 82% GPU resource saving. 82% of their CPUs were serving 98.6% of all traffic. If they reduced the cluster size, they got it to 96.2% of their CPUs serving 98.6% of their traffic. If they reallocated those, which is more likely, then 96.8% of their CPUs are serving 98.6% of all requests, or around 17% more capacity for popular requests on the same hardware.

xor1101 11mo ago

Doesnt sound right