6 ms·

I wonder why everyone keep saying "just put more VRAM" yet no cards seem to do that. If it is that easy to compete with Nvidia, why don't we already have those

I wonder why everyone keep saying "just put more VRAM" yet no cards seem to do that. If it is that easy to compete with Nvidia, why don't we already have those cards?

rocqua 1y ago
I believe that VRAM has massively shot up in price, so this is where a large part of the costs are. Besides I wouldn't be very surprised if Nvidia has such strong market share they can effectively tell suppliers to not let others sell high capacity cards. Especially because VRAM suppliers might worry about ramping up production too much and then being left with an oversupply situation.
- kokada 1y ago
  This could well be the reason why the rumored RDNA5 will use LPDDR5X/LPDDR5X instead of GDDR7 memory, at least for the low/mid range configurations (the top-spec and enthusiast configurations AT0 and AT2 configurations will still use GDDR7 it seems).
  - FuriouslyAdrift 1y ago
    AFAIK, RDNA5 has been cancelled as AMD is moving back to a unified architecture with their Instinct and Radeon lines.
    - kokada 1y ago
      It is not really clear if it will be called as UDNA or RDNA5, I was just referring to the next-gen graphics architecture from AMD and referring as RDNA5 is just clearer that this is the next-gen architecture.
- GTP 1y ago
  > Especially because VRAM suppliers might worry about ramping up production too much and then being left with an oversupply situation. Given the high demand of graphic cards, is this a plausible scenario?
  - williamdclt 1y ago
    I don't really know what I'm talking about (whether about graphic cards or in AI inference), but if someone figures out how to cut the compute needed for AI inference significantly then I'd guess the demand for graphic cards would suddenly drop? Given how young and volatile this domain still is, it doesn't seem unreasonable to be wary of it. Big players (google, openai and the likes) are probably pouring tons of money into trying to do exactly that
    - rtrgrd 1y ago
      I would suspect that for self hosted LLMs, quality >>> performance, so the newer releases will always expand to fill capacity of available hardware even when efficiency is improved.
blkhawk 1y ago
because the cards already sell at very very good prices with 16GB and optimizations in generative AI is bringing down memory requirements. Optimizing profits means yyou sell with the least amount of VRAM possible not only to save the direct cost of the RAM but also to guard future profit and your other market segments. the cost of the ram itself is almost nothing compared to that. any intel competitor can more easily release products with more than 16GB and smoke them. Intel tries for a market segment that was only served by gaming cards twice as expensive up until now. this frees those up to be finally sold at MSRP.
- betimsl 1y ago
  Right, but Intel is in no position to do that. So, they could play and put 32GB VRAM and also, why not produce one with 64GB just for kicks?
  - Workaccount2 1y ago
    If intel was serious about staging a comeback, they would release a 64GB card. But intel is still lost in it's hubris, and still thinks it's a serious player and "one of the boys", so it doesn't seem like they want to break the line.
agilob 1y ago
AMD is growing up into doing that, they have a few decent cards with 20Gb and 24Gb now
fwipsy 1y ago
Maybe because only AI enthusiasts want that much VRAM, and most of them will pony up for a higher-end GPU anyways? Everyone is suggesting it here because that's what they want, but I don't know if this crowd is really representative of broader market sentiment.
- vid 1y ago
  There are a lot of local AI hobbyists, just visit /r/LocalLLama to see how many are using 8GB cards, or all the people asking for higher RAM version of cards. This makes it mysterious since clearly CUDA is an advantage, but higher VRAM lower cost cards with decent open library support would be compelling.
  - rdos 1y ago
    There is no point in using a low-bandwidth card like the B50 for AI. Attempting to use 2x or 4x cards to load a real model will result in poor performance and low generation speed. If you don’t need a larger model, use a 3060 or 2x 3060, and you’ll get significantly better performance than the B50—so much better that the higher power consumption won’t matter (70W vs. 170W for a single card). Higher VRAM wont make the card 'better for AI'.
    - vid 1y ago
      Who said anything about the B50? Plenty of people use eg 2, 4 or 6 3090s to run large models at acceptable speeds. Higher VRAM at decent (much faster than DDR5) speeds will make cards better for AI.
      - hadlock 1y ago
        If VRAM is ~$10/gb I suspect people paying $450 for a 12GB card would be happy to pay $1200 for a 64gb card. Running local LLM only uses about 3-6% of my GPU's capability, but all of it's VRAM. Local LLM has no need for 6 3090s to serve a single or handful of users; they just need the VRAM to run the model locally.
PunchyHamster 1y ago
Because if they do their higher models will sell worse.
izacus 1y ago
Because: - less people care about VRAM than HN commenters give impression of - VRAM is expensive and wouldn't make such cards profitable at the HN desired price points
qudat 1y ago
I'm willing to bet there are technical limitations to just "adding more VRAM" to all these boards.
danielEM 1y ago
Ryzen AI max+ 395 128GB can do 256GBps so lets put all these "ifs" to bed once for all. That is absolutely no brainer to drop more RAM as long as there is enough bits in address space of physical hardware. And there usually is, as same silicons are branded and packaged differently for commercial market and for consumer market. Check up how chinese are doubling 4090s RAM from 24 to 48GB.
robotnikman 1y ago
There does seem to be a grey market for it in China. You can buy cards where they swap the memory modules with higher capacity ones on Aliexpress and ebay.
bsder 1y ago
> If it is that easy to compete with Nvidia, why don't we already have those cards? Businesswise? Because Intel management are morons. And because AMD, like Nvidia, don't want to cannibalize their high end. Technically? "Double the RAM" is the most straightforward (that doesn't make it easy, necessarily ...) way to differentiate as it means that training sets you couldn't run yesterday because it wouldn't fit on the card can now be run today. It also takes a direct shot at how Nvidia is doing market segmentation with RAM sizes. Note that "double the RAM" is necessary but not sufficient. You need to get people to port all the software to your cards to make them useful. To do that, you need to have something compelling about the card. These Intel cards have nothing compelling about them. Intel could also make these cards compelling by cutting the price in half or dropping two dozen of these cards on every single AI department in the US for free. Suddenly, every single grad student in AI will know everything about your cards. The problem is that Intel institutionally sees zero value in software and is incapable of making the moves they need to compete in this market. Since software isn't worth anything to Intel, there is no way to justify any business action isn't just "sell (kinda shitty) chip".

I wonder why everyone keep saying "just put more VRAM" yet no cards seem to do that. If it is that easy to compete with Nvidia, why don't we already have those cards?

rocqua 1y ago

I believe that VRAM has massively shot up in price, so this is where a large part of the costs are. Besides I wouldn't be very surprised if Nvidia has such strong market share they can effectively tell suppliers to not let others sell high capacity cards. Especially because VRAM suppliers might worry about ramping up production too much and then being left with an oversupply situation.

kokada 1y ago
This could well be the reason why the rumored RDNA5 will use LPDDR5X/LPDDR5X instead of GDDR7 memory, at least for the low/mid range configurations (the top-spec and enthusiast configurations AT0 and AT2 configurations will still use GDDR7 it seems).
- FuriouslyAdrift 1y ago
  AFAIK, RDNA5 has been cancelled as AMD is moving back to a unified architecture with their Instinct and Radeon lines.
  - kokada 1y ago
    It is not really clear if it will be called as UDNA or RDNA5, I was just referring to the next-gen graphics architecture from AMD and referring as RDNA5 is just clearer that this is the next-gen architecture.
GTP 1y ago
> Especially because VRAM suppliers might worry about ramping up production too much and then being left with an oversupply situation. Given the high demand of graphic cards, is this a plausible scenario?
- williamdclt 1y ago
  I don't really know what I'm talking about (whether about graphic cards or in AI inference), but if someone figures out how to cut the compute needed for AI inference significantly then I'd guess the demand for graphic cards would suddenly drop? Given how young and volatile this domain still is, it doesn't seem unreasonable to be wary of it. Big players (google, openai and the likes) are probably pouring tons of money into trying to do exactly that
  - rtrgrd 1y ago
    I would suspect that for self hosted LLMs, quality >>> performance, so the newer releases will always expand to fill capacity of available hardware even when efficiency is improved.

blkhawk 1y ago

because the cards already sell at very very good prices with 16GB and optimizations in generative AI is bringing down memory requirements. Optimizing profits means yyou sell with the least amount of VRAM possible not only to save the direct cost of the RAM but also to guard future profit and your other market segments. the cost of the ram itself is almost nothing compared to that. any intel competitor can more easily release products with more than 16GB and smoke them. Intel tries for a market segment that was only served by gaming cards twice as expensive up until now. this frees those up to be finally sold at MSRP.

betimsl 1y ago
Right, but Intel is in no position to do that. So, they could play and put 32GB VRAM and also, why not produce one with 64GB just for kicks?
- Workaccount2 1y ago
  If intel was serious about staging a comeback, they would release a 64GB card. But intel is still lost in it's hubris, and still thinks it's a serious player and "one of the boys", so it doesn't seem like they want to break the line.

agilob 1y ago

AMD is growing up into doing that, they have a few decent cards with 20Gb and 24Gb now

fwipsy 1y ago

Maybe because only AI enthusiasts want that much VRAM, and most of them will pony up for a higher-end GPU anyways? Everyone is suggesting it here because that's what they want, but I don't know if this crowd is really representative of broader market sentiment.

vid 1y ago
There are a lot of local AI hobbyists, just visit /r/LocalLLama to see how many are using 8GB cards, or all the people asking for higher RAM version of cards. This makes it mysterious since clearly CUDA is an advantage, but higher VRAM lower cost cards with decent open library support would be compelling.
- rdos 1y ago
  There is no point in using a low-bandwidth card like the B50 for AI. Attempting to use 2x or 4x cards to load a real model will result in poor performance and low generation speed. If you don’t need a larger model, use a 3060 or 2x 3060, and you’ll get significantly better performance than the B50—so much better that the higher power consumption won’t matter (70W vs. 170W for a single card). Higher VRAM wont make the card 'better for AI'.
  - vid 1y ago
    Who said anything about the B50? Plenty of people use eg 2, 4 or 6 3090s to run large models at acceptable speeds. Higher VRAM at decent (much faster than DDR5) speeds will make cards better for AI.
    - hadlock 1y ago
      If VRAM is ~$10/gb I suspect people paying $450 for a 12GB card would be happy to pay $1200 for a 64gb card. Running local LLM only uses about 3-6% of my GPU's capability, but all of it's VRAM. Local LLM has no need for 6 3090s to serve a single or handful of users; they just need the VRAM to run the model locally.

PunchyHamster 1y ago

Because if they do their higher models will sell worse.

izacus 1y ago

Because: - less people care about VRAM than HN commenters give impression of - VRAM is expensive and wouldn't make such cards profitable at the HN desired price points

qudat 1y ago

I'm willing to bet there are technical limitations to just "adding more VRAM" to all these boards.

danielEM 1y ago

Ryzen AI max+ 395 128GB can do 256GBps so lets put all these "ifs" to bed once for all. That is absolutely no brainer to drop more RAM as long as there is enough bits in address space of physical hardware. And there usually is, as same silicons are branded and packaged differently for commercial market and for consumer market. Check up how chinese are doubling 4090s RAM from 24 to 48GB.

robotnikman 1y ago

There does seem to be a grey market for it in China. You can buy cards where they swap the memory modules with higher capacity ones on Aliexpress and ebay.

bsder 1y ago

> If it is that easy to compete with Nvidia, why don't we already have those cards? Businesswise? Because Intel management are morons. And because AMD, like Nvidia, don't want to cannibalize their high end. Technically? "Double the RAM" is the most straightforward (that doesn't make it easy, necessarily ...) way to differentiate as it means that training sets you couldn't run yesterday because it wouldn't fit on the card can now be run today. It also takes a direct shot at how Nvidia is doing market segmentation with RAM sizes. Note that "double the RAM" is necessary but not sufficient. You need to get people to port all the software to your cards to make them useful. To do that, you need to have something compelling about the card. These Intel cards have nothing compelling about them. Intel could also make these cards compelling by cutting the price in half or dropping two dozen of these cards on every single AI department in the US for free. Suddenly, every single grad student in AI will know everything about your cards. The problem is that Intel institutionally sees zero value in software and is incapable of making the moves they need to compete in this market. Since software isn't worth anything to Intel, there is no way to justify any business action isn't just "sell (kinda shitty) chip".