6 ms·

The vanguard chiplets even steps down to 30M… but they do claim to have a “Mead” design [1] designed to run GPT-3 in a single chip. 3D NAND flash can indeed ro

by dgfl 25d ago

The vanguard chiplets even steps down to 30M… but they do claim to have a “Mead” design [1] designed to run GPT-3 in a single chip. 3D NAND flash can indeed routinely store hundreds of GB per die, so that’s proven. The question is about all the peripheral circuitry needed. Each attention block would need its own KV cache (i.e. SRAM or DRAM somewhere), plus DAC/ADC inputs and outputs, unless they figure out a way to keep it analog all the way (really cool but unlikely). I think this field is very interesting, at least from a technology point of view. Whether it works out or not will sadly be a matter of economics more than physics I fear. [1] https://www.mythic.ai/mead https://www.mythic.ai/mead

mdp2021 25d ago
> unless they figure out a way to keep it analog all the way (really cool but unlikely) Very unlikely for the connection to the cache RAM, seemingly impossible if we want to get an articulate output :) > they do claim to have a “Mead” design [1] designed to run GPT-3 [ - i.e. to hold a 175b NN - ] in a single chip Careful: that is the /intention/, but the chip is just the storage (and CiM) for the NN and other parts are missing - explained in the Vanguard, not explained in the Mead. > Whether it works out or not will sadly be a matter of economics more than physics I fear Yes, but: -- what is not enabled today may often be tomorrow through advances, esp. in the economy of production; -- a very great point about this product seems to be that the chips can be rewritten, the NN is not etched and static (cpr. Taalas): that makes the practical implications extremely relevant, the demand would be "screaming mob" like; -- we have to go in that direction (of NNs in CiM) anyway, so it's just a matter of time, effort after effort we will get there.
K0balt 24d ago
This might end up being a strong use case for FeFET FRAM, nonvolatile, high density, and sram fast. In an alternative universe where fram infra was as widespread as flash, memory is fast, cheap, and nonvolatile. It’s just that flash built the infrastructure and climbing that ladder is nearly impossible.
- dgfl 24d ago
  I wouldn't really say SRAM-fast. Maybe DRAM-fast? At least in write speed. But the thing is, you don't need high write speed for this application. And read speed should be the same between flash and FeFETs. Btw, I've seen some roadmaps in which foundries were heading towards integrating ferroelectrics and floating gates to accelerate switching. The two technologies are not mutually exclusive really, so I think we might see something in the relatively near future. The crazy memory market may help!

by dgfl 25d ago

mdp2021 25d ago

> unless they figure out a way to keep it analog all the way (really cool but unlikely) Very unlikely for the connection to the cache RAM, seemingly impossible if we want to get an articulate output :) > they do claim to have a “Mead” design [1] designed to run GPT-3 [ - i.e. to hold a 175b NN - ] in a single chip Careful: that is the /intention/, but the chip is just the storage (and CiM) for the NN and other parts are missing - explained in the Vanguard, not explained in the Mead. > Whether it works out or not will sadly be a matter of economics more than physics I fear Yes, but: -- what is not enabled today may often be tomorrow through advances, esp. in the economy of production; -- a very great point about this product seems to be that the chips can be rewritten, the NN is not etched and static (cpr. Taalas): that makes the practical implications extremely relevant, the demand would be "screaming mob" like; -- we have to go in that direction (of NNs in CiM) anyway, so it's just a matter of time, effort after effort we will get there.

K0balt 24d ago

This might end up being a strong use case for FeFET FRAM, nonvolatile, high density, and sram fast. In an alternative universe where fram infra was as widespread as flash, memory is fast, cheap, and nonvolatile. It’s just that flash built the infrastructure and climbing that ladder is nearly impossible.

dgfl 24d ago
I wouldn't really say SRAM-fast. Maybe DRAM-fast? At least in write speed. But the thing is, you don't need high write speed for this application. And read speed should be the same between flash and FeFETs. Btw, I've seen some roadmaps in which foundries were heading towards integrating ferroelectrics and floating gates to accelerate switching. The two technologies are not mutually exclusive really, so I think we might see something in the relatively near future. The crazy memory market may help!