6 ms·
For anyone wondering, it includes 4 models: 7/13/30/65 billion parameters, the smallest one is 14Gb, the largest one is 131GB, all four are 235Gb.
by aent 4y ago
For anyone wondering, it includes 4 models: 7/13/30/65 billion parameters, the smallest one is 14Gb, the largest one is 131GB, all four are 235Gb.
- mlboss 4y agoIs it possible to run the smallest one on a consumer gpu with 24gb ram ?
- Tepix 4y agoRunning it is easy but you'll probably want to finetune it, too
- rihegher 4y agoI would be surprised if you can't. The smallest weight file is 14gb apparently
- eightysixfour 4y agohttps://github.com/facebookresearch/llama/blob/main/FAQ.md#3 https://github.com/facebookresearch/llama/blob/main/FAQ.md#3 Looks like it needs 14gb for weights and it isn't clear what the minimum size for the decoding cache is, but it defaults to settings for 30gb GPUs.
- MacsHeadroom 4y agoIn int8 7B needs only 9GB of VRAM and 13B needs only 20GB on a single GPU. https://github.com/oobabooga/text-generation-webui/issues/147#issuecomment-1454841458 https://github.com/oobabooga/text-generation-webui/issues/14...
- MacsHeadroom 4y agoYou can do even better!. You can run the second smallest one (better than GPT-3 175B) on 24GB of vram, ie LLaMA-13B. https://github.com/oobabooga/text-generation-webui/issues/147#issuecomment-1454841458 https://github.com/oobabooga/text-generation-webui/issues/14...
- q1w2 4y agoI wonder how many people are scrambling to set this up on their startup infra. 6x24GB NVRAM on 6 GPUs linked with NVSwitch is a little pricey, but totally doable.
- exo-pla-net 4y agoHow pricey would you estimate?
- deleted 4y ago[deleted]
- Tepix 4y agoIf you want to do it the cheap way by buying used stuff, the most expensive parts are: - $2000 for a Threadripper 3xx5WX with a socket sWRX8 mainboard - $5000 for 6x RTX 3090 - $350 for two 1500W PSUs - $700 for 256GB RAM You will also need PCIe extenders and perhaps some watercooling. And find a suitable case. The 2-card NVLink bridges are between $100 and $300 each (you nay want 3). All in all i think less than $10k.
- touristtam 4y agoWould rather put that in AWS.
- Tepix 4y agoAWS is probably among the priciest options. Privacy costs money.
- deleted 4y ago[deleted]
- arthurcolle 4y agoI got it running using Colab Pro+ (immediately got a V100 40GB VRAM GPU) - the 7B model works with batch size of 8 and a max seq len of 1024