5 ms·

Two or three top GPUs? Thats basically nothing for a professinal project or even an investeded hobby

by lfkdev 4y ago

throwifasd 4y ago
[flagged]
- ronsor 4y ago
  A few 4090Ti GPUs might be good enough, if they do indeed end up with 48GB of VRAM.
  - deleted 4y ago
    [deleted]
  - firsttimebigboy 4y ago
    According to wikipedia ChatGPT is built on GPT-3 which has 175B parameters requiring 800GB of memory, so a few 4090s will fall short.
    - samr71 4y ago
      Let's say the 4090ti is priced around $2500. 16 of them would set you back just $40 grand. That's quite a bit for the average consumer, but people spend that kinda dough on cars/boats/planes etc. It's well within the realm of dedicated hobbyists. For companies it's nothing. Someone will come along selling pre-builts for this purpose, if that's not already available for their current research/enterprise cards.
- wokwokwok 4y ago
  You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L59 https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5... /shrug If you don't want to run it locally, don't. If the hardware you have doesn't support it, don't. ...but to say it's not possible is, again, bluntly, wrong.
  - throwifasd 4y ago
    yeah, that's not what op asked about. That's one of those smaller models that "suck" as you said so confidently in your first comment. Not even close to gpt3/chatgpt. Next time when I talk bullshit I just post a random py file to make my argument. lmao
    - adam_arthur 4y ago
      Though if true, it does beg the question, how can these services possibly be offered profitably if a single query requires many high end GPUs to run? Doesn't really track logically either. Though I'm not in the know. Perhaps ChatGPT is run with massive operating losses and its all VC subsidized.
      - throwifasd 4y ago
        This post [0] is a good primer and they are talking about a 20b model gpt3 has 175b. Their other posts go more into details but yes, it really is a massive operation. Some hard facts from here [1] talking about BLOOM with 175b parameters.. >> Installing the full 175B version is a challenge though as it requires around 350GB of GPU VRAM, which is not something one can easily afford. But hey why deploy a model like chatGPT at all when you guys can be confidently incorrect all by yourselves. [0] https://nlpcloud.com/deploying-gpt-neox-20-production-focus-deepspeed.html https://nlpcloud.com/deploying-gpt-neox-20-production-focus-... [1] https://nlpcloud.com/chatgpt-open-source-alternatives.html https://nlpcloud.com/chatgpt-open-source-alternatives.html
        adam_arthur 4y ago
        So it sounds like this is a question of loading the model into VRAM, and not a question of the cost of a single query. I assume once a model is loaded, many queries can be serviced by that model quickly. There's nothing incorrect about my assertion. If it were to actually take many GPUs to service one query, then there is no mass scale cost viable consumer product. That's just a clear economic fact. Regardless if a model could be theoretically spun up in a cost inefficient manner. And even 100s of GB of VRAM is not far off from consumer hardware. Look at how quickly graphics ram has expanded over time. About ~10x in ~10 years for high end cards, at a cursory glance at various Nvidia cards. At the same trajectory we could see a 400GB vram card within the next decade (though lots of assumptions)

Two or three top GPUs? Thats basically nothing for a professinal project or even an investeded hobby

by lfkdev 4y ago

Two or three top GPUs? Thats basically nothing for a professinal project or even an investeded hobby

throwifasd 4y ago

[flagged]

ronsor 4y ago
A few 4090Ti GPUs might be good enough, if they do indeed end up with 48GB of VRAM.
- deleted 4y ago
  [deleted]
- firsttimebigboy 4y ago
  According to wikipedia ChatGPT is built on GPT-3 which has 175B parameters requiring 800GB of memory, so a few 4090s will fall short.
  - samr71 4y ago
    Let's say the 4090ti is priced around $2500. 16 of them would set you back just $40 grand. That's quite a bit for the average consumer, but people spend that kinda dough on cars/boats/planes etc. It's well within the realm of dedicated hobbyists. For companies it's nothing. Someone will come along selling pre-builts for this purpose, if that's not already available for their current research/enterprise cards.
wokwokwok 4y ago
You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L59 https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5... /shrug If you don't want to run it locally, don't. If the hardware you have doesn't support it, don't. ...but to say it's not possible is, again, bluntly, wrong.
- throwifasd 4y ago
  yeah, that's not what op asked about. That's one of those smaller models that "suck" as you said so confidently in your first comment. Not even close to gpt3/chatgpt. Next time when I talk bullshit I just post a random py file to make my argument. lmao
  - adam_arthur 4y ago
    Though if true, it does beg the question, how can these services possibly be offered profitably if a single query requires many high end GPUs to run? Doesn't really track logically either. Though I'm not in the know. Perhaps ChatGPT is run with massive operating losses and its all VC subsidized.
    - throwifasd 4y ago
      This post [0] is a good primer and they are talking about a 20b model gpt3 has 175b. Their other posts go more into details but yes, it really is a massive operation. Some hard facts from here [1] talking about BLOOM with 175b parameters.. >> Installing the full 175B version is a challenge though as it requires around 350GB of GPU VRAM, which is not something one can easily afford. But hey why deploy a model like chatGPT at all when you guys can be confidently incorrect all by yourselves. [0] https://nlpcloud.com/deploying-gpt-neox-20-production-focus-deepspeed.html https://nlpcloud.com/deploying-gpt-neox-20-production-focus-... [1] https://nlpcloud.com/chatgpt-open-source-alternatives.html https://nlpcloud.com/chatgpt-open-source-alternatives.html
      - adam_arthur 4y ago
        So it sounds like this is a question of loading the model into VRAM, and not a question of the cost of a single query. I assume once a model is loaded, many queries can be serviced by that model quickly. There's nothing incorrect about my assertion. If it were to actually take many GPUs to service one query, then there is no mass scale cost viable consumer product. That's just a clear economic fact. Regardless if a model could be theoretically spun up in a cost inefficient manner. And even 100s of GB of VRAM is not far off from consumer hardware. Look at how quickly graphics ram has expanded over time. About ~10x in ~10 years for high end cards, at a cursory glance at various Nvidia cards. At the same trajectory we could see a 400GB vram card within the next decade (though lots of assumptions)