6 ms·
Can't access the model link due to sone MS auth issue. Does anyone know how large (in GB) the model is? Can it be run locally or is it azure-only?
by collaborative 3y ago
Can't access the model link due to sone MS auth issue. Does anyone know how large (in GB) the model is? Can it be run locally or is it azure-only?
- sytelus 3y agoFile size on disk is ~10GB.
- armcat 3y agoInteresting, so they are using single precision (fp32), so 2.7B x 4 Bytes = ~10 GB. With CUDA overhead and room for context, you would need at least 12GB VRAM. They could use half precision and half that VRAM requirement and save costs for everyone involved. Maybe there is a performance reason why they use full precision.
- collaborative 3y agoYes, the reason I asked is that when I see SLM I got all excited thinking "finally a small model that fits in cheap hardware for simpler tasks"
- eightysixfour 3y agoModels at this size are regularly quantized to 5/4bit to reduce the size. While there is some degradation, it isn’t as substantial as you would expect.
- deleted 3y ago[deleted]
- sigmar 3y agoThat's just the parameters? So 32 bit parameters? This blogpost is incredibly misleading by putting Gemini nano-2's "size" as larger than Phi-2 (in Table 2 displaying only the number of parameters) and saying "Phi-2 matches or outperforms the recently-announced Google Gemini Nano 2, despite being smaller in size." Because Gemini nano parameters are 4 bit. So Gemini nano-2 is 1.6 GB (3.25/2) in size compared to Phi-2's 10GB
- stavros 3y agoLLM size isn't measured in bytes, but in parameters.
- pugworthy 3y agoThis? "Selected user account does not exist in tenant 'Microsoft' and cannot access the application 'd7304df8-741f-47d3-9bc2-df0e24e2071f' in that tenant. The account needs to be added as an external user in the tenant first. Please use a different account."
- collaborative 3y agoyup