5 ms·
Only 2b active also - very fast
by conwayanderson 1y ago
Only 2b active also - very fast
- lawlessone 1y agoCan run it on a phone then? Seems like it could be somewhat useful for people with poor eyesight or blindness
- conwayanderson 1y agoIn terms of size yes, but I think it needs some work to get the model in the right format couple people got it running on a raspberry pi though
- apwell23 1y agosorry what does it mean for only 2b to be active?
- simonw 1y agoMy understanding is that, while all 8B are loaded into memory, for each token inference step only 2B are selected and used - so tokens are produced faster because there is less computation needed. Hoping someone will correct me if that's not the right mental model!