8 ms·
For my grug brain can somebody translate this to ELIgrug terms? Does this mean I would be able to run 500b model on my 48gb macbook without loosing quality?
by ssijak 6mo ago
For my grug brain can somebody translate this to ELIgrug terms?
Does this mean I would be able to run 500b model on my 48gb macbook without loosing quality?
- prabal97 6mo agoI wrote this more intuitive explanation. I think you might find it helpful! https://prabal.ca/posts/google-long-context-cheaper/ https://prabal.ca/posts/google-long-context-cheaper/
- x_may 6mo agoKV cache compression, so how much memory the model needs to use for extending its context. Does not affect the weight size.