hacklas

5 ms·

1. 1.▲
   by hacklas 1mo ago

   Thanks. I agree, GGUF and upstream contributions are on my radar.
2. 2.▲
   Show HN: I shrank DeepSeek V4 Flash to 57GB and it wrote a compiler on my Mac(huggingface.co)

   21 points by hacklas 1mo ago|3 comments
3. 3.▲
   by hacklas 2mo ago

   For a dense model this would be a limitation, but not all of a MoE model needs to be in memory, but the largest part of a MoE are the routed experts. Some parts are needed to generated every single token and these really should fit in memor
4. 4.▲
   by hacklas 2mo ago

   Worth mentioning why this is harder than it looks. There is a different set of experts at every layer, and each layer has a small router that decides which ones to use. The router needs to look at the state produced by the experts below it.
5. 5.▲
   by hacklas 2mo ago

   How large? With 64 GB of unified memory, you should be able to run a DeepSeek V4 Flash quantisation at 7–10 t/s, for example with: https://github.com/antirez/ds4 or https://github.com/steadfastgaze
6. 6.▲
   by hacklas 1y ago

   The Deepseek V3 is a model with 671 billion parameters, of which 37 billion are active. Magistral Small is a 24 billion parameter model. Pretty impressive in terms of efficiency for Mistral. The size of the Magistral Medium is not publicly