5 ms·

>This one is 552B so almost twice that, so the huge gains in benchmark scores make sense - it's not really flash anymore, imo. It uses fewer active parameters,

>This one is 552B so almost twice that, so the huge gains in benchmark scores make sense - it's not really flash anymore, imo. It uses fewer active parameters, though. (8B or 14B instead of always 13B) So ... flash indeed.

tarruda 6d ago
200B of those 552B is PLE, which works more like a database that is read for each token, thus can be offloaded to a fast SSD.
- asamoahf 6d ago
  [flagged]
- azath92 6d ago
  Id love an ELI5 for PLE. Im trying to work it into my back of the napikin math for compute vs memory bandwidth limitations on tok/s in PP vs TG work. My attempt at a simplification of this article on it https://sebastianraschka.com/llm-architecture-gallery/per-layer-embeddings/ https://sebastianraschka.com/llm-architecture-gallery/per-la... into a couple of sentences is that they are linear embeddings of the input token space projected per layer, which are then gated by the transformer outputs per layer. This would mean that the only one set of weights for the ple path needs to be pumped across the memory bandwidth as they are the same linear weights for all layers? Sheit, maybe im trying to simplify something that i need to look at in detail. but id love to leverage others understanding if possible
  - tarruda 6d ago
    [dead]
  - sixothree 6d ago
    While he avoids using the actual PLE acronym, he does actually describe the concept quite well. I think you may enjoy this video. Specifically around 5 minutes into the video is the part you're looking for. https://www.youtube.com/watch?v=1--PzaHafAU https://www.youtube.com/watch?v=1--PzaHafAU
  - hadlock 5d ago
    >Id love an ELI5 for PLE. PLE is, instead of mixture of experts, mixture of associations
- kmike84 6d ago
  Unfortunately no, it's 200B + 552B. It's not as bad as it sounds though, because most of 552B is in 4bit natively.
  - tarruda 6d ago
    Darn, for a moment I thought it would be possible to run in 128G T.T
    - jasonjmcghee 6d ago
      Actually... https://x.com/antirez/status/2098121665771110540 https://x.com/antirez/status/2098121665771110540
      - tarruda 5d ago
        Awesome!

tarruda 6d ago

200B of those 552B is PLE, which works more like a database that is read for each token, thus can be offloaded to a fast SSD.

asamoahf 6d ago
[flagged]
azath92 6d ago
Id love an ELI5 for PLE. Im trying to work it into my back of the napikin math for compute vs memory bandwidth limitations on tok/s in PP vs TG work. My attempt at a simplification of this article on it https://sebastianraschka.com/llm-architecture-gallery/per-layer-embeddings/ https://sebastianraschka.com/llm-architecture-gallery/per-la... into a couple of sentences is that they are linear embeddings of the input token space projected per layer, which are then gated by the transformer outputs per layer. This would mean that the only one set of weights for the ple path needs to be pumped across the memory bandwidth as they are the same linear weights for all layers? Sheit, maybe im trying to simplify something that i need to look at in detail. but id love to leverage others understanding if possible
- tarruda 6d ago
  [dead]
- sixothree 6d ago
  While he avoids using the actual PLE acronym, he does actually describe the concept quite well. I think you may enjoy this video. Specifically around 5 minutes into the video is the part you're looking for. https://www.youtube.com/watch?v=1--PzaHafAU https://www.youtube.com/watch?v=1--PzaHafAU
- hadlock 5d ago
  >Id love an ELI5 for PLE. PLE is, instead of mixture of experts, mixture of associations
kmike84 6d ago
Unfortunately no, it's 200B + 552B. It's not as bad as it sounds though, because most of 552B is in 4bit natively.
- tarruda 6d ago
  Darn, for a moment I thought it would be possible to run in 128G T.T
  - jasonjmcghee 6d ago
    Actually... https://x.com/antirez/status/2098121665771110540 https://x.com/antirez/status/2098121665771110540
    - tarruda 5d ago
      Awesome!