7 ms·

and is Griffin a state space model?

• imjonse 2y ago
  No, it's a combination of RNN and Transformer.
  • michwilinski 2y ago
    I mean, SSMs are in fact under the hood RNNs
    • VHRanger 2y ago
      At the end of the day, either you carry around a hidden state, or you have a fixed window for autoregression. You can call hidden states "RNN-like" and autoregressive windows "transformer-like", but apart from those two core paradigms I don't know of other ways to do sequence modelling. Mamba/RWKV/Griffin are somewhere between those two extremes.