5 ms·
I would be curious to know if anyone has tried a hybrid approach where you have a Mamba-like architecture for longer term recall but it's combined with a transf
by blainm 3y ago
I would be curious to know if anyone has tried a hybrid approach where you have a Mamba-like architecture for longer term recall but it's combined with a transformer for short term memory?
- enonimal 3y agomaybe a fun karpathy video here...
- logicchains 3y agoYep, https://arxiv.org/abs/2402.04248 https://arxiv.org/abs/2402.04248 tried a Mambaformer which seemed to perform well.