6 ms·
This argument works better for state space models. A transformer would still steps context one token at a time, not maintain an internal 1e18 state.
by rar00 1y ago
This argument works better for state space models. A transformer would still steps context one token at a time, not maintain an internal 1e18 state.
- mgraczyk 1y agoThat doesn't matter, are you familiar with any theoretical results in which the computation is somehow limited in ways that practically matter when the context length is very long? I am not