6 ms·
No, an autoregressive language model is conditioned on all prior states, not the previous one.
by pigscantfly 4y ago
No, an autoregressive language model is conditioned on all prior states, not the previous one.
- blueflow 4y agoMultiply out the states, "all prior states" is then the "previous one". Easy to model as Markov chain.
- adgjlsfhk1 4y agothis is technically true but the Markov chain would be too big to store even with petabytes of storage.
- tgv 4y agoIndeed. The argument boils down to: since it's finite, I can turn it into a FSA. Not only is that unhelpful, it doesn't tell you how to construct it, i.e. the learning process.
- The_Amp_Walrus 4y agoAlso 'easy' to model as a lookup table containing all possible solutions.