4 ms·
Can someone help me understand Figure 2? Why does the newest token appear at the beginning of the sequence rather than next to its neighboring token?
by mhartz 3y ago
Can someone help me understand Figure 2? Why does the newest token appear at the beginning of the sequence rather than next to its neighboring token?
- nivekkevin 3y agoit's a rolling buffer, so it just upsert index % 4 in this case
- mhartz 3y agoThanks, so does that mean position within the buffer is irrelevant?
- nivekkevin 3y agoit does feel like so, the position eventually loses its meaning as more and more data gets crunched by the training process, eventually it's just a context of the past 4 tokens it feels like