when an LLM is trained backpropagation reaches into the entirety of the key value table. or rather, all the layers (not just the tokens) which come before the c

when an LLM is trained backpropagation reaches into the entirety of the key value table. or rather, all the layers (not just the tokens) which come before the current tick. langauge is only produced by the final layer every tick. and every layer can only interact with other layers on the same level. the mechanics of LLMs and the restriction in how we can train them makes it appear as though all we are doing is forcing language onto them but once RL gets involved all bets are off regarding what's happening inside them (it's quite possible that a static corpus alone is sufficient for all the bets being off).