At least one mechanism to update transformers' internal states already exists, there is nothing stopping anyone from performing backprop after every session. I

At least one mechanism to update transformers' internal states already exists, there is nothing stopping anyone from performing backprop after every session. It just has big technical and economic challenges. But I expect advances there. There have actually already been big advances, though done in bulk fashion (RLHF).