This really frustrates me, because it shouldn't be much harder to train a generative transformer models to be aware not just of previous tokens but also program

This really frustrates me, because it shouldn't be much harder to train a generative transformer models to be aware not just of previous tokens but also programming context, that is the current pile of open parens and their associated verb. Lisp is obviously the best place to experiment this idea, but other programming languages are full of braces which need closing too.