Thanks for the great questions! I've been responding to this thread for the last few hours and I'm about to need to run, so I hope you'll forgive me redirecting

Thanks for the great questions! I've been responding to this thread for the last few hours and I'm about to need to run, so I hope you'll forgive me redirecting you to some of the other answers I've given. On whether the model is looking ahead, please see this comment which discusses the fact that there's both behavioral evidence, and also (more crucially) direct mechanistic evidence -- we can literally make an attribution graph and see an astronomer feature trigger "an"! https://news.ycombinator.com/item?id=43497010 https://news.ycombinator.com/item?id=43497010 And also this comment, also on the mechanism underlying the model saying "an": https://news.ycombinator.com/item?id=43499671 https://news.ycombinator.com/item?id=43499671 On the question of whether this constitutes planning, please see this other question, which links it to the more sophisticated "poetry planning" example from our paper: https://news.ycombinator.com/item?id=43497760 https://news.ycombinator.com/item?id=43497760