I like methods of this flavour, pioneered as I understand it by Fabio Petroni. Very elegant, particularly because you can change the distribution over substring

I like methods of this flavour, pioneered as I understand it by Fabio Petroni. Very elegant, particularly because you can change the distribution over substrings in O(params) time instead of O(index size). What's funny about them is that it's a fairly involved procedure that turns your language model into an actual stochastic parrot, both showing that such a model useful and demonstrating that the original parrot concept was rather ill conceived.