Actually, I think you could save yourself some trouble and use scikit-learn's built-in text preprocessing utils: Word counter: http://scikit-learn.org/stable/m

Actually, I think you could save yourself some trouble and use scikit-learn's built-in text preprocessing utils: Word counter: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html http://scikit-learn.org/stable/modules/generated/sklearn.fea... Hashing vectorizer if you want to trade off explainability for speed and scalability: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html http://scikit-learn.org/stable/modules/generated/sklearn.fea... TF-IDF weighing: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html http://scikit-learn.org/stable/modules/generated/sklearn.fea... Also, if you transform bag-of-words vectors into a dense form, you're gonna have a bad time (insert appropriate meme picture here). In large corpora, dimensionality grows quite substantially - if you work with news corpora or Wikipedia, you're in the 100k-1M dimensional space pretty quickly. Great to see an approachable explanation for NLP. As they say sometimes, when you know how it's done, it stops being "Artificial Intelligence".