8 ms·
I would not consider word embeddings to be state of the art anymore. Word Embeddings are like TF-IDF when word embeddings came out. Have a look at BERT model t
by arnocaj 8y ago
I would not consider word embeddings to be state of the art anymore.
Word Embeddings are like TF-IDF when word embeddings came out. Have a look at BERT model that just recently got published and is outperforming all kind if NLP tasks with one main Architecture.
I would consider BERT language model two levels higher than word embeddings, as it considers full context sensitive embeddings, dependent on the text left and right of the word in parallel.
- mlucy 8y agoI would second this; sentence embeddings outperform word embeddings on basically all tasks where you actually have sentences to work with. The only downside is that they're significantly more computationally intensive, especially for Transformer models like BERT. (Note: I'm fairly biased, since I work on https://www.basilica.ai https://www.basilica.ai, which among other things makes sentence embeddings available over a REST interface.)
- yazr 8y agoIs BERT computationally (and sample-wise) equivalent to previous SOTA? (I do DRL but not NLP) I sometimes read these DL papers and the requirements are not really feasible if you have to re-implement them in a modified domain.
- irodov_rg 8y agoBERT is more computationally expensive. It might end up giving better results on the task mentioned in the paper but we don't know. At the time of writing this all of the contextual word embedding techniques were fairly new and were not tried.
- deleted 8y ago[deleted]
- DoctorOetker 8y agoI can see utility in demonstrating breakthroughs with the simpler compatible technique as opposed to a more complicated state of the art technique. The goal of scientific communication is to communicate with the simplest examples possible the rationale and effect of a proposal. Then anyone using more advanced frameworks can understand and consider implementing it in theirs.