14 ms·
Highway connections are actually used heavily in NLP in combination with LSTMs.
by jimfleming 7y ago
Highway connections are actually used heavily in NLP in combination with LSTMs.
- slashcom 7y agoAnd replaced by residual connections in transformers, which are absolutely dominating LSTMs now.
- stochastic_monk 7y agoTransformer-XL uses recurrence, and most NLP SOTA is still with LSTMs. I’m not sure I’d expect attention mechanisms to fully replace recurrence.