5 ms·
I was always curious about how Tay worked technically, since it was build before the Transformers era. Was it based on a specific scientific paper or research?
by InfiniteLoup 6mo ago
I was always curious about how Tay worked technically, since it was build before the Transformers era.
Was it based on a specific scientific paper or research?
The controversy surrounding it seemed to have polluted any search for a technical breakdown or a discussion, or the insights gained from it.
- Kye 6mo agohttps://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/ https://blogs.microsoft.com/blog/2016/03/25/learning-tays-in... https://arxiv.org/abs/1812.08989 https://arxiv.org/abs/1812.08989
- mapmeld 6mo agoPeople have tried to suss this out on the ML subreddit, and it is confusing. Most of the worst messages from Tay were just people discovering a "repeat after me: __" function, so it's hard just to figure out which Tay messages to consider as responses of the model. There seems to have been interest in a model which would pick up language and style of its conversations (not actually learning information or looking up facts). If you haven't trained an LSTM model before - you could train on Shakespeare's plays and get out ye olde English in a screenplay format, but from line to line there was no consistency in plot, characters, entrances and exits, etc. in a way which you'd expect after GPT-2. Twitter would be good for keeping a short-form conversation. So I believe Tay and the Watson that appeared on Jeopardy are more from this 'classical NLP' thinking and not proto-LLMs, if that makes sense.