6 ms·

Pretty sure it's one of those "All squares are rectangles but not all rectangles are squares" situations. Transformers are what really started the LLM Boom, and

Pretty sure it's one of those "All squares are rectangles but not all rectangles are squares" situations. Transformers are what really started the LLM Boom, and seem to be crucial to the technology. They also have other applications, such as what OP created

porridgeraisin 15d ago
In my unpopular opinion, it wasn't transformers or attention, but pretraining on language data that kicked off the LLM Boom. Alec radford in his little jupyter notebook trained a very small non-transformer to predict simply the next-character on Amazon reviews. He noticed emergence of a neuron which when toggled controlled the sentiment of the remaining text. This is published as the sentiment neuron paper. Till then, transformers were being used primarily for stuff like translation and such and no one was even pretraining at scale, even tho transformers and attention existed. Openai and google if you count T5 persisting with pretrained generative models was what led to the LLM boom. Yes they used transformers, but that's just one IMO minor aspect.
- razodactyl 7d ago
  Wasn't it an LSTM neuron that changed sign and magnitude over a stretch of text? I remember seeing color coding against the current sentiment which was neat.
  - porridgeraisin 6d ago
    Yep. a multiplicative LSTM to be exact.
calebkaiser 15d ago
It gets extremely blurry, because people commonly refer to any model that uses a component associated with the Transformer architecture as a Transformer (i.e. using some kind of QKV-esque attention mechanism). I think it's easier to think of it like this: A large language model is just what it says--a very large statistical model trained for language tasks. This covers the spectrum of GPT-style models, but also those hard to classify ones, like Liquid's "Liquid Foundation Models", which can get up to 24 billion parameters and use grouped query attention, but are closely related to state-space models as well: https://huggingface.co/LiquidAI/LFM2-24B-A2B https://huggingface.co/LiquidAI/LFM2-24B-A2B Also, as others have pointed out, a Transformer isn't inherently a language model. So really they're sort of two different axes, one classifying the model size and task, the other referring to a specific architecture.

porridgeraisin 15d ago

In my unpopular opinion, it wasn't transformers or attention, but pretraining on language data that kicked off the LLM Boom. Alec radford in his little jupyter notebook trained a very small non-transformer to predict simply the next-character on Amazon reviews. He noticed emergence of a neuron which when toggled controlled the sentiment of the remaining text. This is published as the sentiment neuron paper. Till then, transformers were being used primarily for stuff like translation and such and no one was even pretraining at scale, even tho transformers and attention existed. Openai and google if you count T5 persisting with pretrained generative models was what led to the LLM boom. Yes they used transformers, but that's just one IMO minor aspect.

razodactyl 7d ago
Wasn't it an LSTM neuron that changed sign and magnitude over a stretch of text? I remember seeing color coding against the current sentiment which was neat.
- porridgeraisin 6d ago
  Yep. a multiplicative LSTM to be exact.

calebkaiser 15d ago

It gets extremely blurry, because people commonly refer to any model that uses a component associated with the Transformer architecture as a Transformer (i.e. using some kind of QKV-esque attention mechanism). I think it's easier to think of it like this: A large language model is just what it says--a very large statistical model trained for language tasks. This covers the spectrum of GPT-style models, but also those hard to classify ones, like Liquid's "Liquid Foundation Models", which can get up to 24 billion parameters and use grouped query attention, but are closely related to state-space models as well: https://huggingface.co/LiquidAI/LFM2-24B-A2B https://huggingface.co/LiquidAI/LFM2-24B-A2B Also, as others have pointed out, a Transformer isn't inherently a language model. So really they're sort of two different axes, one classifying the model size and task, the other referring to a specific architecture.