5 ms·
Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.
by dfdydx 16d ago
Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.
- m4nu3l 12d agoAlso, not all language models are transformers. You can have language models based on diffusion models or state-space models, or any other model that can be used to model sequences (so all of them, as sequences are just functions). Transformers are just the ones that are most common and successful today.
- perfmode 16d agoIn this case, what are the tokens?
- evilmathkid 16d ago9 color tokens + 4 special tokens (start, end, newline, inp_out_sep)
- shsshs 16d ago[flagged]
- Retric 16d agoWords and symbols alone don’t make a language. Zipper and and? and coin is pimple, Ocean! run bumper.
- shsshs 16d ago[flagged]
- fc417fc802 16d agoYour example is a meaningless sequence. So consider a different scenario where the sequence is meaningful but does not map to any human language. What exactly disqualifies it as a non-human language? When you encounter a human language that you can't personally read presumably you don't proceed to claim that it doesn't constitute language on the basis of your own lack of ability. To come at it from a slightly different angle - does compiled binary code count as a language? If not, why not? (I'll suggest that it's a language albeit not a natural one.)
- Retric 15d agoAny sequence could be meaningful or meaningless depending on the grammar involved. Open a word document in photoshop and the program has no idea what it’s dealing with. The reason the language term in LLM is meaningful is how the training, symbol mapping, etc is designed around human languages. The model doesn’t process raw text, instead there’s a critical processing step which allows the magic to happen.
- fc417fc802 15d agoIs the same processing step not happening here? To my mind the argument against this model qualifying as a language model is that while the sequence of tokens may technically qualify as a sort of language it doesn't appear to be generalized by any reasonable interpretation. Further, the model doesn't appear to be able to handle unstructured inputs and outputs in the "language" - everything seems to be highly structured. My line of reasoning could be approximately summarized as compiled binaries constituting a "real" (though not natural) language versus a sequential listing of chess positions that represent sequential game actions only being language "shaped". Still, it's interesting to consider that if scaled up I expect the "repeat yourself" experiment would likely apply to the internal representation of the model in the same manner.
- Retric 15d agoThe distinction is meaningful because the process described here isn’t bound by the same constraints, resulting in meaningful consequences. Suppose we flipped the initial stings and fed that into the process. There would still be meaning to extract from the training set but our new Reversed English but it is not English so the preprocessing step can’t be based on that assumption.
- fc417fc802 15d agoI don't understand what point you're trying to make here. At the risk of being tangential (due to not understanding) LLMs have famously been able to accept base64 encoded natural language as input "out of the box" because (it seems) under the hood they already transparently translate all inputs into their own internal "language" system on the fly. Presumably a similar translation could be learned for input of reverse english (or rather merely adjusting attention) although at a glance it seems like output might not be feasible.
- Forgeties79 16d agoThis is incredibly pedantic if you ask me.
- deleted 16d ago[deleted]
- kgwgk 16d ago> The distinction is meaningless IMO. Not when discussing whether it's an LLM. The second L in LLM does have a meaning.
- shsshs 16d ago[flagged]
- kgwgk 16d agoYou're the first! Congratulations.
- fc417fc802 16d agoI'd argue that meaningful sequences of symbols constitute a language. This example doesn't use a human language but it does use a language IMO (at least AFAIU).
- dfdydx 15d agoAre number sequences a language?
- fc417fc802 15d agoAre rectangles squares? I can express a passage written in english as a sequence of numbers (ie tokens).
- mkehrt 16d agoSure, and those aren't LLMs?
- deleted 16d ago[deleted]
- literalAardvark 15d agoLLMs also don't use words
- dfdydx 15d agoYes, but they're models of _language_ - words in, words out. The fact that they're embedded to vectors does not change this.