13 ms·
DALL-E 2 has a secret language
- ortusdux 4y agoI wonder if any linguists are training a neural network to generate Esperanto 2.0.
- throw457 4y agoI bet it's just a form of copy protection.
- ml_basics 4y agoI find it really interesting how these new large models (DALLE, GPT3, PaLM etc) are opening up new research areas that do not require the same massive resources required to actually train the models. This may act as a counter balance to the trends of the last few years of all major research becoming concentrated in a few tech companies.
- dpierce9 4y agoGavagai!
- alxndr 4y ago(explaining the joke: https://en.m.wikipedia.org/wiki/Indeterminacy_of_translation https://en.m.wikipedia.org/wiki/Indeterminacy_of_translation )
- dang 4y agoLater related thread: No, DALL-E doesn’t have a secret language - https://news.ycombinator.com/item?id=31587316 https://news.ycombinator.com/item?id=31587316 - June 2022 (7 comments)
- kazinator 4y agoThat's reminiscent of small children making up their own words for things. Those words are stable in that you can converse with the child using those words.
- jws 4y agoIn short: DALLE-2 generates apparent gibberish for text in some circumstances, but feeding the gibberish back in gets recognized and you can tease out the meaning of words in this unknown language.
- a_f 4y agoIs it gibberish in the true sense, or is it some sort of AI generated/learned latin text for the input models used? I wonder if they used a large number of biological images in their training data along with their scientific names, which led to this second order effect.
- voxl 4y agoI think calling it gibberish is a misnomer, it would be gibberish if inconsistent, but if the same strings of characters lead to the same semantic objects then that is not gibberish.
- chii 4y agoThis reminds me of the film https://en.wikipedia.org/wiki/The_Machine_(film) https://en.wikipedia.org/wiki/The_Machine_(film) Spoilers: These AI controlled soldiers have developed their own language (communicated wirelessly), and the humans think that they are just mute.
- wongarsu 4y agoLink to the 5 page paper, for those that don't like twitter threads: https://giannisdaras.github.io/publications/Discovering_the_Secret_Language_of_Dalle.pdf https://giannisdaras.github.io/publications/Discovering_the_...
- 726D7266 4y agoPossibly related: In 2017 AI bots formed a derived shorthand that allowed them to communicate faster: https://www.facebook.com/dhruv.batra.dbatra/posts/1943791229195215 https://www.facebook.com/dhruv.batra.dbatra/posts/1943791229... > While the idea of AI agents inventing their own language may sound alarming/unexpected to people outside the field, it is a well-established sub-field of AI, with publications dating back decades. > Simply put, agents in environments attempting to solve a task will often find unintuitive ways to maximize reward.
- gibolt 4y agoUnintuitive to biased humans. The solutions may actually be super intuitive/efficient, and we just can't wrap our heads around it yet
- joshstrange 4y agoWhich, to a lessor extent, isn't too terribly different from humans if you think about. We don't use a full new language but every profession has it's own jargon. Some of it spans the whole industry and some is company-specific.
- MatthiasPortzel 4y agoIt’s wild to see the discoveries being made in ML research. Like most of these ‘discoveries,’ it makes a fair amount of sense after thinking about it. Of course it’s not just going to spit out random noise for random input, it’s been trained to generate realistic looking images. But I think it is an interesting discovery because I don’t think anyone could have predicted this. One of my favorite examples is the classification model that will identify an apple with a sticker on it that says “pear” as a pear—it makes sense, but is still surprising when you first see it.
- astrange 4y ago> One of my favorite examples is the classification model that will identify an apple with a sticker on it that says “pear” as a pear—it makes sense, but is still surprising when you first see it. That classification model (CLIP) is the first stage of this image generator (DALLE) - and actually this shows that it doesn't think they're exactly the same thing, or at least that's not the full story, because DALL-E doesn't confuse the two. However, other CLIP guided image generation models do like to start writing the prompt as text into the image if you push them too hard.
- wongarsu 4y agoWas DALL-E 2 trained on captions from multiple languages? If so, this makes a lot of sense. Somewhere early in the model the words "bird", "vogel", "oiseau" and "pájaro" have to be mapped to the same concept. And "Apoploe vesrreaitais" happens to map to the same concept. Or maybe "Apoploe vesrreaitais" is rather the tokenization of that concept, since it also appears in the output. So in a sense DALL-E is using an internal language to make sense of our world.
- melony 4y agoBut that's expected behavior for a language model (especially VAEs), where's the novelty? In a VAE, the vectors are probabilistic in the latent space so this is basically the NLP version of the classic VAE facial image generation where you can tweak the parameters to emphasize or de-emphasize a feature.
- tomrod 4y agoNovel in engineering together of multiple concepts, if nothing else!
- link0ff 4y agoThis looks like the artificial language Lojban was constructed: its words share parts from completely unrelated languages to the point when none of the original words are recognizable in the result.
- alxndr 4y agoThe original words aren't recognizable at first glance, but they do serve as potential mnemonics for remembering the terms/definitions for any learners who speak one of those source languages (English, Spanish, Mandarin, Arabic, Russian, Hindi)
- godelski 4y agoInterestingly Google detects these words as Greek. I know they are nonsensical and not actually Greek but I'm wondering if any Greek speakers might be able to provide some insights. Are these gibberish words close to meaningful words? (clear shot in the dark here) Maybe a linguist could find more meaning?
- deckeraa 4y agoOne could conjecture that "Apoploe" is similar to από πουλί, "from bird". But I don't have much support for that conjecture.
- PartiallyTyped 4y agoThe word is απόπλους, or αποπλοΐ
- noizejoy 4y agoOr maybe it’s a subtle joke by Google as a play on the idiom “it’s all Greek to me”?
- IncRnd 4y agoOr for something that is only somewhat subtle, it's a chicken and egg problem.
- PartiallyTyped 4y agoAs a native Greek, no, they don't make any sense.. sort of. My hunch is that they read significantly more like Latin than they do Greek. However it tells us something about google translate. The reason "Apoploe vesrreaitais" is detected as Greek is because the first "word" is "phonetically" similar to the word απόπλους, which means sailing/shipping and it is rooted in ancient Greek. If we were to write Αποπλοuς using roman characters, we would write apoplous or apoloi (plural, in Greek is αποπλοΐ). So I think that the model understands that "oe" suffix is used to represent the Greek suffix "οι" that is used for plurals. The rest of the word is rather close phonetically, so there is some model that maps phonetic representations to the correct word. The other phrase seems to be combined of words classified as Portuguese, Spanish, Lithuanian, and Luxembourgish.
- astrange 4y agoIt seems obvious this would happen (it's just adversarial inputs again) - they didn't make DALL-E reject "nonsense" prompts, so it doesn't try to, and indeed there's no reason you'd want to make it do that. Seems like a useful enhancement would be to invert the text and image prior stages, so it'd be able to explain what it thinks your prompt meant along with making images of it.
- deleted 4y ago[deleted]
- schroeding 4y agoInteresting! I wonder if the model would "understand" the made-up names from today's stained glass window post[1] like "Oila Whamm" for William Ockham and output similar images. [1] https://astralcodexten.substack.com/p/a-guide-to-asking-robots-to-design?s=r https://astralcodexten.substack.com/p/a-guide-to-asking-robo...
- jsnell 4y agoOne of the replies is a thread with a fairly convincing rebuttal, with examples: https://twitter.com/Thomas_Woodside/status/1531710251015081984 https://twitter.com/Thomas_Woodside/status/15317102510150819...
- jimhi 4y agoWe don't know the rules or grammar of this "language". Maybe nouns change based on how they are used https://en.wikipedia.org/wiki/Declension https://en.wikipedia.org/wiki/Declension
- lmc 4y agoA rebuttal to the rebuttal (without examples)... How many French people speak Breton?
- dwallin 4y agoI'm not sure it's a convincing rebuttal, the examples shown all seem to have some visible commonality. Eg. "Apoploe vesrreaitais" Could refer to something along the lines of a "fan / wedge" or "wing-like" If you look at the examples of cheese, when compared to the "birds and cheese" the cheese tends to be laid out in a fan like pattern and shaped in sharp angled wedges.
- sudosysgen 4y agoIt seems to refer to "bird plant" which means birds on trees, so it would make sense there would be cheese and plants if it can't find how to fit a bird.
- f38zf5vdt 4y agoI'm curious what it generates when given randomly generated strings of seemingly pronounceable words like "Fedlope Dipeioreitcus".
- joshcryer 4y agoYeah, and his example about bugs in the kitchen. Everything is edible and 'wild' or 'heirloom' and "contarra ccetnxniams luryca tanniounons" comes from the farmers talking about ... vegetables. So there's a definite interrelationship between the 'words' and the images. I'm unconvinced by the rebuttal as well, not to say I am convinced we have a fully formal language going on here, but there's definitely some shared concepts with the generated text. I wonder what imagen would come up with or if it's 'language' is more correlated to real language.
- normaldist 4y agoI'm seeing a lot more people experimenting with DALL-E 2. How does getting access work, do you need a referral?
- minimaxir 4y agoThere is a waitlist, but OpenAI just announced they are opening access more widely from it.
- Cloudef 4y agoI wonder why they call it "Open"AI
- mikequinlan 4y agohttps://labs.openai.com/waitlist https://labs.openai.com/waitlist
- DonHopkins 4y agoHas anyone tried talking to it in Simlish? https://en.wikipedia.org/wiki/Simlish https://en.wikipedia.org/wiki/Simlish https://web.archive.org/web/20040722043906/http://thesims.ea.com/us/getcool/graphics/index.html https://web.archive.org/web/20040722043906/http://thesims.ea... https://web.archive.org/web/20121102012431/http://bbs.thesims2.ea.com/community/bbs/messages.php?&openItemID=item.2,item.43,item.61,item.41,item.23&threadID=8d04f2582c30dca38b0a2d07d28fb420&directoryID=2&startRow=1#5b3c9c18c3808d99f1e04c01fdb828ea#5b3c9c18c3808d99f1e04c01fdb828ea https://web.archive.org/web/20121102012431/http://bbs.thesim...
- pwillia7 4y agosul sul
- Imnimo 4y agoI tried a few of these in one of the available CLIP-guided diffusion notebooks, but wasn't able to get anything that looks like DALL-E meanings. Not sure if DALL-E retrained CLIP (I don't think they did?), but it maybe suggests that whatever weirdness is going on here is on the decoder side? All the cool images that DALL-E spits out are fun to look at, but this sort of thing is an even more interesting experiment in my book. I've been patiently sitting on the waitlist for access, but I can't wait to play around with it.
- notimpotent 4y agoMy first thought upon reading this: what if DALL-E (or a similar AI) uncovers some kind of hidden universal language that is somehow more "optimal" than any existing language? i.e. anything can be completely described in a more succinct manner than any current spoken language. Or maybe some kind of universal language that naturally occurs and any semi-intelligence life can understand it. Fun stuff!
- jcims 4y agoI think something like this is actually quite likely. I’ve been wondering if there is a way to do psychological experiments on these large language models that we couldn’t do with a person.
- julianbuse 4y agoI imagine these would be very interesting, but not very applicable to humans (which I presume is the intended outcome). OTOH, since these language models are trained on human language and media, they might have some value. I'm quite split on which I think is more likely (I don't have any experience in ai/ml nor in psychology so what do I know).
- jcims 4y agoOne example of an ’experiment’ would be to explore the latent space with random/procedurally generated prompts and do semantic analysis on the results to look for topics or sentiments to emerge. My guess is that the current language models don’t have enough information in the training data to do this usefully today, but over time it seems potentially viable.
- sbierwagen 4y agoIthkuil (Ithkuil: Iţkuîl) is an experimental constructed language created by John Quijada.[1] It is designed to express more profound levels of human cognition briefly yet overtly and clearly, particularly about human categorization. Meaningful phrases or sentences can usually be expressed in Ithkuil with fewer linguistic units than natural languages.[2] For example, the two-word Ithkuil sentence "Tram-mļöi hhâsmařpţuktôx" can be translated into English as "On the contrary, I think it may turn out that this rugged mountain range trails off at some point."[2] https://en.wikipedia.org/wiki/Ithkuil https://en.wikipedia.org/wiki/Ithkuil
- carabiner 4y agoScience has gone too far.
- layer8 4y agoSounds like an effect similar to illegal opcodes: https://en.m.wikipedia.org/wiki/Illegal_opcode https://en.m.wikipedia.org/wiki/Illegal_opcode
- TOMDM 4y agoShouldn't this be expected to a certain extent? Gibberish has to map _somewhere_ in the models concept space. Whether is maps onto anything we'd recognise as consistent doesn't mean that the AI wouldn't have some concept of where it relates, as other people have noted, the gibberish breaks down when you move it into another context, but who's to say that Dall-E 2 isn't remaining consistent to some concept it understands that isn't immediately recognisable to us. The interesting part is if you can trick it to spit out gibberish in targeted areas of that concept space using crafted queries.
- momojo 4y ago> Shouldn't this be expected to a certain extent? In hindsight, sure. Given enough time someone might have predicted the phenomenon. But I don't think most of us did. What's more fascinating to me is how often this has happened in this space in just the last few years. 1. Some phenomenon is discovered 2. I'm surprised 3. It makes sense in hindsight
- burrows 4y agoNP
- EvgeniyZh 4y agoYou could expect that gibberish is distributed uniformly in latent space, disconnected from it's langual counterpart -- after all those are textual inputs that model have never seen, and it can't even map words it have seen many times to their writing in image properly: "seafood" word and "seafood" image are in the same place in latent space, but "seafood" word in image isn't. Yet some gibberish word in image is, and also the same gibberish word is. It's very counterintuitive for me.
- TOMDM 4y agoA uniform distribution makes sense for gibberish, not something I'd considered. A counterpoint I'd raise is I wonder how aggressive Dall-E 2 is in making assumptions about words it hasn't seen before. Hard to do given that it's read essentially the entire internet, however someone could make up some latin-esque words that people would be able to guess the meaning of. If the model is as good as people at assuming the meaning of such made up words, it could stand to reason that if it were aggressive enough in this it might be doing the same thing with gibberish and thus ending up with it's own interpretation of the word, which would land it back in a more targeted concept space. I'd love to see someone craft some words that most people could guess the meaning of, and see how Dall-E 2 fairs.
- seydor 4y agodamn. i hope arcaeologists can use that to decipher old scripts
- smusamashah 4y agoA few days ago I was wondering what DALL-E would generate if given gibberish (tried to request which wasn't entertained). This sounds like an answer to that to some extent. I think, there will be multiple words for the same thing. Also, unlike 'bird' the word 'Apoploe vesrreaitais' might actually mean specific kind of bird in specific setting.
- MaxBorsch228 4y agoWhat if give it the same promt but "with subtitles in French" for example?
- la64710 4y agoDoes google translate supports this?
- ricardobeat 4y agoThe paper is just as long as the twitter thread.
- Veedrac 4y agoWow, I am totally going to need to wait for more experimentation before believing any given thing here, but this seems like a big deal. It's one thing if DALL-E 2 was trying to map words in the prompt to their letter sequences and failing because of BPEs; that shows an impressive amount of compositionality but it's still image-model territory. It's another if DALL-E 2 was trying to map the prompt to semantically meaningful content and then failing to finish converting that content to language because it's too small and diffusion is a poor fit for language generation. That makes for worse images but it says terrifying things about how much DALL-E 2 has understood the semantic structure of dialog in images, and how this is likely to change with scale. Normally I'd expect the physical representation to precede semantic understanding, not follow it! That said I reiterate that a degree of skepticism seems warranted at this point.
- trebligdivad 4y agoIs this finally a need for a xenolinguist?
- neopallium 4y agoWould it be possible to build a rosetta stone for this secret language with prompts asking for labeled pictures of different categories of objects? Or prompts about teaching kids different words?
- deleted 4y ago[deleted]
- PoignardAzur 4y agoWait, how does that make any sense? I thought DALL-E's language model was tokenized, so it doesn't understand that eg "car" is made up of the letters 'c', 'a' and 'r'. So how could the generated pictures contain letters that form words that are tokenized into DALL-E's internal "language"? Shouldn't we expect that feeding those words to the model would give the same result as feeding it random invented words? Actually, now that I think about it, how does DALL-E react when given words made of completely random letters?
- YeGoblynQueenne 4y agoIf I understand correctly from the twitter thread (I haven't read the linked technical report) the author and a collaborator found that DALL-E generated some gibberish in an image that showed two men talking, one holding two ... cabbages? They fed (some of) the gibberish back to DALL-E and it generated images of birds, pecking at things. Conclusion: the gibberish is the expression for birds eating things in DALL-E's secret language. But, wait. Why is the same gibberish in the first image, that has the two men and the cabbages(?), but no birds? Explanation: the two men are clearly talking about birds: >> We then feed the words: "Apoploe vesrreaitars" and we get birds. It seems that the farmers are talking about birds, messing with their vegetables! With apologies to my two compatriots, but that is circular thinking to make my head spin. I'm reminded of nothing else as much as the scene in the Knights of the Round Table where the wise Sir Bedivere explains why witches are made of wood: https://youtu.be/zrzMhU_4m-g https://youtu.be/zrzMhU_4m-g
- GamerUncle 4y agohttps://nitter.net/giannis_daras/status/1531693093040230402 https://nitter.net/giannis_daras/status/1531693093040230402
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- tiborsaas 4y agoI love this scientific curiosity towards DALL-E. Many people just say that it's bad at text generation (including me), but someone stopped to wonder if this is really gibberish or it has some logic to it. Classic "hmm, that's odd" case. It will be fun to see people experimenting with extracting text prompts from generated images. I'd try something like "An open children book about animals" or "Random thought written on a paper". Maybe do a feedback loop of extracted prompts :)
- softcactus 4y agoFor some reason this comment from someone else was deleted: "My first reaction to this was, "It probably has to do with tokenization. If there's a 'language' buried in here, its native alphabet is GPT-3 tokens, and the text we see is a concatenation of how it thinks those tokens map to Unicode text." Most randomly concatenated pairs of tokens simply do not occur in any training text, because their translation to Unicode doesn't correspond to any real word. There are also combinations that do correspond to real words ("pres" + "ident" + "ial") but still never occur in training because some other tokenization is preferred to represent the same string ("president" + "ial"). Maybe DALL-E 2 is assigning some sort of isolated (as in, no bound morphemes) meaning to tokens — e.g., combinations of letters that are statistically likely to mean "bird" in some language when more letters are revealed. When a group of such tokens are combined, you get a word that's more "birdlike" than the word "bird" could ever be, because it's composed exclusively of tokens that mean "bird": tokens that, unlike "bird" itself, never describe non-birds (e.g., a Pontiac Firebird). The exact tokens it uses to achieve this aren't directly accessible to us, because all we get is poorly rendered roman text." I wonder if this is why the term for "bird" seemed to be in faux binomial nomenclature, the scientific names for animals. I assume that in the training set there were images of birds/insects with their scientific name. An image labeled with the scientific name would always be an image of an animal, unlike images with the word bird in them which could be of a birdhouse, Pontiac Firebird, or someone playing golf. That would mean that in the latent space when DALLE wants to represent a bird as accurately as possible, it will use the scientific name, or a gibberish/tokenized version of the scientific name-- like someone trying to make up a name that sounds regal might say "Sir Reginard Swellington III". Even though it's not a real name it encodes into the latent space of royal-sounding names. I wonder if this could be extended to other things with very specific naming conventions. For example aircraft names: "Gruoeing B-26 Froovet" might encode into military aircraft latent space.
- nutanc 4y agoI don't think it's a secret language per se. It's just that the tokens generated for these sentences are for some reason coming close to a bird latent space. Maybe if we can dig deep and do a google search for kinds of birds we can find the connection. Tokens from OpenAI below. https://t.co/Of8CBGdGAE https://t.co/Of8CBGdGAE. Found this answer: https://twitter.com/BarneyFlames/status/1531736708903051265?t=q6tpKuE0KmUArNwuaXr6kw&s=19 https://twitter.com/BarneyFlames/status/1531736708903051265?...
- axg11 4y agoIndeed, this is an expected outcome from the way that DALL-E is trained. It’s still an interesting finding and sometimes affirming the expected can lay the groundwork for future interesting discoveries. This is an example of an application where uncertainty modelling would help greatly. Any and every input will lead to an output. That doesn’t mean that all regions of latent/embedding space are equally valid. I’m in the camp that large/modern ML models are nearing human intelligence, in some aspects. What’s currently missing is the universal ability to estimate uncertainty and identify inputs that are out of distribution. Many groups are working on this and perhaps we already have the solution but are not combining the right uncertainty estimation approach with the right foundational model.
- adastra22 4y ago> I don't think it's a secret language per se. It's just that the tokens generated for these sentences are for some reason coming close to a bird latent space. Serious question: what else do you think language is? How else would your brain associate the word "bird" with the concept?
- nutanc 4y agoI would associate a bird with a bird. I wouldn't associate birmongle with bird just because both start with the token "bir". What Dalle is doing here is what it has been trained to do. Just find the closest token match and try to draw something. It does not understand something. And that's absolutely fine. I am just saying it does not have a secret language or something. It's just regular language and it's closest matches.
- bla3 4y agoAnother convincing rebuttal: https://mobile.twitter.com/benjamin_hilton/status/1531780892972175361 https://mobile.twitter.com/benjamin_hilton/status/1531780892... It'd be cool if this was true, but it looks like it mostly isn't.
- afro88 4y agoI love the weird edge cases of ML. Imagine discussing security concerns and saying "what if it creates it's own secret language that we don't know about, which is discovered later, and people can use to circumvent privacy and obscenity controls?"
- teddykoker 4y agoAccording to [1], the byte pair encoding for “Apoploe vesrreaitais” (the words producing bird images) is "apo, plo, e</w>, ,ve, sr, re, ait, ais</w>", and Apo-didae & Plo-ceidae are families of birds. [1] https://twitter.com/barneyflames/status/1531736708903051265?s=21&t=cynRdfVRr4tlsqG2Vz9XqQ https://twitter.com/barneyflames/status/1531736708903051265?...
- deleted 4y ago[deleted]
- DalasNoin 4y agoOn the other hand the openai tokenizer gives me a different tokenization ap - opl - oe [0]. If you capitalize A the result is A - pop - loe. The dalle 2 paper only specifies that it uses a BPE encoding, I would assume they used the same one as for gpt3 [0] https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer
- karmasimida 4y agoIf they use BPE dropout, then the split can be different and not unique. And for the record, they use BPE dropout for DALLE-1, see https://arxiv.org/pdf/2102.12092.pdf https://arxiv.org/pdf/2102.12092.pdf
- DalasNoin 4y agoI believe they only apply it during training.
- karmasimida 4y agoright, that is my point. It is hard to know which combination triggers the current tokenization to be interpreted as bird.
- qgin 4y agohttps://twitter.com/giannis_daras/status/1531693104821985280 https://twitter.com/giannis_daras/status/1531693104821985280 This one melts my brain a bit, I’m not going to lie. Whales talking about food, with subtitles. “Translate” the subtitles and you get food that whales would actually eat.
- mola 4y agoSo now we're reverting to haruspex... The deemphasizing of peer review BEFORE publication will kill science. The amount of noise and nonsense proliferating just causes confusion and lost of trust...