8 ms·
Stochastic Parrot
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- rsynnott 3y ago> They go on to note that because of these limitations, a learning machine might produce results which are "dangerously wrong" I was initially thinking "well, yes, Nobel Prize for Stating the Obvious there", but looks like the paper was written in the far distant past of 2021, when LLMs were largely still in their babbling obvious nonsense stage, rather than the current state of the art, where they babble dangerously convincing nonsense, so, well, fair enough I suppose. Amazing how fast progress has been there, though it's progress in an arguably rather worrying direction, of course.
- sharikous 3y agoNot to reduce the value of the insight, but since she coauthored the paper with Google employees she probably had access to models more advanced than those which were available to the general public
- daniel_reetz 3y agoThanks for pointing this out. I've spent years in R&D and awareness always lags technology.
- rsynnott 3y agoI do wonder what the state of Google's stuff in 2021 was. Here's something produced by the 2020 version of GPT-3: https://www.aiweirdness.com/roses-are-red/ https://www.aiweirdness.com/roses-are-red/ At that point, OpenAI was still fairly clearly at the babbling obvious nonsense phase; I would wonder was Google's stuff much better. I also wonder if the original authors would have been surprised to learn that, by 2023, lawyers would be citing fake precedent made up by a machine. The progression to "dangerous nonsense" really does seem to have been worryingly fast.
- dekhn 3y agoI was really impressed with the work that Noam Shazeer was doing at Google before he left (I worked on TPUs and frequently had to debug problems at scale for researchers). It was clear he was making some pretty impressive improvements, but the results weren't super obvious even to most people inside google, and they didn't translate to externally visible projects. This isn't that dissimilar to working at any sufficiently advanced R&D outfit, which strongly demonstrates the principle "the future is already here but isn't evenly distributed".
- api 3y agoI’d argue that all these models are stochastic parrots because they’re not embodied in any way. There is no way they can actually understand what they are talking about in any way that is tied back to the physical world. What these LLMs and diffusion models and such actually are is a lossy compression method that permits structural queries. The fact that they can learn structure as well as content allows them to reason as well, but only to the extent that the rules they’re following existed somewhere in the training data and its structure. If one were given access to senses and memory and feedback mechanisms and learned language that way, it might be considered actually intelligent or even sentient if it exhibited autonomy and value judgments.
- jstx1 3y ago> if it exhibited autonomy and value judgments. Who wants this from ML systems? I want them to be useful, not to have autonomy and value judgments.
- kelseyfrog 3y agoI have a few projects in mind where that's a requirement.
- grumbel 3y agoIt has to have some degree of autonomy to be useful. The current approach with ChatGPT to just have all the knowledge in the world directly in the base model not only doesn't scale, it would also run into issues with copyright if it could actually recite books and stuff word for word. A ChatGPT that can just use Google to look up the necessary information itself would be far more useful. BingChat sort of tries that, but it doesn't really have any autonomy either, so it just summarizes the first Bing search result it gets. It would be far more useful if it could search around two or three layers depth into the search results to actually find what you are looking for. In general current AI systems have the problem that you have to babysit them far to much. If you want to get specific answers, it's you that has to provide all the necessary context to make it happen, the AI can't figure out by itself what you want from past conversations.
- 3y ago
- mach1ne 3y ago>"stochastic parrot" is a term coined by Emily M. Bender in the 2021 artificial intelligence research paper "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" This might be the first time the term was seen in an ’official’ context, but is it really the origin? It feels like the term has been hovering around for longer, and even Google Trends shows significant search trends way before 2021
- naillo 3y agoI feel this way too. Maybe there's some very similar term that we're both thinking of though that is just at the tip of the tongue because I can't find what it'd be
- krapp 3y agoMost people have never actually dealt with something like modern LLMs, so we haven't really developed the proper language to describe them and how they behave. It's either too simplistic and reductive (stochastic parrot, xerox machine) or presupposes sentience and intent ("fabricates","hallucinates", etc.)
- visarga 3y agoAlso "a blurry jpeg of the internet". LOL, all we need is "a series of tubes, not a truck" and we're set. I think we are focusing on the model too much and miss the real hero - language. The corpus of text these models are trained on is a marvel of human creativity. This cultural artefact is the diff between primitive and modern humans. And it is the diff between a random initialisation and a trained GPT-4. Maybe the brain or the model don't matter, but what you train them on. Even more, language is special. Ideas are self replicators, they have a lifecycle, they have evolutionary pressure to improve. Ideas travel a lot. No single human can recreate this knowledge, it is the result of massive search. I'd say more than 99% of human intelligence is based on applying ideas invented by someone else. So let's be more lenient on the parroting accusations. AIs can be smart if they get feedback, like AlphaZero, but without feedback they of course have to parrot.
- 3y ago
- Invictus0 3y agoFeels like this wikipedia page is overly (self-?)promotional of the paper and its authors
- ttpphd 3y agoConsidering that men are taking credit for their work, maybe some over-correction is understandable.
- constantcrying 3y agoOne massive flaw of the Wikipedia modell is that the people who edit Wikipedia the most "aggressively" are the ones with the most emotional investment in the topic. This can lead to very detailed articles written by very enthusiastic people. In other cases the people who are very pro/against the subject will be the ones who put in the most effort, especially on smaller/controversial subjects. I have seen Wikipedia pages which basically read like ads for small companies.
- isp 3y agoTopical tweet from 2018: > Optimist: AI has achieved human-level performance! > Realist: “AI” is a collection of brittle hacks that, under very specific circumstances, mimic the surface appearance of intelligence. > Pessimist: AI has achieved human-level performance. https://twitter.com/dmimno/status/949302857651671040 https://twitter.com/dmimno/status/949302857651671040
- seydor 3y agoLLMs are not stochastic though, they are deterministic and dont even require random numbers, right? The term in general seems to be unfortunate because the models seem to do more than parroting. LLMs are more like central pattern generators of the nervous systems, able to flexibly create well coordinated patterns when guided appropriately
- dudebro314 3y agoSimulations of Brownian motion are not stochastic though, they are deterministic if you fix their random seed, right?
- seydor 3y agoStochasticty is mandatory for modeling brownian motion. Actually transformers do not require ramndomness at all, so not at all
- FabHK 3y agoMy understanding is the opposite. The entire process results in a "score" over all output tokens, which is then converted into a probability of being picked, using a softmax that takes a temperature as a parameter. With a temperature of zero, the "best" token is always picked, but interestingly enough, that does not give optimal results. So sometimes you want the second or even third best. Thus, a "good" (GPT-like) LLM is intrinsically random. To put it differently: You can make them deterministic by using a temperature of zero (then the output would be pretty bad and repetitive), or having a "better" temperature and fixing a random seed (then the output would be better, but it would only be deterministic in the same sense as a simulation of Brownian motion with fixed random seed). https://ai.stackexchange.com/questions/32477/what-is-the-temperature-in-the-gpt-models https://ai.stackexchange.com/questions/32477/what-is-the-tem... Section 3.3 in https://www.lesswrong.com/posts/pHPmMGEMYefk9jLeh/llm-basics-embedding-spaces-transformer-token-vectors-are https://www.lesswrong.com/posts/pHPmMGEMYefk9jLeh/llm-basics... https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
- Workaccount2 3y agoI worry that the "stochastic parrot" was premature, an idea sown early in development that will now carry along through any advances made. Basically there is this innate idea that if the basic building blocks are simple systems with deterministic behavior, then the greater system can never be more than that. I've seen this is spades within the AI community, "It's just matrix multiplication! It's not capable of thinking or feeling!" Which to me always felt more like a hopeful statement rather than a factual one. These guys have no idea what consciousness is (nobody does) nor have any reference point for what exactly is "thinking" or "feeling". They can't prove I'm not a stochastic parrot anymore than they can prove whatever cutting edge LLM isn't. So while yes, present LLMs likely are just stochastic parrots, the same technology scaled might bring us a model that actually is "something that is something to be like", and we'll have everyone treating it with reckless carelessness because "its just a stochastic parrot".
- cjalmeida 3y agoIndeed. It's like saying life is just long strands of DNA composed of only four simple molecules. Or economy is just people trading their surpluses. Emergent behavior... emerges. It's hard to predict or explain from constituents. Scale changes everything.
- jschveibinz 3y ago[flagged]
- ac2u 3y agoAgreed. I think the stochastic parrot concept is useful to ground our expectations on LLMs for now, but it could outlive it's usefulness if there ends up being multiple jumps in sophistication similar to that of GPT-2 to 4 in the next 10 years. If that happens, then stochastic parrot as an argument as to why a machine isn't thinking can be made pretty useless if One chooses to drag the argument further into philosophy.
- Ireallyapart 3y ago
- cubefox 3y agoGPT-3 was released less than a year before that, even though this now seems to be long ago. Time is moving fast with AI.
- ChatGTP 3y agoClimate change moves fast too, what’s your point ?
- hombre_fatal 3y ago[flagged]
- dehrmann 3y agoSomething good that came out of crypto was a lot of people thought about what money actually is. LLMs are doing the same with intelligence.
- pydry 3y agoIt wasn't particularly deep thinking though. The same is also true here.
- brandly 3y agoRegardless it's a good thing! Many people have had no reason until recently to break out of thinking that money=usd or intelligence=humans.
- deeviant 3y agoEh, the thing I feel most people (who lost a lot of money on crypto) learned about what money is, is that crypto is not money.
- IshKebab 3y agoYeah but they don't seem to be thinking about it very much. People keep spouting "stochastic parrot" nonsense!
- renewiltord 3y agoIn the end, it turned out the actual innovation was doing the opposite of what this paper recommended: scaling up the LLM, improving quality by throwing lots of data at it rather than curating, and limiting bias by RLHF rather than picking the right datasets. The organizations that listened to these people for even some amount of time got hosed in this situation. Google managed to oust this flock from within but not before their AIs were so lobotomized that they are wildly renowned for being the village idiot. Ultimately, this paper is a triumph of branding over science. Read it if you'd like. But if you let these kinds of people into your organization, they'll cripple it. It costs a lot to get them out. Instead, simply never let them in.
- cratermoon 3y ago> limiting bias by RLHF rather than picking the right datasets This is the same as curation and picking out the dataset, except as post-processing. The reason why RLHF has to happen (and traumatize the people <https://www.bigtechnology.com/p/he-helped-train-chatgpt-it-traumatized https://www.bigtechnology.com/p/he-helped-train-chatgpt-it-t...>) is to address the problems by censoring the model.
- Sunhold 3y agoThe word "trauma" is getting overused. The idea of someone being traumatized by reading fictional text is just silly. It's unpleasant or gross at worst unless you already have other issues.
- torginus 3y agoIs it though? If you wanted to teach humans so that they don't develop unfortunate beliefs, would it be a good approach to just keep them from reading material that you find objectionable? If you read a book that you disagree with, or one that contains falsehoods and bad reasoning as far as you can tell, would that make you believe those things?
- cratermoon 3y agoA reminder that LLM transformers aren't humans, they don't learn the way humans learn.
- dekhn 3y agoThe real question to me is: in the next decade, as ML researchers roll out progressively more sophisticated systems, we can expect that generative systems- which may actually be "only stochastic parrots"- are going to create works that would fool any reasonable human being. At what point does a stochastic parrot fake it till it makes it? Does it even matter? We can imagine that, within 10 years, we'll have a fully synthetic virtual human simulator- a generative AI combined with knowledge base, language parsing, audio and video recognition, basically a talking head that could join your next technical meeting and look like full contributor. If that happens, will the Timnits and the Benders of the world admit that, perhaps, systems which are indistinguishable from a human may not just be parrots, or perhaps, we are just sufficiently advanced parrotS? Seen from that perspective, the promoters of stochastic parrots would seem to be luddites and close-minded, as well as discouraging legitimate, important, and valuable scientific research.
- NoGravitas 3y agoOnce you have a knowledge base connected to the language model, it's no longer a Stochastic Parrot, but something else entirely. The point of the paper is that simply continuing to scale up LLMs will not produce understanding, because a pure LLM has no connection between form and meaning. That link can provided in other ways, though (multimodal models, robot embodiment).
- dekhn 3y agoBut these language models are implicitly trained on knowledge by being fed large amounts of factual text, which (I presume) allows it to generate text that is factual (statistically more frequently than hallucinating nonfactual information). So probably recent models (which were being trained around the time the parrots paper came out) are really implict knowledge models already. Obviously they don't have embodiment, and it's still unclear to me what level of true embodiment in the actual, real, physical world is required to make these models more than just "parrots".
- samgilb 3y agoFun fact: philosopher Regina Rini referred to GPT-3 as a "statistical parrot" six months before the Bender et al paper came out: https://dailynous.com/2020/07/30/philosophers-gpt-3/#rini https://dailynous.com/2020/07/30/philosophers-gpt-3/#rini
- rchaud 3y agoI've got another word for it: recipe-fication. Everything we revile about online recipe websites that spend 1000 words about the history of cooking before getting to the point, will be part and parcel of AI-written anything. It won't be properly proofread or edited by a human, because that would defeat the purpose.
- constantcrying 3y agoThis also relates to vision models. The existence of adversarial attacks (e.g. imperceptable changes in the image drastically changing the output) essentially demonstrate that the model has not reached the point at which the network "understands" the generalized concept it wants to disinguish.
- zone411 3y agoThe same argument could apply to humans. For example https://en.wikipedia.org/wiki/Change_blindness https://en.wikipedia.org/wiki/Change_blindness.
- constantcrying 3y agoNot really an example, there are many ways human vision is flawed and can be tricked, but none are on the level of these adversarial examples. There imperceptible differences between an image lead to a category error. Human perception can be ambigous, but minimal changes never cause drastic category errors.
- cubefox 3y agoThat's something else. The OP was talking about small changes in pictures causing a very different classification.
- adamsmith143 3y agoYoshua Bengio, Andrew Ng, Anrej Karpathy, and many other of the top researchers in the field do not believe these models are stochastic parrots, they believe they have internal world models and prompts are methods to probe those world models. Stochastic parrots is one of the dumbest takes in AI/ML.
- cubefox 3y agoYeah. See e.g. https://arxiv.org/abs/2306.03341 https://arxiv.org/abs/2306.03341 > Our findings suggest that LLMs may have an internal representation of the likelihood of something being true, even as they produce falsehoods on the surface. The problem here is that there is currently no reliable way to extract information from this hypothetical world model. Language models do not always say what they "believe", they might instead say what is politically correct, what sounds good etc. Researchers try to optimize (fine-tune) language models to be helpful, honest, and harmless, but honesty ("truthfulness") can't be easily optimized for.
- Ireallyapart 3y ago[dead]
- the8472 3y agoThe first step to defeating a tiger is to realize that it cannot hurt you, for it is only made of simple atoms.
- cubefox 3y ago"Machine learning? It's just statistics bro."
- koalala 3y agoFrom the article: A "stochastic parrot", according to Bender, is an entity "for haphazardly stitching together sequences of linguistic forms … according to probabilistic information about how they combine, but without any reference to meaning." It seems to me that the great success transformers are now enjoying is precisely due to the fact that 'probabilistic information about how they combine' _is_ meaning.
- NoGravitas 3y agoIt's really not. Read the National Library of Thailand thought experiment to understand the difference. But this isn't saying that AGI is impossible, only that it can't come purely from LLMs, and that pure LLMs will remain stochastic parrots no matter how they are scaled up.
- IshKebab 3y agoI agree. There's a quote in that paper about how ML models can never access meaning (semantics of words) because they only see the form (syntax and letters) and the two are somehow completely divorced. It's obvious nonsense. I can describe a new concept to you using only words and letters and you can understand it. Therefore you can build up knowledge using only syntax. Nobody is saying that LLMs understand the layout of a bus or the feel of leather, but they understand that buses are vehicles with four wheels that transport people etc. Face-slappingly poor philosophy.
- nologic01 3y agoRehashed language imitating sequences is a term that does not denigrate parrots.
- hackandthink 3y agoA nice paper: "Meaning without reference in large language models" "we argue that LLM likely capture important aspects of meaning, and moreover work in a way that approximates a compelling account of human cognition in which meaning arises from con- ceptual role" https://arxiv.org/pdf/2208.02957.pdf https://arxiv.org/pdf/2208.02957.pdf I remember Quine's meaning holism it seems to be related. https://en.wikipedia.org/wiki/Semantic_holism https://en.wikipedia.org/wiki/Semantic_holism
- aaroninsf 3y agoTL;DR: the focus on the implementation details, and descriptions like this, are detrimental, even perilous, because such accounts are both accurate, and deeply misleading. This is description, but it is neither predictive, nor explanatory. It implies a false model, rather than providing one. Evergreen: Ximm's Law: every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon. Lemma: any statement about AI which uses the word "never" to preclude some feature from future realization is false.
- RHSman2 3y agoWhat do you think parrots think about this? Insulted.
- browningstreet 3y ago“stochastic” is to the tech forum as “sapiosexual” is to the online dating profile