6 ms·

ChatGPT as a Calculator for Words

201 points by stevefink 3y ago

mif 3y ago
I think the editorial capabilities of ChatGPT are fantastic, and the author provides a good list of examples. On top of that I would add that ChatGPT is really good as composing text. The meaning making is therefore still what we have to do.
ftxbro 3y ago
As a long time LLM enjoyer I just want to mention https://generative.ink/posts/simulators/ https://generative.ink/posts/simulators/ as I think it's by far the most insightful take on the GPT LLMs even though it was from before ChatGPT. It's better than blurry jpeg and stochastic parrot etc.
- skybrian 3y ago
  I think that article is misleading, because a simulator has rules. An LLM is better thought of as a storyteller, because at best it's going to follow whatever implicit rules there are very loosely and let you make rule changes of your own, more like Calvinball. Also, whatever loose rules it has are more literary than mathematical. Plot twists often work.
  - tbalsam 3y ago
    I find this to be a better explanation than "it's just regurgitating strings of text" No, it is clearly not, and that is a very easily testable hypothesis. Thank you for sharing.
    - Al0neStar 3y ago
      testable how?
      - tbalsam 3y ago
        out of distribution tests. if the concept holds consistently over a long period, the concept is the stable thing. If not, then it's only memorizing string densities. For a number of years we've been basically showing the first to be the case, especially as the model is scaled and the context increases, differentially against the second. String density probabilities can be surprisingly brittle, to be honest. The curse of dimensionality applies to them too, believe it or not, which I believe is why topic discussion, reasoning, and integration over longer distances of text is that differential test that shows pretty clearly that substring memorization/text density stuff is not 'just' what the model is learning. Because mathematically/statistically/from an information density perspective/etc etc otherwise it would be basically impossible, I think. That's my best understanding, at least.
  - ftxbro 3y ago
    The essay is long and complicated so I'm not sure how much of it you read closely, but it specifically addresses this distinction between the simulator and the simulacrum. In the analogy of the essay, your argument would be like saying that reality cannot be simply the application of quantum physics, because you are allowed to make new rules like Calvinball within reality which are different from the rules of quantum physics.
    - skybrian 3y ago
      I do understand the difference between a simulator and what's being simulated. I still think they got it all wrong, that the simulator is better called a "writer," the simulated world is better called a "story," and the agent is a better called a "fictional character." We know there's no deeper level to the simulation/game because we have the entire "game history" (the chat history) and we understand it in approximately same way that the LLM does. (That's what the LLM was trained to do, understand and respond to text the same way we do.) We know that the bot has no hidden state when it's not the bot's turn because of how the bot's API works. So there's nowhere for a deeper simulation to live. It's as shallow as it looks. More: https://skybrian.substack.com/p/ai-chats-are-turn-based-games https://skybrian.substack.com/p/ai-chats-are-turn-based-game...
  - blueblimp 3y ago
    I've always thought of them a bit like improv, since they tend to follow the "yes, and..." rule, by happily continuing whatever direction you want to go. Now that the base models have been fine-tuned to avoid some topics, that's less true than it used to be, but it still feels like the most natural mode of operation.
- stevenhuang 3y ago
  Yup, that's a classic article. My favorite is Chalmer's engine bit: > What pops out of self-supervised predictive training is noticeably not a classical agent. Shortly after GPT-3’s release, David Chalmers lucidly observed that the policy’s relation to agents is like that of a “chameleon” or “engine”: >> GPT-3 does not look much like an agent. It does not seem to have goals or preferences beyond completing text, for example. It is more like a chameleon that can take the shape of many different agents. Or perhaps it is an engine that can be used under the hood to drive many agents. But it is then perhaps these systems that we should assess for agency, consciousness, and so on.6
  - fnordpiglet 3y ago
    Yeah that’s been my thought since the beginning. We have created all sorts of agents to date that are very compelling but lack a bridge into knowledge and synthesis of complex semantics. Wiring an ensemble of the various AI models created over the last 70 years I think is the final step.
- pmoriarty 3y ago
  One thing I've been impressed by recently is ChatGPT's ability to explain epigrams. How can it explain them effectively if it's just a "stochastic parrot" or a "blurry jpeg"? Neither fits what it does. Don't such effective explanations require some sort of understanding? Below is an example explanation, and there are more here: [1] --- Explain "In software systems it is often the early bird that makes the worm." This statement refers to the importance of early planning and implementation in software systems. In software development, the early stages of the project, such as requirements gathering, design, and implementation, are crucial for the success of the project. If the early stages of the project are done well, it can set the foundation for the rest of the development process, making it easier to build and maintain the software system. On the other hand, if these stages are not done well, it can lead to delays, rework, and budget overruns. The phrase "the early bird that makes the worm" is a reference to the proverb "the early bird catches the worm", which means that those who act early and decisively will have an advantage over others. Similarly, in software development, if the development team acts early and puts effort into the early stages of the project, they can gain an advantage over competitors, reducing the risk of project failure, and increasing the chances of success. In summary, the statement "In software systems, it is often the early bird that makes the worm" emphasizes the importance of early planning and implementation in software development, which can set the foundation for the rest of the development process and increase the chances of success. [1] - https://diaphora.blogspot.com/2023/04/chatgpt-explains-epigrams-on.html https://diaphora.blogspot.com/2023/04/chatgpt-explains-epigr...
  - krainboltgreene 3y ago
    That's not even a good explanation for that epigram and the output is similar to many written pieces of content on the internet, which ChatGPT almost surely borrows from via CommonCrawl.
  - dragonwriter 3y ago
    It literally misses any explanation of the only thing that distinguishes the epigram from the proverb it refers to. While identifying, and reproducing, with simple contextual substitutions, and explanation of the proverb referenced is not a small feat compared to where this kind of technology was a few years ago, its pretty darned far from either (1) actually explaining the epigram, or (2) making the argument you try to make based on explaining the epigram.
- Myrmornis 3y ago
  I’d like to know who the author is and what their background/professional experience is. Is that information available? I see they use a handle “moire”.
  - ftxbro 3y ago
    it's not me and they have wanted to be pseudonymous. pls no dox
booleandilemma 3y ago
It makes me sad that the next time I enjoy a piece of writing, I'm going to have to wonder if it was "enhanced" or even written wholesale by ChatGPT. I don't feel the same with arithmetic at all.
- majormajor 3y ago
  Is this a result of the machine-instead-of-human-reviewer/editor difference, or because of a sense that the writing is less a "pure" output of a singular author?
  - booleandilemma 3y ago
    Well I think it's like, you come to enjoy an author's particular style of writing, and if people are just going to use ChatGPT to write things for them, then they're not going to develop any style. Everyone might even end up all sounding the same. With a calculator this is a feature. We want computations to be the same after all. Everyone should be able to get the same results when they enter the same numbers in. But this homogeneity doesn't belong in writing.
    - galleywest200 3y ago
      What if people use them to try to learn a style? If someone is trying to write in a more active voice, or avoid gendered language, then are they suddenly not worth reading anymore? I think it is worth pointing out that plenty of prose editors advised on words to remove from your sentences (IA Writer as an example) before LLMs were a thing. If they used one of these tools making suggestions, would they no longer be worth reading? What about the green squiggle of grammar errors before this?
    - majormajor 3y ago
      Part of the appeal of ChatGPT compared to traditional writing assistant/grammar checkers is that you can tell it to write in different personas, although IME it's pretty spotty at this still. Lots of conversations that turn bland and repetitive quickly if you don't get lucky. I'm not too worried about "default GPT style" becoming common, though, because I think it's more likely to be used by the people who have no style beyond "what I see on TV and in my family." Raising the floor, basically. Anyone who wants their writing to stand out will still have to differentiate themselves. To put it another way: you're gonna be able to recognize the lazy users pretty quickly cause they're gonna have "GPT voice."
skybrian 3y ago
Often it's a hint generator that tells you where to look and what you could try. The hints are not calculated from the input, they're from the training set.
SomewhatLikely 3y ago
The author does a good job of pointing out what may be the strongest skills of LLMs but the claim they aren't useful as a search engine didn't ring particularly true. For many questions I have ChatGPT is the best tool to use because I know the topics I'm asking about are mentioned hundreds of times in the web, and the LLM can distill down at knowledge to the specifics I'm asking about. If you treat it as a friend who has a ton of esoteric knowledge in many areas but is prone to making stuff up to sound like they know what they're talking about you can still get lots of use pulling facts and some basic reasoning out of the models.
- skybrian 3y ago
  I think it's best used in conjunction in with a search engine. For example, you can ask it to recommend a paper to read and then search for the paper.
- stavros 3y ago
  I'm convinced LLMs are amazing for search, because they're the only thing that managed to give me an answer when I was looking for some software I didn't know the general category name for. I described what I wanted the software to do, corrected its understanding with a few followup messages, and it gave me a list of alternatives that are exactly what I wanted, along with a category name. Google, in comparison, returned absolutely irrelevant SEO spam.
  - fwlr 3y ago
    There’s two different things and we refer to both of them with the word “search”. Sometimes search means “I can sort of describe what I’m looking for, can you tell me what it’s called?”. LLMs excel here. I told GPT4 I’m doing computer animation and want to do smooth blending, it told me that’s called “interpolation”, I asked for some common terms in the literature about this to help me look and it told me about LERP, SLERP, quaternions, splines, Beziers, keyframes, inverse kinematics, and motion capture. All useful jumping-off points. (A subset of this type of search is “I know what this is called, can you tell me more about it?”. This is probably the place where LLMs sell snake oil the most; they always provide a convincing explanation of the thing, but there’s no guarantee on veracity.) Other times, search means “I have a specific phrase and I want to find occurrences of it”. LLMs aren’t just bad at this, they are constitutionally incapable of it. The way you build an LLM necessarily involves taking all specific phrases and occurrences thereof, and blending them up into a word slurry that is then condensed and abstracted into floating point weights. It no longer has the specifics to give you. It’s a shame that search engines have let this task (“ctrl-f the web”) fall by the wayside. It’s probably a large part of why people think Google search sucks now, it certainly is for me. (There’s this one essay about the Harappan civilization that I used to be able to find by searching for “strange builders mist of time”, I definitively remember that exact phrase working for me many years ago, and now it does not work and I cannot find that essay anymore.)
tbalsam 3y ago
I've seen this take a lot, and I find it frustrating as it flies in the face of the information theory underpinning how large neural networks learn information. This is more than just a fancy zip file of Markov sequences. Someone has got to put a stop to this silly line of reasoning, I'm not sure why more people familiar with the math of deep learning aren't doing their best to dispel this particular belief (which people will then use as the foundation for other arguments, and so on, and so on, and this is how misconceptions somehow become canon in the larger body of work).
- LelouBil 3y ago
  Can you explain more ? I know the basics of deep learning and I found the article accurate.
  - pretendscholar 3y ago
    As an amatuer I think Markov chains are explicitly a crude frequency association whereas what exactly a neural network is storing to predict the next token involves stored representations in neural weights which can be far more nuanced.
    - tbalsam 3y ago
      Thank you, this is a good compressed and less-salty response. I appreciate your contribution to the conversation and will use aspects of it when trying to explain the matter in the future. <3 :thumbsup:
  - trc001 3y ago
    I think parent is getting at the idea that dnn learns abstract representations of the data in whatever way is useful for the mode objective function. But I agree, the article still seems to present a reasonable argument
  - tbalsam 3y ago
    The fuzzy jpeg analogy and related kin ignore the internal disentanglement of ideas, which is what separates LLMs from, say, a probabilistic chain producer. I.e. one can think of it as a NERF of an underlying manifold instead of just assembling pictures taken of the manifold, which is an important distinction to make. I.e. it learns the manifold, not the manifold samples. That's what makes it so powerful and lets it coherently mix and match very abstract concepts together. Even if it gets it wrong, one could link that to the fuzziness of a NERF where there is not as much data. That's why this whole "average" business is silly nonsense. We're reducing the empirical risk over the dataset, not the L2 loss over it for Pete's sake.
trc001 3y ago
Good to see more people pointing out the problems using standalone llm (I.e. not connected to an external data source/the internet) for search. So many people I talk to dismiss gpt because it can’t answer some subject-specific question accurately.
transitivebs 3y ago
Great analogy & breakdown. My go-to explanation is to think of ChatGPT like a really intelligent friend who's always available to help you out – but they're also super autistic, and you need to learn the best way to interact with them over time.
zone411 3y ago
Please stop linking to Ted Chiang's article. I like him as an imaginative writer, but his article is just wrong and gives readers incorrect intuitions. His claim that GPT models are not able to learn decimal addition has been known to be false for years and you can verify yourself that GPT-4 can do it.
davidthewatson 3y ago
The most telling thing about the state-of-the-art currently is that somewhere between all the marketecture in no-code and low-code, we can essentially create a recursive crawler on-the-fly on par with a large cloud provider (because it is) and ask it to do recursive crawls that at least begin to approach a sensemaking machine, i.e. it can provide consensus for subjective truth agreed upon by experts above and beyond simple objective truth. To me, that's the great innovation that truly embraces and extends average intelligence by providing a prosthetic device for the brain to make inferences that would have only been approachable by high functioning individuals previously, i.e. the kind of argument about consciousness that you see from low latency inhibition researchers like Peterson. What any individual does with this is really where prompt engineering becomes the human-computer agent collaboration that should be our default mode in computing - a kind of tortoise wins the race story where we lost our minds in the race to interaction via javascript. It's not terribly interesting to watch a computer type. There's a place for batch mode (queuing, etc) , if the tools built up around it handle long running job management well. Sadly, that seems rarer to me now than 30 years ago.
stevenhuang 3y ago
> Language models don’t—if you run the same prompt through a LLM several times you’ll get a slightly different reply every time. You can get deterministic output (on a given machine) by setting temperature=0. The Chatgpt interface doesn't let you do that, but the playground API does.
- spencerchubb 3y ago
  Also if you specify the seed you get deterministic output, assuming that the interface allows to specify a seed.
- xigency 3y ago
  This is an interesting argument. It's evident that the LLM /can/ give different answers to the same question and that they are stochastic in nature. But being computer programs we can make them deterministic with respect to input as you suggest. More to the point, I don't think a "calculator for words" should be deterministic. Operating on language is much more subjective than operating on numbers. If anything, this is a human limitation that we expect only one answer to one question. I'm a contrarian to Chomsky's philosophy, as he's always been pessimistic of statistical language processing and often approaches from the more objective-side like grammar and parsing. I'm waiting for the point where we can tap knowledge from Deep Learning models to build rule-sets that appease the deterministic crowd (and get the insight of what an LLM is really modeling). A breakthrough here could also help with two big problems a) alignment and b) copyright.
freediver 3y ago
Another powerful use not mentioned in the article is the ability to convert unstructured data into structured date. For example you can copy paste a page describing API documentation and ask an LLM to not only make an API call but then also interpret results. This is the most fascinating use of LLMs to me so far.
- travisjungroth 3y ago
  I think this is one of the most powerful uses cases for the next one maybe two years (I have a hard time making guesses beyond that point these days). There’s a lot of stuff you can do with structured data. Millions of existing applications. Until now, that world is connected with our everyday world with something like a rope bridge. It’s like a six lane steel suspension bridge just popped up.
hbarka 3y ago
Confabulate is a better word than hallucinate. How did “hallucinating” get popularized? It’s a terrible term in this context.
- leobg 3y ago
  Ha! Totally agree. Used it in my emails to OpenAI in the early days of GPT-3, hoping they’d adapt the term. I knew it from psychology, where it refers to people giving rational explanations for something that isn’t there (specifically in split brain patients). I guess “hallucinate” stuck because it works across all disciplines: text, audio, vision…
- jazzyjackson 3y ago
  I feel like it had something to do with DeepDream - in the popular consciousness, tripping on acid / hallucinating became something that computers are surprisingly good at, and maybe that transferred to text models.
- hoseja 3y ago
  Probably from the early deep dream pictures?
rockzom 3y ago
[flagged]
teekert 3y ago
"The ChatGPT model is huge, but it’s not huge enough to retain every exact fact it’s encountered in its training set." That's because there is no way for the model to take the internet and separate fact from fiction or truth from falsehood. So it should not even try to, unless it can somehow weigh options (or preform its own experiments). And that doesn't mean counting occurrences, it means figuring out a coherent worldview and using it as a prior to interpret information, and then still acknowledging that it could be wrong.
gloosx 3y ago
I think a better analogy would be: "ChatGPT as a Crappy Primary School Teacher", has hard-coded traditionalistic morales, and can answer anything but information quality is strictly not guaranteed.
CamelRocketFish 3y ago
> if you run the same prompt through a LLM several times you’ll get a slightly different reply every time. If it has the same seed, why would you get a different reply?

201 points by stevefink 3y ago

mif 3y ago

I think the editorial capabilities of ChatGPT are fantastic, and the author provides a good list of examples. On top of that I would add that ChatGPT is really good as composing text. The meaning making is therefore still what we have to do.

ftxbro 3y ago

As a long time LLM enjoyer I just want to mention https://generative.ink/posts/simulators/ https://generative.ink/posts/simulators/ as I think it's by far the most insightful take on the GPT LLMs even though it was from before ChatGPT. It's better than blurry jpeg and stochastic parrot etc.

skybrian 3y ago
I think that article is misleading, because a simulator has rules. An LLM is better thought of as a storyteller, because at best it's going to follow whatever implicit rules there are very loosely and let you make rule changes of your own, more like Calvinball. Also, whatever loose rules it has are more literary than mathematical. Plot twists often work.
- tbalsam 3y ago
  I find this to be a better explanation than "it's just regurgitating strings of text" No, it is clearly not, and that is a very easily testable hypothesis. Thank you for sharing.
  - Al0neStar 3y ago
    testable how?
    - tbalsam 3y ago
      out of distribution tests. if the concept holds consistently over a long period, the concept is the stable thing. If not, then it's only memorizing string densities. For a number of years we've been basically showing the first to be the case, especially as the model is scaled and the context increases, differentially against the second. String density probabilities can be surprisingly brittle, to be honest. The curse of dimensionality applies to them too, believe it or not, which I believe is why topic discussion, reasoning, and integration over longer distances of text is that differential test that shows pretty clearly that substring memorization/text density stuff is not 'just' what the model is learning. Because mathematically/statistically/from an information density perspective/etc etc otherwise it would be basically impossible, I think. That's my best understanding, at least.
- ftxbro 3y ago
  The essay is long and complicated so I'm not sure how much of it you read closely, but it specifically addresses this distinction between the simulator and the simulacrum. In the analogy of the essay, your argument would be like saying that reality cannot be simply the application of quantum physics, because you are allowed to make new rules like Calvinball within reality which are different from the rules of quantum physics.
  - skybrian 3y ago
    I do understand the difference between a simulator and what's being simulated. I still think they got it all wrong, that the simulator is better called a "writer," the simulated world is better called a "story," and the agent is a better called a "fictional character." We know there's no deeper level to the simulation/game because we have the entire "game history" (the chat history) and we understand it in approximately same way that the LLM does. (That's what the LLM was trained to do, understand and respond to text the same way we do.) We know that the bot has no hidden state when it's not the bot's turn because of how the bot's API works. So there's nowhere for a deeper simulation to live. It's as shallow as it looks. More: https://skybrian.substack.com/p/ai-chats-are-turn-based-games https://skybrian.substack.com/p/ai-chats-are-turn-based-game...
- blueblimp 3y ago
  I've always thought of them a bit like improv, since they tend to follow the "yes, and..." rule, by happily continuing whatever direction you want to go. Now that the base models have been fine-tuned to avoid some topics, that's less true than it used to be, but it still feels like the most natural mode of operation.
stevenhuang 3y ago
Yup, that's a classic article. My favorite is Chalmer's engine bit: > What pops out of self-supervised predictive training is noticeably not a classical agent. Shortly after GPT-3’s release, David Chalmers lucidly observed that the policy’s relation to agents is like that of a “chameleon” or “engine”: >> GPT-3 does not look much like an agent. It does not seem to have goals or preferences beyond completing text, for example. It is more like a chameleon that can take the shape of many different agents. Or perhaps it is an engine that can be used under the hood to drive many agents. But it is then perhaps these systems that we should assess for agency, consciousness, and so on.6
- fnordpiglet 3y ago
  Yeah that’s been my thought since the beginning. We have created all sorts of agents to date that are very compelling but lack a bridge into knowledge and synthesis of complex semantics. Wiring an ensemble of the various AI models created over the last 70 years I think is the final step.
pmoriarty 3y ago
One thing I've been impressed by recently is ChatGPT's ability to explain epigrams. How can it explain them effectively if it's just a "stochastic parrot" or a "blurry jpeg"? Neither fits what it does. Don't such effective explanations require some sort of understanding? Below is an example explanation, and there are more here: [1] --- Explain "In software systems it is often the early bird that makes the worm." This statement refers to the importance of early planning and implementation in software systems. In software development, the early stages of the project, such as requirements gathering, design, and implementation, are crucial for the success of the project. If the early stages of the project are done well, it can set the foundation for the rest of the development process, making it easier to build and maintain the software system. On the other hand, if these stages are not done well, it can lead to delays, rework, and budget overruns. The phrase "the early bird that makes the worm" is a reference to the proverb "the early bird catches the worm", which means that those who act early and decisively will have an advantage over others. Similarly, in software development, if the development team acts early and puts effort into the early stages of the project, they can gain an advantage over competitors, reducing the risk of project failure, and increasing the chances of success. In summary, the statement "In software systems, it is often the early bird that makes the worm" emphasizes the importance of early planning and implementation in software development, which can set the foundation for the rest of the development process and increase the chances of success. [1] - https://diaphora.blogspot.com/2023/04/chatgpt-explains-epigrams-on.html https://diaphora.blogspot.com/2023/04/chatgpt-explains-epigr...
- krainboltgreene 3y ago
  That's not even a good explanation for that epigram and the output is similar to many written pieces of content on the internet, which ChatGPT almost surely borrows from via CommonCrawl.
- dragonwriter 3y ago
  It literally misses any explanation of the only thing that distinguishes the epigram from the proverb it refers to. While identifying, and reproducing, with simple contextual substitutions, and explanation of the proverb referenced is not a small feat compared to where this kind of technology was a few years ago, its pretty darned far from either (1) actually explaining the epigram, or (2) making the argument you try to make based on explaining the epigram.
Myrmornis 3y ago
I’d like to know who the author is and what their background/professional experience is. Is that information available? I see they use a handle “moire”.
- ftxbro 3y ago
  it's not me and they have wanted to be pseudonymous. pls no dox

booleandilemma 3y ago

It makes me sad that the next time I enjoy a piece of writing, I'm going to have to wonder if it was "enhanced" or even written wholesale by ChatGPT. I don't feel the same with arithmetic at all.

majormajor 3y ago
Is this a result of the machine-instead-of-human-reviewer/editor difference, or because of a sense that the writing is less a "pure" output of a singular author?
- booleandilemma 3y ago
  Well I think it's like, you come to enjoy an author's particular style of writing, and if people are just going to use ChatGPT to write things for them, then they're not going to develop any style. Everyone might even end up all sounding the same. With a calculator this is a feature. We want computations to be the same after all. Everyone should be able to get the same results when they enter the same numbers in. But this homogeneity doesn't belong in writing.
  - galleywest200 3y ago
    What if people use them to try to learn a style? If someone is trying to write in a more active voice, or avoid gendered language, then are they suddenly not worth reading anymore? I think it is worth pointing out that plenty of prose editors advised on words to remove from your sentences (IA Writer as an example) before LLMs were a thing. If they used one of these tools making suggestions, would they no longer be worth reading? What about the green squiggle of grammar errors before this?
  - majormajor 3y ago
    Part of the appeal of ChatGPT compared to traditional writing assistant/grammar checkers is that you can tell it to write in different personas, although IME it's pretty spotty at this still. Lots of conversations that turn bland and repetitive quickly if you don't get lucky. I'm not too worried about "default GPT style" becoming common, though, because I think it's more likely to be used by the people who have no style beyond "what I see on TV and in my family." Raising the floor, basically. Anyone who wants their writing to stand out will still have to differentiate themselves. To put it another way: you're gonna be able to recognize the lazy users pretty quickly cause they're gonna have "GPT voice."

skybrian 3y ago

Often it's a hint generator that tells you where to look and what you could try. The hints are not calculated from the input, they're from the training set.

SomewhatLikely 3y ago

The author does a good job of pointing out what may be the strongest skills of LLMs but the claim they aren't useful as a search engine didn't ring particularly true. For many questions I have ChatGPT is the best tool to use because I know the topics I'm asking about are mentioned hundreds of times in the web, and the LLM can distill down at knowledge to the specifics I'm asking about. If you treat it as a friend who has a ton of esoteric knowledge in many areas but is prone to making stuff up to sound like they know what they're talking about you can still get lots of use pulling facts and some basic reasoning out of the models.

skybrian 3y ago
I think it's best used in conjunction in with a search engine. For example, you can ask it to recommend a paper to read and then search for the paper.
stavros 3y ago
I'm convinced LLMs are amazing for search, because they're the only thing that managed to give me an answer when I was looking for some software I didn't know the general category name for. I described what I wanted the software to do, corrected its understanding with a few followup messages, and it gave me a list of alternatives that are exactly what I wanted, along with a category name. Google, in comparison, returned absolutely irrelevant SEO spam.
- fwlr 3y ago
  There’s two different things and we refer to both of them with the word “search”. Sometimes search means “I can sort of describe what I’m looking for, can you tell me what it’s called?”. LLMs excel here. I told GPT4 I’m doing computer animation and want to do smooth blending, it told me that’s called “interpolation”, I asked for some common terms in the literature about this to help me look and it told me about LERP, SLERP, quaternions, splines, Beziers, keyframes, inverse kinematics, and motion capture. All useful jumping-off points. (A subset of this type of search is “I know what this is called, can you tell me more about it?”. This is probably the place where LLMs sell snake oil the most; they always provide a convincing explanation of the thing, but there’s no guarantee on veracity.) Other times, search means “I have a specific phrase and I want to find occurrences of it”. LLMs aren’t just bad at this, they are constitutionally incapable of it. The way you build an LLM necessarily involves taking all specific phrases and occurrences thereof, and blending them up into a word slurry that is then condensed and abstracted into floating point weights. It no longer has the specifics to give you. It’s a shame that search engines have let this task (“ctrl-f the web”) fall by the wayside. It’s probably a large part of why people think Google search sucks now, it certainly is for me. (There’s this one essay about the Harappan civilization that I used to be able to find by searching for “strange builders mist of time”, I definitively remember that exact phrase working for me many years ago, and now it does not work and I cannot find that essay anymore.)

tbalsam 3y ago

I've seen this take a lot, and I find it frustrating as it flies in the face of the information theory underpinning how large neural networks learn information. This is more than just a fancy zip file of Markov sequences. Someone has got to put a stop to this silly line of reasoning, I'm not sure why more people familiar with the math of deep learning aren't doing their best to dispel this particular belief (which people will then use as the foundation for other arguments, and so on, and so on, and this is how misconceptions somehow become canon in the larger body of work).

LelouBil 3y ago
Can you explain more ? I know the basics of deep learning and I found the article accurate.
- pretendscholar 3y ago
  As an amatuer I think Markov chains are explicitly a crude frequency association whereas what exactly a neural network is storing to predict the next token involves stored representations in neural weights which can be far more nuanced.
  - tbalsam 3y ago
    Thank you, this is a good compressed and less-salty response. I appreciate your contribution to the conversation and will use aspects of it when trying to explain the matter in the future. <3 :thumbsup:
- trc001 3y ago
  I think parent is getting at the idea that dnn learns abstract representations of the data in whatever way is useful for the mode objective function. But I agree, the article still seems to present a reasonable argument
- tbalsam 3y ago
  The fuzzy jpeg analogy and related kin ignore the internal disentanglement of ideas, which is what separates LLMs from, say, a probabilistic chain producer. I.e. one can think of it as a NERF of an underlying manifold instead of just assembling pictures taken of the manifold, which is an important distinction to make. I.e. it learns the manifold, not the manifold samples. That's what makes it so powerful and lets it coherently mix and match very abstract concepts together. Even if it gets it wrong, one could link that to the fuzziness of a NERF where there is not as much data. That's why this whole "average" business is silly nonsense. We're reducing the empirical risk over the dataset, not the L2 loss over it for Pete's sake.

trc001 3y ago

Good to see more people pointing out the problems using standalone llm (I.e. not connected to an external data source/the internet) for search. So many people I talk to dismiss gpt because it can’t answer some subject-specific question accurately.

transitivebs 3y ago

Great analogy & breakdown. My go-to explanation is to think of ChatGPT like a really intelligent friend who's always available to help you out – but they're also super autistic, and you need to learn the best way to interact with them over time.

zone411 3y ago

Please stop linking to Ted Chiang's article. I like him as an imaginative writer, but his article is just wrong and gives readers incorrect intuitions. His claim that GPT models are not able to learn decimal addition has been known to be false for years and you can verify yourself that GPT-4 can do it.

davidthewatson 3y ago

The most telling thing about the state-of-the-art currently is that somewhere between all the marketecture in no-code and low-code, we can essentially create a recursive crawler on-the-fly on par with a large cloud provider (because it is) and ask it to do recursive crawls that at least begin to approach a sensemaking machine, i.e. it can provide consensus for subjective truth agreed upon by experts above and beyond simple objective truth. To me, that's the great innovation that truly embraces and extends average intelligence by providing a prosthetic device for the brain to make inferences that would have only been approachable by high functioning individuals previously, i.e. the kind of argument about consciousness that you see from low latency inhibition researchers like Peterson. What any individual does with this is really where prompt engineering becomes the human-computer agent collaboration that should be our default mode in computing - a kind of tortoise wins the race story where we lost our minds in the race to interaction via javascript. It's not terribly interesting to watch a computer type. There's a place for batch mode (queuing, etc) , if the tools built up around it handle long running job management well. Sadly, that seems rarer to me now than 30 years ago.

stevenhuang 3y ago

> Language models don’t—if you run the same prompt through a LLM several times you’ll get a slightly different reply every time. You can get deterministic output (on a given machine) by setting temperature=0. The Chatgpt interface doesn't let you do that, but the playground API does.

spencerchubb 3y ago
Also if you specify the seed you get deterministic output, assuming that the interface allows to specify a seed.
xigency 3y ago
This is an interesting argument. It's evident that the LLM /can/ give different answers to the same question and that they are stochastic in nature. But being computer programs we can make them deterministic with respect to input as you suggest. More to the point, I don't think a "calculator for words" should be deterministic. Operating on language is much more subjective than operating on numbers. If anything, this is a human limitation that we expect only one answer to one question. I'm a contrarian to Chomsky's philosophy, as he's always been pessimistic of statistical language processing and often approaches from the more objective-side like grammar and parsing. I'm waiting for the point where we can tap knowledge from Deep Learning models to build rule-sets that appease the deterministic crowd (and get the insight of what an LLM is really modeling). A breakthrough here could also help with two big problems a) alignment and b) copyright.

freediver 3y ago

Another powerful use not mentioned in the article is the ability to convert unstructured data into structured date. For example you can copy paste a page describing API documentation and ask an LLM to not only make an API call but then also interpret results. This is the most fascinating use of LLMs to me so far.

travisjungroth 3y ago
I think this is one of the most powerful uses cases for the next one maybe two years (I have a hard time making guesses beyond that point these days). There’s a lot of stuff you can do with structured data. Millions of existing applications. Until now, that world is connected with our everyday world with something like a rope bridge. It’s like a six lane steel suspension bridge just popped up.

hbarka 3y ago

Confabulate is a better word than hallucinate. How did “hallucinating” get popularized? It’s a terrible term in this context.

leobg 3y ago
Ha! Totally agree. Used it in my emails to OpenAI in the early days of GPT-3, hoping they’d adapt the term. I knew it from psychology, where it refers to people giving rational explanations for something that isn’t there (specifically in split brain patients). I guess “hallucinate” stuck because it works across all disciplines: text, audio, vision…
jazzyjackson 3y ago
I feel like it had something to do with DeepDream - in the popular consciousness, tripping on acid / hallucinating became something that computers are surprisingly good at, and maybe that transferred to text models.
hoseja 3y ago
Probably from the early deep dream pictures?

rockzom 3y ago

[flagged]

teekert 3y ago

"The ChatGPT model is huge, but it’s not huge enough to retain every exact fact it’s encountered in its training set." That's because there is no way for the model to take the internet and separate fact from fiction or truth from falsehood. So it should not even try to, unless it can somehow weigh options (or preform its own experiments). And that doesn't mean counting occurrences, it means figuring out a coherent worldview and using it as a prior to interpret information, and then still acknowledging that it could be wrong.

gloosx 3y ago

I think a better analogy would be: "ChatGPT as a Crappy Primary School Teacher", has hard-coded traditionalistic morales, and can answer anything but information quality is strictly not guaranteed.

CamelRocketFish 3y ago

> if you run the same prompt through a LLM several times you’ll get a slightly different reply every time. If it has the same seed, why would you get a different reply?