7 ms·
Is NLP a solved problem now?
by sorz 3y ago
Is NLP a solved problem now?
- kolinko 3y agoNot sure why you got downvotes. It seems like it's a solved issue indeed. AI reasoning has still some way to go, but it seems language understanding is a finished subject.
- tgv 3y agoRegurgitating training data trigram by trigram is not how human language processing works.
- hallway_monitor 3y agoYou sure about that? The more I interact with LLMs and learn how they operate, the more it seems to me like people operate on very similar principles and algorithms with their use of language.
- tgv 3y agoThe "if it walks like a duck" school of ontology.
- maxdoop 3y agoAnd how does it work, then? Everyone shrugs and says, “nope, humans are different”. I’ve commented about 100 times recently asking for detail as to how human language / thought works, yet have not seen an answer.
- tgv 3y agoWe interpret what we hear, make a mental representation of that (incrementally; this process sometimes fails), which links to concepts, which in turn can link to memories, then "look for the answer" (if it's a question) by association and puzzling, the former is pretty quick, the latter slow, check if the answer makes sense, and formulate a reply. We can start formulating a reply from similarly formed structures while completing the thought, because we monitor our speech. When that happens, you often say "er..." That's basic linguistics and cognitive psychology. Nothing an LLM has done has invalidated that.
- est 3y agofor stuff that before 2021-09, mostly.
- ninjin 3y agoObviously not. If we had a general solution to language intelligence we would have artificial intelligence at the level of at least human intelligence – which we do not. Rather, the right question to ask is which language intelligence tasks currently have acceptable performance and under which conditions (text domain, etc.). Clearly this is a much more difficult question and with a lot more nuance to it, even if it is undeniably that things have moved very quickly over the last few years. Skimming the abstract as a senior academic in the area. This looks like preliminary work and a limited investigation for a single (non-standard) task. Thus far from a strong result published at say a top-tier conference or journal. Still, interesting direction and if expanded upon could absolutely be impactful. I should also mention that I am not familiar with the related literature, so it could very much be that there is similar (better?) work out there exploring the same question.
- famouswaffles 3y agoWe have artificial intelligence that is general and above average human intelligence for the majority of tasks it can perform. Near expert level for some. NLP is a solved problem. Bespoke models are out the door. Large enough LLMs crush anything else for any NLP task. Honestly, this whole "they are not intelligent" argument is becoming ridiculous. might as well argue that a plane isn’t a real bird or a car isn’t a real horse.
- mistrial9 3y agono, a short answer to this is .. these models are probabilistic, therefore they will always have errors along with whatever else. Secondly "intelligence" is not one thing; no one has all of it or none of it, including computers.
- nico 3y ago> these models are probabilistic, therefore they will always have errors There's nothing perfect. Even computers and computer networks need to have error-correcting code because information gets randomly corrupted. Our whole reality is probabilistic. And us humans are way worse than AI at consistency. We even overwrite our own memories all the time, so we can't even be sure what we remember is actually what happened! (btw, this is currently being used in therapy to re-write traumatic memories and help people overcome PTSD). https://www.npr.org/sections/health-shots/2014/02/04/271527934/our-brains-rewrite-our-memories-putting-present-in-the-past https://www.npr.org/sections/health-shots/2014/02/04/2715279...
- nynx 3y agoMostly, yes. LLMs will turn natural language into really whatever form you want.
- lisasays 3y agoNLP is just a matter doing tasks at the accuracy level of an MTurk, you say?
- rjh29 3y agoIdiomatic translation of text matching a human professional (e.g. free of errors for legal terms, interesting and natural for fiction) is unlikely to be achieved until we have AGI. So no.
- mistrial9 3y agoI wrote accepted corrections to state regulations law on a particular topic, and I can tell you that super-dense legalese for big-time industrial topics, had loopy and inconsistent language.
- famouswaffles 3y agoI won't comment on the first bit as i've not personally tested in that are but GPT-4 can absolutely make short work on the second. I don't think people realize how good Bilingual LLMs are at translations. Yes you have idioms transfer between languages. Feel free to test it yourself.
- rjh29 3y agoI have tested it :) I've asked it to translate English fictional text into Japanese, it falls over often. It's unnatural and often makes no sense at all. It doesn't compare to a typical professional translation (which are often not that idiomatic either), let alone a really good one. I'm sure it'll be doing that in five years, but not now. One interesting thing is that's it's nondeterministic, so sometimes 'For chrissakes' turns to ちくしょう (Damn!) but sometimes to クリスのために (for Chris' sake). Sometimes 'the goddamn door' turns into クソドア ('shit door'), sometimes the goddamn changes the phrasing of the whole sentence instead. If you run it five times and take the best sentences out of all five runs it's probably quite good. Maybe prompting would help too, I said "idiomatic Japanese" but it still usually translated it in a very "foreigner Japanese" way typical of US drama/movie translations.
- bumbledraven 3y agoAre you giving it multiple paragraphs to translate at once so that it has enough context for a good translation? If so, would you mind sharing a sample input and output that you found unsatisfactory? In "Can GPT-4 translate literature?" (Mar 18, 2023) [https://youtu.be/5KKDCp3OaMo?t=377 https://youtu.be/5KKDCp3OaMo?t=377], Tom Gally, a former professional translator and current professor at the University of Tokyo, said: > …the point is, to my eye at least, the the basic quality of the translation [from Japanese to English] is as good as a human might do, and with some relatively mild editing by a sensitive human editor, it could be turned into a novel that would be readable and enjoyable.
- rtkwe 3y agoI still think we're a long ways off. LLMs can't to my knowledge process a request into a lookup on say an actual database of facts at the moment or parse a request into API actions. So far it's shown it's really really good at continuing a conversation with more text but as far as I understand them there's not a usable comprehension of what's actually being asked and answered. The point that would say to me the LLM actually has any "understanding" of what it's saying would be when it's able to reliably say "I don't know the answer to that" instead of making up things from scratch. You see that a lot if you ask Bing/Bard "Who is _____?" Most of them are kind of right but a lot of large details are just completely fabricated. A lot of the facts it gets wrong are things Google is already able to produce when queried like where was Person X born or where did they go to school so the fact these LLMs can't slot in actual available facts says to me they're not really going to be that useful with the kind of tasks we've been working on NLP for.
- ummonk 3y ago“LLMs can't to my knowledge process a request into a lookup on say an actual database of facts at the moment or parse a request into API actions.” Both Bing chat and ChatGPT plugins are examples of being able to do just this. You’re right about how they make up answers though, but humans are often quite prone to that too…
- rtkwe 3y agoA human, if not incentivized to lie or directly incentivized to be truthful, could at least tell you when they're making something up themselves where Bing/Bard seemingly cannot. Once it can do that I think they'll be far more useful, at least then you can have a rough idea of how much you need to check the bots work. If I have to do that for every thing it spits out the best it can do for me is give me new words to use while searching. Granted getting the name for something to search is often half the battle in tech.
- smallnamespace 3y ago> could at least tell you when they're making something up themselves where Bing/Bard seemingly cannot. In fact GPT-4 is quite good at catching hallucinations when the question-answer pair is fed back to itself. This isn’t automatically applied already because the model is expensive to run, but you can just do it yourself (or automate it with a plug-in or LangChain) and pay the extra cost. Remember that the model only performs a fixed amount of computation per generated token, so just asking it to think out loud or evaluate its own responses is basically giving it a scratchpad to think harder about your question.
- matusp 3y agoShameless self-promotion, I have recently written a blog about this. ChatGPT actually is usually a little bit worse than older models for these classical NLP tasks. Of course the older models are not zero-shot. https://www.opensamizdat.com/posts/chatgpt_survey/ https://www.opensamizdat.com/posts/chatgpt_survey/
- sorz 3y agoJust watched a talk[0] about natural language understanding research in post-GPT-3 era. Old issues may has been solved, while new topics are coming to this area (quoted from the slides): - Retrieval augmented in-context learning - Better benchmarks - Last mile for productive application - Faithful, human-interoperable explanations [0] https://www.youtube.com/watch?v=-lnHHWRCDGk https://www.youtube.com/watch?v=-lnHHWRCDGk