9 ms·
ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
- deleted 4y ago[deleted]
- totoglazer 4y agoThis might be an interesting language model. However people care about ChatGPT entirely due to its quality, which this doesn’t demonstrate yet.
- phist_mcgee 4y agoThe leap in public exposure wasn't so much GPT3 to GPT3.5, it was attaching a clean UI to the model, (with sane defaults) and allowing people to talk to it like a person. Suddenly it became something 'real' then. (This is purely talking about the public popularity of GPT)
- TJSomething 4y agoOne of the important parts of ChatGPT over plain GPT-3 is the reinforcement learning from human feedback to ensure alignment, without which it's not quite as good of a product for the public.
- tinsmith 4y agoThis is a remarkably good take that just didn't dawn on me until I read your comment. Even if ChatGPT had a lesser quality than the current iteration, the fact that they had a way for anyone to easily interact with it really was a homerun, snd can be for any software, really.
- b33j0r 4y agoMy family told me that ChatGPT came up from the pulpit AT CHURCH Me? I made a few comments like a scared luddite when ChatGPT solved two of my outstanding engineering problems instantly. I got better. But this is exactly right. The world in general now knows about AI and ML. It’s a pivot point. When something scares a seasoned engineer for a minute, and anyone can now make use of this… write it down in your diary as a moment in history
- sinuhe69 4y agoI wonder what kind of engineering problems you’ve talked with ChatGPT. Because its limited understanding of spatial problems, I found it hard to use it for mechanical engineering. How did you do it?
- deleted 4y ago[deleted]
- b33j0r 4y agoI wrote another response earlier, but found myself ranting ;) ChatGPT is currently best at things programmers would think about. You’re correct about spatial reasoning. But try stuff like this: “Write a python program that calculates the static forces on a cantilevered ledge 15 feet long, with a support beam” Haha it took the longest I’ve ever seen. You may have a point. It’s really good at writing code though. Caution. I tried my example with matlab instead of python, and I think I may have set a server rack on fire ;)
- junipertea 4y agoThey also did reinforcement learning on top of a frozen trained model. It is considerably more than just attaching a UI as that would just finish sentences compared to answering questions. https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf
- totoglazer 4y agoNo. ChatGPT’s UI is incredibly simple and basically exactly what ever chat bot test repl looks like. The delta of GPT3 -> ChatGPT is from the expanded context and control the model offers through fine tuning. Eg read the instructgpt paper to see the path on the way to ChatGPT.
- redox99 4y agoIt's not just the UI. ChatGPT (which is further finetuned and uses RLHF) definitely produces better output than GPT3, especially without prompt engineering.
- gamegoblin 4y agoThis is mostly correct. GPT3.5 is better, has a larger context window, etc. But it's a very incremental step above GPT3. I had wired up GPT3 to a Twilio phone number and made something basically like ChatGPT months before ChatGPT was released -- me and my friends texted it all the time to get information, similar to how people use ChatGPT. The prompt to get decent performance is super simple. Just something like: The following is a transcript between a human and a helpful AI assistant. The AI assistant is knowledgeable about most facts of the world and provides concise answers to questions. Transcript: {splice in the last 30 messages of the conversation} The next thing the assistant says is: Over time I did upgrade the prompt a bit to improve performance for specific kinds of queries, but nothing crazy. Cost me $10-20/mo to run for the low/moderate use by me and a few friends. Interestingly, for people who didn't know its limitations / how to break it, it was basically passing the turing test. ChatGPT is inhumanly wordy, whereas GPT3 can actually be much more concise when prompted to do so. If, instead of prompting it that it is an AI assistant, you prompt it that it is a close friend with XYZ personality traits, it does a very good job of carrying on a light SMS conversation.
- merely-unlikely 4y agoI half remember one of Google’s many chat apps having an AI assistant a number of years ago (Allo maybe?)
- andai 4y ago>If [...] you prompt it that it is a close friend with XYZ personality traits A couple years ago a friend and I trained GPT-2 on our WhatsApp chat history. GPT-2 was more primitive, but it still managed to capture the gist of our personalities and interests, which was equal parts amusing and embarrassing. We'd have it generate random chats, or ask it questions to see what simulated versions of ourselves would say.
- moffkalast 4y agoWell yes, having no context memory, being slightly worse and requiring either a monster rig to run or paying per prompt made it completely and utterly irrelevant. Even now that it's improved and free to use its actual practical usability is marginal at best given the rate of blatantly wrong info being spewed with 105% confidence at the moment.
- visarga 4y ago> blatantly wrong info being spewed with 105% confidence There are some approaches. For example in this paper they say truth has a certain logical consistency that is lacking in hallucinations and deception. So they find this latent direction that indicates truth in a frozen LLM. This actually works better than asking the model to self evaluate by text generation, or training with RLHF. "Discovering Latent Knowledge in Language Models Without Supervision" https://arxiv.org/abs/2212.03827 https://arxiv.org/abs/2212.03827 There's also a video with the first author: "Making LLMs Say The Truth" https://www.youtube.com/watch?v=XSQ495wpWXs&t=1515s https://www.youtube.com/watch?v=XSQ495wpWXs&t=1515s Btw, I think this is one of the deepest discussions about LLM hallucinations and alignment I ever saw. Worth a watch, even if it is a bit long. Not every day something like this comes long.
- HarHarVeryFunny 4y agoVery interesting video - thanks for posting that. It makes you wonder what other abstract concepts current models may have had to learn to get as good as they are. If they're doing a good job of modelling when someone is speaking the truth, then what else have they learnt about us? How complete of a "world model" can you learn purely in a passive way by consuming whatever online text is available to train on, or maybe by consuming all existent written material were it to be digitized? At some point I'm sure you need to be able to interact with the world to test hypothesis etc, but how far can predictive "intelligence" go without that?
- t3estabc 4y ago[dead]
- axiom92 4y agoSome evidence to confirm this: 1. Twitter thread with examples: https://twitter.com/sjwhitmore/status/1601254826947784705 https://twitter.com/sjwhitmore/status/1601254826947784705 2. Tweet/screenshot + Colab notebook:https://twitter.com/aman_madaan/status/1599549721030246401 https://twitter.com/aman_madaan/status/1599549721030246401, https://tinyurl.com/codex-chat-gpt https://tinyurl.com/codex-chat-gpt The second tweet is mine.
- didntreadarticl 4y agoI dunno, I played a lot of AI Dungeon when it was GPT3 powered. It was quite nonsencial sometimes. ChatGPT is much better due to reinforcement learning and optimisation they did with it
- deleted 4y ago[deleted]
- nl 4y agoFor those wondering how on earth they are getting decent results from a RNN without long range forgetting, I don't really know either! But they reference https://arxiv.org/abs/2105.14103 https://arxiv.org/abs/2105.14103 and the bottom section of https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM has an explainer.
- beernet 4y ago[flagged]
- deleted 4y ago[deleted]
- leodriesch 4y agoThe readme does not seem to be geared towards people not familiar with the topic. My questions: - Is this on the run on consumer GPU scale, or run on 8 A100 scale or you can’t run it yourself ever scale? - How does it compare to other language models in quality/abilities? - What is the training data?
- zone411 4y agoThe large model weights are 14B, so at 16 bits per weight, it won't quite fit on one 3090 or 4090.
- deleted 4y ago[deleted]
- aljungberg 4y agoIt does say on there they are training it on the Pile training data. And they have this bit comparing inference with GPT2-XL: RWKV-3 1.5B on A40 (tf32) = always 0.015 sec/token, tested using simple pytorch code (no CUDA), GPU utilization 45%, VRAM 7823M GPT2-XL 1.3B on A40 (tf32) = 0.032 sec/token (for ctxlen 1000), tested using HF, GPU utilization 45% too (interesting), VRAM 9655M So it looks about twice as fast for inference while using only about 80% as much VRAM. Obviously at such a small size, just 1.5B, you can run it even on consumer GPUs but you could do that with GPT2 as well. If it remains 80% of VRAM usage when scaled up, we’re still talking 282GB once it’s the size of BLOOM w/ 176B parameters. So yeah still 8x A100 40GB cards I guess. Not going to be the Stable Diffusion of LLMs.
- taktoa 4y agoI'm pretty sure those numbers are for training, not inference. I've run it on _CPU_ and gotten ~1 token per second.
- aljungberg 4y agoTHe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a combination of short and long term memory. But isn’t that what LSTMs were supposed to do too?
- swyx 4y ago> RNNs famously have infinite context in theory and very short context in reality. any sources to read more about this please? its the first ive heard of it
- georgehill 4y agoI am not sure this article will answer your question, but Karpathy has an article about RNNs. https://karpathy.github.io/2015/05/21/rnn-effectiveness https://karpathy.github.io/2015/05/21/rnn-effectiveness
- swyx 4y agoit doesnt touch on the "infinite context in theory and very short context in reality" piece which is what i was asking about
- sillysaurusx 4y agoI can confirm it from what we’re seeing on a video prediction task. Future frames end up blurry. The first frame is sharp, but by frame 3 it’s only crisp when it’s very certain of its prediction. Any kind of rare movement, it goes “I kinda know what it roughly looks like” and smears fingerpaint all over the canvas. The overall trajectory looks ok, so I’ll be more rigorously investigating whether it’s possible to squeeze more precise context out of it. For example, since the first frame is sharp, you could discard the other future frames and use that first frame as the last history entry (rolling completion window). If “the first frame is always sharp” is true, then it seems reasonable that you can generate N sharp frames with that technique, which might work better than predicting N all at once.
- rkwasny 4y agoTurns out it does not matter if you have transformer/MLP/lstm or whatever, as long as there are enough parameters and training epochs over large dataset things "just work"
- BulgarianIdiot 4y agoTo some degree, because we keep recreating the truly essential components the crude "Turing machine completeness" way. In time as we analyze the resulting models, we may find what patterns emerge and optimize for them. The result will be smaller, faster models that perform like larger slower ones.
- nl 4y agoThis isn't true - the model architecture matters a lot. In general RNNs cannot handle long term dependencies (ie, long pieces of text) because the gradient vanishes. It's unclear how this solves this problem although they do reference the "attention free transformer" paper: https://arxiv.org/abs/2105.14103 https://arxiv.org/abs/2105.14103
- cztomsik 4y agoI believe it's because you train it in GPT-mode and then only use RNN-mode for inference.
- PartiallyTyped 4y agoThe key component is the linear attention[1] and residual connections. [1] https://arxiv.org/abs/2006.16236 https://arxiv.org/abs/2006.16236 > Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from (N2) to (N), where N is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
- adam_arthur 4y agoSo what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large value add, competitors can easily undercut eachother)
- numbchuckskills 4y ago[dead]
- wokwokwok 4y ago> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for inference using Bloom: https://github.com/bigscience-workshop/petals/wiki/Launch-your-own-swarm https://github.com/bigscience-workshop/petals/wiki/Launch-yo... (You'll need around 200GB of GPU memory across the machines in the swarm)
- lfkdev 4y agoTwo or three top GPUs? Thats basically nothing for a professinal project or even an investeded hobby
- throwifasd 4y ago[flagged]
- ronsor 4y agoA few 4090Ti GPUs might be good enough, if they do indeed end up with 48GB of VRAM.
- leaving 4y ago[flagged]
- anon291 4y agoThis is a git repo, not a published paper. Hacker news is not a published journal. It's a casual space for technically oriented people. And you can say whatever you want on your own GitHub.
- VadimPR 4y agoHow does this compare to BLOOMZ's performance, if anyone knows?
- euclaise 4y agoAssuming you're referring to the largest model - BLOOM is huge, this is not, so presumably much worse
- moneywoes 4y agoName rolls of the tongue
- avmich 4y agoYeah, when the Web was young, and people told URLs to each other, pronouncing "www", which was almost always the prefix of any web server host name also sounded funny.
- klabb3 4y agoNot sure what you’re talking about. Eitch tee tee pee ess colon slash slash doubleview doubleview doubleview dot just rolls off the tongue so easily.
- golem14 4y agoit's "dub-dub-dub", isn't it?
- denton-scratch 4y agoWuh-wuh-wuh.
- joshxyz 4y agochat are-woo-kei-vee. gpt rolls out real better lol.
- haint_ 4y agoFrom the provided example: Q: How would I make for loop in python? A: I can help you create an AI chat bot. It would talk to you like a human. (additional text that is not relevant to the prompt) It is just me or this does not seem right?
- zaptrem 4y agoThis is only a 1.5b parameter model. This is in line with that. GPT3.5 is ~175b params.
- cztomsik 4y agoGive it at least few examples. ~1B networks are not good in zero-shot. Also, don't expect to get answers for things it was not trained on. the_pile is not programming dataset. RWKV is important because it's fast, it can be trained in parallel and it gives very good results (compared to other networks trained on the same dataset).