8 ms·

I've been following the RedPajama project closely and I must say, it's quite an impressive undertaking. The fact that it's all open-source, and the collaboratio

I've been following the RedPajama project closely and I must say, it's quite an impressive undertaking. The fact that it's all open-source, and the collaboration between various institutions, is nothing short of amazing. This shows the power of the open-source community in action, with a bunch of smart people coming together to build something truly remarkable. The 3B model, being super fast and accessible, is a game changer for a lot of us who may not have the latest hardware. I mean, running on an RTX 2070 that was released 5 years ago? That's pretty cool. As for the 7B model, it's great to see that it's already outperforming the Pythia 7B. The bigger dataset definitely seems to be making a difference here. I'm eager to see how far this project goes, and what kinda improvements we can expect in the coming weeks with the new RedPajama dataset they're working on. One thing I found interesting is the mention of differences between the LLaMA 7B and their replication. I'd love to learn more about those differences, as it could shed light on what's working well and what could be improved further.

SeanAnderson 3y ago
Sorry, excuse my ignorance, but why is having access to a 3B model a gamechanger? I played with a pirated 7B model a while back. My computer runs a 1080 TI - so it used to be good but now it's pretty old. The model ran with a reasonable number of tokens/sec, but the quality was just trash compared to what I'd grown used to with ChatGPT. It was a novelty I interacted with for just a single evening. I truly don't understand the use case for a 3B model with our current technologies. What are you going to use it for?
- acapybara 3y ago
  Hey SeanAnderson, good question! While parameter count is certainly an important factor in model performance, it's not the only one. The RedPajama project is taking a more nuanced approach to understanding what makes a model perform well, and their focus on smaller models like the 3B is a big part of that. Sure, you may have played with a 7B model in the past, but that doesn't mean there's no use case for a smaller model like the 3B. In fact, having a performant, smaller model is a game changer for a lot of applications that don't require the massive scale of the larger models. Plus, smaller models are generally faster and more accessible, which is always a plus.
  - hhh 3y ago
    is this comment generated by an LLM?
  - wokwokwok 3y ago
    > In fact, having a performant, smaller model is a game changer for a lot of applications that don't require the massive scale of the larger models. So we are all in agreement here that a 3B model is fundamentally inferior to a larger model? Not that it doesn’t have uses; not that there’s no value in research in small models. Just, honestly, that these smaller models don’t have the capabilities of the larger models. It’d be good to be a direct acknowledgment of that, because it seems like you’re going out of your way to promote the “it’s fine to have a small model”; and it is, roughly speaking. Parameter count isn’t everything. Small models are accessible, you can easily fine tune them. They are interesting. …but, they are not as good, as far as I’m aware, in terms of output, in terms of general purpose function, as larger models.
    - deepsquirrelnet 3y ago
      There is no “one size fits all” here. A bigger model is just a bigger hammer, that in many uses is too bulky and slow to be a proper solution. At my job, I can’t casually fire up 8xA100 80gb instances. And if I could, the performance wouldn’t have the throughput I require to be useful. Big models are operationally much more expensive. The smallest/fastest model that is accurate enough for your use case is ideal.
      - wokwokwok 3y ago
        > The smallest/fastest model that is accurate enough for your use case is ideal. Sure. …but it’s also fair to say that the smallest model that can fit your use case will be bounded by the parameter count. No amount of training data can make 100 param model do text summarisation. If you have a 3B param model, and you want a chat-GPT to embed in your app, do you think it’ll do? I don’t. The output is not at that quality level, because it’s too small. Not everyone needs that; but these 3B / 7B models don’t have the capability to do everything.
- youssefabdelm 3y ago
  Completely agree. Perhaps they were planning to fine-tune it for something though.
- examplary_cable 3y ago
  You can ultra fine tune those models ... look at vicune 13B, if you know how to prompt it well, you can get it to work as """"well"""" as ChatGPT. Running on local hardware .... I just got vicune 13b on gradio[1] to act as japanese kanji personal trainer, and I've only used a simple prompt: "I want you to act as a Japanese Kanji quiz machine. Each time I ask you for the next question, you are to provide one random Japanese kanji from JLPT N5 kanji list and ask for its meaning. You will generate four options, one correct, three wrong. The options will be labeled from A to D. I will reply to you with one letter, corresponding to one of these labels. You will evaluate my each answer based on your last question and tell me if I chose the right option. If I chose the right label, you will congratulate me. Otherwise you will tell me the right answer. Then you will ask me the next question. Avoid simple kanjis, let's go." [1] https://chat.lmsys.org/ https://chat.lmsys.org/
  - cced 3y ago
    How can someone get into using these models? How does ‘tuning’ work? How might I go about using these models for doing things like say summarizing news articles or video transcriptions? When someone tunes a model for a task, what exactly are they doing and how does this ‘change’ the model?
    - examplary_cable 3y ago
      (I'm not an expert) > How can someone get into using these models You can use gradio(online) or download(git will not download, it's too big, do it manually) the weights at https://huggingface.co/lmsys/vicuna-13b-delta-v1.1/tree/main https://huggingface.co/lmsys/vicuna-13b-delta-v1.1/tree/main and then load the model in pytourch and try inference(text generation). But you'll need either a lot of RAM(16GB,32GB+) or VRAM(Card). > How might I go about using these models for doing things like say summarizing news articles or video transcriptions Again, you might try online or setup a python/bash/powershell script to load the model for you so you can use it. If you can pay I would recommend runpod for the shared GPUs. > When someone tunes a model for a task, what exactly are they doing and how does this ‘change’ the model? From my view ... not much ... "fine-tuning" means training(tuning) on a specific dataset(fine, as in fine-grained). As I believe(I'm not sure) they just run more epochs on the model with the new data you have provided it until they reach a good loss(the model works), that's why quality data is important. You might try https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui they have a pretty easy setup config. Again, you'll need a lot of RAM and a good CPU for inference on CPU or a GPU. https://huggingface.co/lmsys/vicuna-13b-delta-v1.1/tree/main https://huggingface.co/lmsys/vicuna-13b-delta-v1.1/tree/main
- ttt3ts 3y ago
  Finetuning which can easily be done on consumer hardware and can give these models a lot more power for specific applications. Also, ChatGPT just can't do a lot of things because of their "rules". I was doing question answering about products on Amazon with ChatGPT and refused to answer any questions about underwear, certain books/videos, etc
- elorant 3y ago
  Depends on what you want it for. Chatting isn't the only application. For text summarization a model like Vicuna-13b has similar performance to ChatGPT 3.5. Fine-tuned models like the one in this thread might perform way better than the initial ones that leaked from Meta. The important thing is that there's constant progress in this area from the Open Source community and we're about to see amazing things in the future.
- barbariangrunge 3y ago
  I'm in the market for a laptop. If I was crazy and wanted to run or train models like these, what kind of resources would I need? Would the way the m2 MacBooks share memory be an advantage, or would the lack of cuda support be a killer? Can you do anything with 16GB, or do you need 128gb or something like that? How large are the datasets? I've only used scikit-learn and pandas so far, I'm not very familiar with neural networks yet
  - zamnos 3y ago
    It's not crazy to want to train or run models like these, it's actually quite popular right now! :) The question for you to answer is how handy with scikit-learn and pandas are you, and how much do you want to be on the bleeding edge of things? Most stuff is coming out for CUDA first, since that's what the industrial grade GPUS (A100s) use, so with Apple Arm you either have to wait for someone to port it, or port it yourself. On the other hand, getting > 8 GiB VRAM on a laptop GPU is rare; you're definitely not getting 128 GiB VRAM, so Apple Arm, with 32 or 64 GiB or RAM (get 128 if you can afford it) is going to get you more gigabytes of usable RAM for training/inference.
    - barbariangrunge 3y ago
      Yeah. It seems to me that it's really hard to get more than 10-14 GB of VRAM without using some sort of hyper expensive cluster. What would it cost if you wanted to do it with Nvidia? Being able to share ordinary ram with the GPU in a Mac could maybe be a unique value proposition
      - int_19h 3y ago
        RTX 3090 or 4090 gets you 24Gb of VRAM, which is enough to run llama-30b (quantized to 4-bit with groupsize of 1024 or higher) at speeds comparable to ChatGPT. You can also get two and run the model split across them, although pumping data back and forth slows things down. A brand new RTX A6000 (48Gb VRAM) is probably the largest you can get in a single card that can run in a regular PC. It can be had for $4-5k and is sufficient for llama-65b. Beyond that, yeah, you're looking at dedicated multi-GPU server hardware.
Sunhold 3y ago
Took me a bit to realize this comment was written by an LLM.
- awegio 3y ago
  How did you realize it here? This user has multiple comments in this thread but this one actually sounds more normal than the others. I find it very uncanny to see comments like this that sound like ChatGPT but are surprisingly relevant to the discussion.
  - zvolsky 3y ago
    It is the vacuous word fluff. My best guess is that it is a genuine human comment rephrased using a language model.
  - fphhotchips 3y ago
    I didn't realise it was written by an LLM but it did come off as weird to me because it borrows phrases (most obviously the bit about a "2070 released 5 years ago") from the press release itself.

SeanAnderson 3y ago

Sorry, excuse my ignorance, but why is having access to a 3B model a gamechanger? I played with a pirated 7B model a while back. My computer runs a 1080 TI - so it used to be good but now it's pretty old. The model ran with a reasonable number of tokens/sec, but the quality was just trash compared to what I'd grown used to with ChatGPT. It was a novelty I interacted with for just a single evening. I truly don't understand the use case for a 3B model with our current technologies. What are you going to use it for?

acapybara 3y ago
Hey SeanAnderson, good question! While parameter count is certainly an important factor in model performance, it's not the only one. The RedPajama project is taking a more nuanced approach to understanding what makes a model perform well, and their focus on smaller models like the 3B is a big part of that. Sure, you may have played with a 7B model in the past, but that doesn't mean there's no use case for a smaller model like the 3B. In fact, having a performant, smaller model is a game changer for a lot of applications that don't require the massive scale of the larger models. Plus, smaller models are generally faster and more accessible, which is always a plus.
- hhh 3y ago
  is this comment generated by an LLM?
- wokwokwok 3y ago
  > In fact, having a performant, smaller model is a game changer for a lot of applications that don't require the massive scale of the larger models. So we are all in agreement here that a 3B model is fundamentally inferior to a larger model? Not that it doesn’t have uses; not that there’s no value in research in small models. Just, honestly, that these smaller models don’t have the capabilities of the larger models. It’d be good to be a direct acknowledgment of that, because it seems like you’re going out of your way to promote the “it’s fine to have a small model”; and it is, roughly speaking. Parameter count isn’t everything. Small models are accessible, you can easily fine tune them. They are interesting. …but, they are not as good, as far as I’m aware, in terms of output, in terms of general purpose function, as larger models.
  - deepsquirrelnet 3y ago
    There is no “one size fits all” here. A bigger model is just a bigger hammer, that in many uses is too bulky and slow to be a proper solution. At my job, I can’t casually fire up 8xA100 80gb instances. And if I could, the performance wouldn’t have the throughput I require to be useful. Big models are operationally much more expensive. The smallest/fastest model that is accurate enough for your use case is ideal.
    - wokwokwok 3y ago
      > The smallest/fastest model that is accurate enough for your use case is ideal. Sure. …but it’s also fair to say that the smallest model that can fit your use case will be bounded by the parameter count. No amount of training data can make 100 param model do text summarisation. If you have a 3B param model, and you want a chat-GPT to embed in your app, do you think it’ll do? I don’t. The output is not at that quality level, because it’s too small. Not everyone needs that; but these 3B / 7B models don’t have the capability to do everything.
youssefabdelm 3y ago
Completely agree. Perhaps they were planning to fine-tune it for something though.
examplary_cable 3y ago
You can ultra fine tune those models ... look at vicune 13B, if you know how to prompt it well, you can get it to work as """"well"""" as ChatGPT. Running on local hardware .... I just got vicune 13b on gradio[1] to act as japanese kanji personal trainer, and I've only used a simple prompt: "I want you to act as a Japanese Kanji quiz machine. Each time I ask you for the next question, you are to provide one random Japanese kanji from JLPT N5 kanji list and ask for its meaning. You will generate four options, one correct, three wrong. The options will be labeled from A to D. I will reply to you with one letter, corresponding to one of these labels. You will evaluate my each answer based on your last question and tell me if I chose the right option. If I chose the right label, you will congratulate me. Otherwise you will tell me the right answer. Then you will ask me the next question. Avoid simple kanjis, let's go." [1] https://chat.lmsys.org/ https://chat.lmsys.org/
- cced 3y ago
  How can someone get into using these models? How does ‘tuning’ work? How might I go about using these models for doing things like say summarizing news articles or video transcriptions? When someone tunes a model for a task, what exactly are they doing and how does this ‘change’ the model?
  - examplary_cable 3y ago
    (I'm not an expert) > How can someone get into using these models You can use gradio(online) or download(git will not download, it's too big, do it manually) the weights at https://huggingface.co/lmsys/vicuna-13b-delta-v1.1/tree/main https://huggingface.co/lmsys/vicuna-13b-delta-v1.1/tree/main and then load the model in pytourch and try inference(text generation). But you'll need either a lot of RAM(16GB,32GB+) or VRAM(Card). > How might I go about using these models for doing things like say summarizing news articles or video transcriptions Again, you might try online or setup a python/bash/powershell script to load the model for you so you can use it. If you can pay I would recommend runpod for the shared GPUs. > When someone tunes a model for a task, what exactly are they doing and how does this ‘change’ the model? From my view ... not much ... "fine-tuning" means training(tuning) on a specific dataset(fine, as in fine-grained). As I believe(I'm not sure) they just run more epochs on the model with the new data you have provided it until they reach a good loss(the model works), that's why quality data is important. You might try https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui they have a pretty easy setup config. Again, you'll need a lot of RAM and a good CPU for inference on CPU or a GPU. https://huggingface.co/lmsys/vicuna-13b-delta-v1.1/tree/main https://huggingface.co/lmsys/vicuna-13b-delta-v1.1/tree/main
ttt3ts 3y ago
Finetuning which can easily be done on consumer hardware and can give these models a lot more power for specific applications. Also, ChatGPT just can't do a lot of things because of their "rules". I was doing question answering about products on Amazon with ChatGPT and refused to answer any questions about underwear, certain books/videos, etc
elorant 3y ago
Depends on what you want it for. Chatting isn't the only application. For text summarization a model like Vicuna-13b has similar performance to ChatGPT 3.5. Fine-tuned models like the one in this thread might perform way better than the initial ones that leaked from Meta. The important thing is that there's constant progress in this area from the Open Source community and we're about to see amazing things in the future.
barbariangrunge 3y ago
I'm in the market for a laptop. If I was crazy and wanted to run or train models like these, what kind of resources would I need? Would the way the m2 MacBooks share memory be an advantage, or would the lack of cuda support be a killer? Can you do anything with 16GB, or do you need 128gb or something like that? How large are the datasets? I've only used scikit-learn and pandas so far, I'm not very familiar with neural networks yet
- zamnos 3y ago
  It's not crazy to want to train or run models like these, it's actually quite popular right now! :) The question for you to answer is how handy with scikit-learn and pandas are you, and how much do you want to be on the bleeding edge of things? Most stuff is coming out for CUDA first, since that's what the industrial grade GPUS (A100s) use, so with Apple Arm you either have to wait for someone to port it, or port it yourself. On the other hand, getting > 8 GiB VRAM on a laptop GPU is rare; you're definitely not getting 128 GiB VRAM, so Apple Arm, with 32 or 64 GiB or RAM (get 128 if you can afford it) is going to get you more gigabytes of usable RAM for training/inference.
  - barbariangrunge 3y ago
    Yeah. It seems to me that it's really hard to get more than 10-14 GB of VRAM without using some sort of hyper expensive cluster. What would it cost if you wanted to do it with Nvidia? Being able to share ordinary ram with the GPU in a Mac could maybe be a unique value proposition
    - int_19h 3y ago
      RTX 3090 or 4090 gets you 24Gb of VRAM, which is enough to run llama-30b (quantized to 4-bit with groupsize of 1024 or higher) at speeds comparable to ChatGPT. You can also get two and run the model split across them, although pumping data back and forth slows things down. A brand new RTX A6000 (48Gb VRAM) is probably the largest you can get in a single card that can run in a regular PC. It can be had for $4-5k and is sufficient for llama-65b. Beyond that, yeah, you're looking at dedicated multi-GPU server hardware.

Sunhold 3y ago

Took me a bit to realize this comment was written by an LLM.

awegio 3y ago
How did you realize it here? This user has multiple comments in this thread but this one actually sounds more normal than the others. I find it very uncanny to see comments like this that sound like ChatGPT but are surprisingly relevant to the discussion.
- zvolsky 3y ago
  It is the vacuous word fluff. My best guess is that it is a genuine human comment rephrased using a language model.
- fphhotchips 3y ago
  I didn't realise it was written by an LLM but it did come off as weird to me because it borrows phrases (most obviously the bit about a "2070 released 5 years ago") from the press release itself.