6 ms·

Understanding, using, and finetuning Gemma

118 points by rasbt 3y ago

behnamoh 3y ago
Gemma (and Gemini) are heavily nerfed. Why are they on the news lately? Also, Gemma is a +9B model. I think it's not okay that Google compared it with Mistral and Llama 2 (7B) models. Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool. All this hype seems to be backed by Google to boost their models whereas in practice, the models are not that good. Google also made a big claim about Gemini 1.5 1M context window, but at the end of their article they said they'll limit it to 128K. So all that 1M flex was for nothing? Not to mention their absurd approach in alignment in image creation.
- andy99 3y ago
  Gemma has a 7B parameters model https://huggingface.co/google/gemma-7b https://huggingface.co/google/gemma-7b that's what I saw compared to Mistral (Edit: I'm wrong)
  - light_hue_1 3y ago
    No it doesn't. Gemma 7B is a 9B model. The name is a lie. Then they really played games with Gemma 2B as well. I don't get how Google can be this incompetent and far behind everyone else. They have amazing people and the kinds of resources that almost no one else does but somehow need to resort to faking demos, blatant lies about model sizes, etc. Google used to be the place everyone wanted to go. Someone at Google AI needs to be fired so they can start being productive again.
    - andy99 3y ago
      Haha really? I almost added the caveat that I didn't count the parameters myself. And I couldn't see the weights file size because it requires login (because of their restrictive licensing choice). If true and it's 9B that's really dishonest.
      - rasbt 3y ago
        Yes, it's 8.5B params if you account for weight tying, and 9.3B if you count the embedding layer and output layer weights separately as shown in the 2nd figure in the article. In the paper, I think they justified 7B by only counting the non-embedding parameters (7,751,248,896), which is kind of cheating in my opinion, because if you do that, then Llama 2 is basically a 5B-6B param model.
        andy99 3y ago
        Is the 2B measured like that as well? I did use it with llama.cpp and noticed it ran slower than I expected. That's the danger of too much abstraction, it's easy to have big gaps in one's understanding of what's really going on.
        rasbt 3y ago
        Yes, it's somewhat similar to the 2B model as it uses the same vocabulary size.
      - brucethemoose2 3y ago
        Practically, speaking, I OOM'd running Gemma on a 3090 using a config that had VRAM to spare for Mistral 7B. It kinda surprised me at first, until I realized why.
    - xfalcox 3y ago
      > Gemma 7B is a 9B model. The name is a lie Ohhh so that explains why I couldn't load it on my RTX 4090, while other 7B models load just fine!
- gliptic 3y ago
  > Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool. Are you talking about gemma.cpp? Then no, they didn't.
  - andy99 3y ago
    Presumably he means this https://cloud.google.com/blog/products/application-development/new-localllm-lets-you-develop-gen-ai-apps-locally-without-gpus https://cloud.google.com/blog/products/application-developme... The claim is correct but not related to gemma
    - behnamoh 3y ago
      Yes, I was talking about this.
    - gliptic 3y ago
      From the repo: > Run LLMs locally on Cloud Workstations. Uses: > Quantized models from [Huggingface] > llama-cpp-python's webserver But sure, the blog post doesn't mention it.
- rasbt 3y ago
  > Gemma is a +9B model Yes that's correct. It's 9.3B parameters if you count the embedding layer and final projection layer separately. However, since they used weight tying, the adjusted count is 8.5B as discussed in the article.
  - neodymiumphish 3y ago
    Which still rounds to 9B and is 21.4% larger.
    - rasbt 3y ago
      Yes, it's definitely unfair to count it as a 7B model. In that case, we could call Llama 2, which is 6.6B parameters, a 6B (or even 5B) parameter model.
      - neodymiumphish 3y ago
        Except 6.6 rounds to 7. That’s completely reasonable. Arguing otherwise is pedantic.
- cyanydeez 3y ago
  the context window is entirely limited by VRAM size do you even LLM?
- brucethemoose2 3y ago
  Counterpoints: - Local models are pretty easy to de-censor, if thats what you mean. - ...Yeah, it should not be labeled as a 7B. Its sort of 7B class. - The repo mentions they use the llama-cpp-python server - 1M context brute forced across TPUs is insanely expensive, I can see why Google reigned it in. But overall your message is not wrong. Google is hyping Gemma a ton when its... Well, not very remarkable. And they could have certainly made something niche and interesting, like a long context 8.5B model, a specialized model, a vastly more multilingual model, something to differentiate it from Mistral 7B 0.2
- sivakon 3y ago
  It’s objectively worse in my local tests compared to Mistral. Again their model doesn’t include MT-bench benchmark because it’s really really bad at answering a follow up question(s). (this is also a problem in Ultra). It’s reasoning is also pretty bad compared to mistral.
  - b33j0r 3y ago
    I can’t get it to recognize the stop token consistently in the 7b models. About 50% of the shots, I get a sentence and a half of beautiful poetry, then a codeswitch into kanji, and then ral ral ral ral ral 膳 ral 杯 ral ral Until I kill the process. Not every time, but way more often than the other llamas (which is basically never, these days). I think they underestimated the impact of training on bulleted lists. It seems to love those!
- verticalscaler 3y ago
  [flagged]
- htrp 3y ago
  > Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool. They said it was inspired by llama >This is inspired by vertically-integrated model implementations such as ggml, llama.c, and llama.rs. from https://github.com/google/gemma.cpp https://github.com/google/gemma.cpp
  - pests 3y ago
    Not gemma.cpp He meant this: https://cloud.google.com/blog/products/application-development/new-localllm-lets-you-develop-gen-ai-apps-locally-without-gpus https://cloud.google.com/blog/products/application-developme...
- d-z-m 3y ago
  > Also, Gemma is a +9B model. I think it's not okay that Google compared it with Mistral and Llama 2 (7B) models. They say it's because they're not counting embedding parameters[0]. Although apparently even with the embedding parameters subtracted it still rounds to 8B not 7B. From what understand, rounding to the nearest B is the standard. Seems slightly disingenuous to call it 7B, but not a big deal IMO since I don't hear anyone saying this model is outperforming popular OSS 7Bs. [0]: https://huggingface.co/google/gemma-7b/discussions/34 https://huggingface.co/google/gemma-7b/discussions/34
- verticalscaler 3y ago
  [flagged]
brucethemoose2 3y ago
What are HNers looking for in this article? The architectural differences, or how to run/finetune it?
- deleted 3y ago
  [deleted]
- BryanLegend 3y ago
  I was pleased to see the architectural differences highlighted. It's insightful to see how they're evolving.
Solvency 3y ago
Can we just stop talking about Gemini/Gemma for at least two years before it's improved? In fact, the two-year mark is rather strategic recommendation, because I guarantee it'll become vaporware by then anyway with Google's track record. It's outrageously poorly performing.
- breezeTrowel 3y ago
  How can it be vaporware if it's already been released?
  - nurettin 3y ago
    It was "released", like Tesla's fully automated self-driving was "released".
    - relativ575 3y ago
      What makes this a "release" compared to the release of non-Google models?
      - nurettin 3y ago
        It under-delivers and over-promises.
  - Solvency 3y ago
    Pardon, I should have said "vaporizedware".
brunooliv 3y ago
Anyone who uses these models for more than 10 min will immediately realize that they're really, really bad compared to other free, OSS models. Even Phi-2 was giving me "on par" results except that its a model of a different league. Many models are being released now, which is good to keep OpenAI on their toes and not mess up, but, truth be told, I've yet to see _any_ OSS model that I can run on my machine being as good as ChatGPT 3 (not 3.5, not 4, but the original one from when everyone went crazy). My hopes for consumer hardware ChatGPT-3.5 within 2024 probably lie with what Meta will keep building upon. Google was great, once. Now, they're a mere bystander in the larger scheme of things. I think that's a good thing. Everything in the world is cyclic and ephemeral and Google enjoyed their time while it lasted, but, newer and better things are and will, keep on coming. PS: Completely unrelated, but, gmail is now the only Google product I actively use. I don't, genuinely, remember the last time I did a Google Search... When I need to do my own digging I use Phind these days. Times are changing and that's great for tech and future generations joining the field and workforce!
- bradley13 3y ago
  My initial impressions of mixtral (not mistral) are quite good. It runs fairly well on my PC, using ollama.
- CuriouslyC 3y ago
  If Mixtral isn't outperforming chatgpt 3 you're configuring it wrong. It gives somewhat terse answers by default, but you can prompt it to spit out wordy answers of the sort chatgpt has been aligned to prefer easily enough.
  - brunooliv 3y ago
    Mixtral aka the 8x7B the "sparse mixture of experts" one is not the same as, eg. Mistral-7B which is still very, very good, just not quite hitting the mark on some things. I still couldn't run Mixtral 8x7B on an M1 Macbook Pro with 32Gb ram, so maybe I am indeed doing it wrong? Or are there better quantized versions available now or..?
lopkeny12ko 3y ago
Gemma, despite being developed by a company worth billions of dollars, is a phenomonally poor model. I tried the open source release yesterday. I started with the input string "hello" and it responded "I am a new user to this forum and I am looking for 100000000000000..." with zeros repeating forever. Ok, cool I guess. Looks like I'll be sticking with GPT-4.
- fsmv 3y ago
  Did you use the raw model or the instruction tuned one? 2B or 7B? You didn't give it much to go on.
- SunlitCat 3y ago
  The Mistral model I tried when it came out produced "blog posts" as responses. I assume this somehow depends on where those models get much of their training data from (please correct me if I'm wrong).

118 points by rasbt 3y ago

Gemma (and Gemini) are heavily nerfed. Why are they on the news lately? Also, Gemma is a +9B model. I think it's not okay that Google compared it with Mistral and Llama 2 (7B) models. Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool. All this hype seems to be backed by Google to boost their models whereas in practice, the models are not that good. Google also made a big claim about Gemini 1.5 1M context window, but at the end of their article they said they'll limit it to 128K. So all that 1M flex was for nothing? Not to mention their absurd approach in alignment in image creation.

andy99 3y ago
Gemma has a 7B parameters model https://huggingface.co/google/gemma-7b https://huggingface.co/google/gemma-7b that's what I saw compared to Mistral (Edit: I'm wrong)
- light_hue_1 3y ago
  No it doesn't. Gemma 7B is a 9B model. The name is a lie. Then they really played games with Gemma 2B as well. I don't get how Google can be this incompetent and far behind everyone else. They have amazing people and the kinds of resources that almost no one else does but somehow need to resort to faking demos, blatant lies about model sizes, etc. Google used to be the place everyone wanted to go. Someone at Google AI needs to be fired so they can start being productive again.
  - andy99 3y ago
    Haha really? I almost added the caveat that I didn't count the parameters myself. And I couldn't see the weights file size because it requires login (because of their restrictive licensing choice). If true and it's 9B that's really dishonest.
    - rasbt 3y ago
      Yes, it's 8.5B params if you account for weight tying, and 9.3B if you count the embedding layer and output layer weights separately as shown in the 2nd figure in the article. In the paper, I think they justified 7B by only counting the non-embedding parameters (7,751,248,896), which is kind of cheating in my opinion, because if you do that, then Llama 2 is basically a 5B-6B param model.
      - andy99 3y ago
        Is the 2B measured like that as well? I did use it with llama.cpp and noticed it ran slower than I expected. That's the danger of too much abstraction, it's easy to have big gaps in one's understanding of what's really going on.
        rasbt 3y ago
        Yes, it's somewhat similar to the 2B model as it uses the same vocabulary size.
    - brucethemoose2 3y ago
      Practically, speaking, I OOM'd running Gemma on a 3090 using a config that had VRAM to spare for Mistral 7B. It kinda surprised me at first, until I realized why.
  - xfalcox 3y ago
    > Gemma 7B is a 9B model. The name is a lie Ohhh so that explains why I couldn't load it on my RTX 4090, while other 7B models load just fine!
gliptic 3y ago
> Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool. Are you talking about gemma.cpp? Then no, they didn't.
- andy99 3y ago
  Presumably he means this https://cloud.google.com/blog/products/application-development/new-localllm-lets-you-develop-gen-ai-apps-locally-without-gpus https://cloud.google.com/blog/products/application-developme... The claim is correct but not related to gemma
  - behnamoh 3y ago
    Yes, I was talking about this.
  - gliptic 3y ago
    From the repo: > Run LLMs locally on Cloud Workstations. Uses: > Quantized models from [Huggingface] > llama-cpp-python's webserver But sure, the blog post doesn't mention it.
rasbt 3y ago
> Gemma is a +9B model Yes that's correct. It's 9.3B parameters if you count the embedding layer and final projection layer separately. However, since they used weight tying, the adjusted count is 8.5B as discussed in the article.
- neodymiumphish 3y ago
  Which still rounds to 9B and is 21.4% larger.
  - rasbt 3y ago
    Yes, it's definitely unfair to count it as a 7B model. In that case, we could call Llama 2, which is 6.6B parameters, a 6B (or even 5B) parameter model.
    - neodymiumphish 3y ago
      Except 6.6 rounds to 7. That’s completely reasonable. Arguing otherwise is pedantic.
cyanydeez 3y ago
the context window is entirely limited by VRAM size do you even LLM?
brucethemoose2 3y ago
Counterpoints: - Local models are pretty easy to de-censor, if thats what you mean. - ...Yeah, it should not be labeled as a 7B. Its sort of 7B class. - The repo mentions they use the llama-cpp-python server - 1M context brute forced across TPUs is insanely expensive, I can see why Google reigned it in. But overall your message is not wrong. Google is hyping Gemma a ton when its... Well, not very remarkable. And they could have certainly made something niche and interesting, like a long context 8.5B model, a specialized model, a vastly more multilingual model, something to differentiate it from Mistral 7B 0.2
sivakon 3y ago
It’s objectively worse in my local tests compared to Mistral. Again their model doesn’t include MT-bench benchmark because it’s really really bad at answering a follow up question(s). (this is also a problem in Ultra). It’s reasoning is also pretty bad compared to mistral.
- b33j0r 3y ago
  I can’t get it to recognize the stop token consistently in the 7b models. About 50% of the shots, I get a sentence and a half of beautiful poetry, then a codeswitch into kanji, and then ral ral ral ral ral 膳 ral 杯 ral ral Until I kill the process. Not every time, but way more often than the other llamas (which is basically never, these days). I think they underestimated the impact of training on bulleted lists. It seems to love those!
verticalscaler 3y ago
[flagged]
htrp 3y ago
> Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool. They said it was inspired by llama >This is inspired by vertically-integrated model implementations such as ggml, llama.c, and llama.rs. from https://github.com/google/gemma.cpp https://github.com/google/gemma.cpp
- pests 3y ago
  Not gemma.cpp He meant this: https://cloud.google.com/blog/products/application-development/new-localllm-lets-you-develop-gen-ai-apps-locally-without-gpus https://cloud.google.com/blog/products/application-developme...
d-z-m 3y ago
> Also, Gemma is a +9B model. I think it's not okay that Google compared it with Mistral and Llama 2 (7B) models. They say it's because they're not counting embedding parameters[0]. Although apparently even with the embedding parameters subtracted it still rounds to 8B not 7B. From what understand, rounding to the nearest B is the standard. Seems slightly disingenuous to call it 7B, but not a big deal IMO since I don't hear anyone saying this model is outperforming popular OSS 7Bs. [0]: https://huggingface.co/google/gemma-7b/discussions/34 https://huggingface.co/google/gemma-7b/discussions/34
verticalscaler 3y ago
[flagged]

brucethemoose2 3y ago

What are HNers looking for in this article? The architectural differences, or how to run/finetune it?

deleted 3y ago
[deleted]
BryanLegend 3y ago
I was pleased to see the architectural differences highlighted. It's insightful to see how they're evolving.

Solvency 3y ago

Can we just stop talking about Gemini/Gemma for at least two years before it's improved? In fact, the two-year mark is rather strategic recommendation, because I guarantee it'll become vaporware by then anyway with Google's track record. It's outrageously poorly performing.

breezeTrowel 3y ago
How can it be vaporware if it's already been released?
- nurettin 3y ago
  It was "released", like Tesla's fully automated self-driving was "released".
  - relativ575 3y ago
    What makes this a "release" compared to the release of non-Google models?
    - nurettin 3y ago
      It under-delivers and over-promises.
- Solvency 3y ago
  Pardon, I should have said "vaporizedware".

brunooliv 3y ago

Anyone who uses these models for more than 10 min will immediately realize that they're really, really bad compared to other free, OSS models. Even Phi-2 was giving me "on par" results except that its a model of a different league. Many models are being released now, which is good to keep OpenAI on their toes and not mess up, but, truth be told, I've yet to see _any_ OSS model that I can run on my machine being as good as ChatGPT 3 (not 3.5, not 4, but the original one from when everyone went crazy). My hopes for consumer hardware ChatGPT-3.5 within 2024 probably lie with what Meta will keep building upon. Google was great, once. Now, they're a mere bystander in the larger scheme of things. I think that's a good thing. Everything in the world is cyclic and ephemeral and Google enjoyed their time while it lasted, but, newer and better things are and will, keep on coming. PS: Completely unrelated, but, gmail is now the only Google product I actively use. I don't, genuinely, remember the last time I did a Google Search... When I need to do my own digging I use Phind these days. Times are changing and that's great for tech and future generations joining the field and workforce!

bradley13 3y ago
My initial impressions of mixtral (not mistral) are quite good. It runs fairly well on my PC, using ollama.
CuriouslyC 3y ago
If Mixtral isn't outperforming chatgpt 3 you're configuring it wrong. It gives somewhat terse answers by default, but you can prompt it to spit out wordy answers of the sort chatgpt has been aligned to prefer easily enough.
- brunooliv 3y ago
  Mixtral aka the 8x7B the "sparse mixture of experts" one is not the same as, eg. Mistral-7B which is still very, very good, just not quite hitting the mark on some things. I still couldn't run Mixtral 8x7B on an M1 Macbook Pro with 32Gb ram, so maybe I am indeed doing it wrong? Or are there better quantized versions available now or..?

lopkeny12ko 3y ago

Gemma, despite being developed by a company worth billions of dollars, is a phenomonally poor model. I tried the open source release yesterday. I started with the input string "hello" and it responded "I am a new user to this forum and I am looking for 100000000000000..." with zeros repeating forever. Ok, cool I guess. Looks like I'll be sticking with GPT-4.

fsmv 3y ago
Did you use the raw model or the instruction tuned one? 2B or 7B? You didn't give it much to go on.
SunlitCat 3y ago
The Mistral model I tried when it came out produced "blog posts" as responses. I assume this somehow depends on where those models get much of their training data from (please correct me if I'm wrong).