7 ms·

I have used their gemma 4 31b model through kagi and getting real instantaneous answers is absolutely crazy. A very different feeling and UX. Even if the model

I have used their gemma 4 31b model through kagi and getting real instantaneous answers is absolutely crazy. A very different feeling and UX. Even if the model is smaller, there is definitely a use case for these. I was wondering if they would put the qwen 27b model, it sounds very interesting to try.

bitexploder 13d ago
The thing I didn’t realize for a while is 27B is rather smart. As many (or more) activated parameters as the flash models of the universe that we know about. It reasons very well. It just doesn’t have a lot of knowledge.
- nicce 13d ago
  They seem to have good enough general intelligence that missing knowledge is not that big thing. If you are able to have a proper [free search engine], they can do almost anything. Having own local search index about relevant stuff can help a lof if you don’t want to pay for search API.
  - bitexploder 13d ago
    But running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)
    - nicce 13d ago
      You can run it with 2x r9700 with 150-200 tokens per second. It is intelligent enough if you just point the docs / whatever for it.
      - bitexploder 12d ago
        I believe. I run it on my mac M5 pro at like 30t/s with some RAGs and let it work on stuff overnight and it's great. It isn't the same as the big models where things can be more unbounded, but if local models keep progressing there is a universe where a 200-300B model is all most of us will need to stay out of the big tech moats.
codazoda 13d ago
Really an aside, but yesterday I got the Gemma-4-12b (128k context) to build it's first web app in the minimal Dark Software Factory I've been building for myself. https://joeldare.com/a-local-open-weight-model-builds-its-first-web-app https://joeldare.com/a-local-open-weight-model-builds-its-fi...
blaesus 13d ago
May I ask what you used Gemma 31B for? Last time I try it wasn't bad but then it wasn't particularly good either.
- freehorse 12d ago
  It has been used in kagi's "quick assistant", so basically summarisation tasks (gets context from a bunch of webpages and finds the answer to a question in them, spits all the relevant to the query information, or similar). It is great UX when you are in a search results page, but I don't use it in the assistant directly because usually this kind of speed is less relevant there.

bitexploder 13d ago

The thing I didn’t realize for a while is 27B is rather smart. As many (or more) activated parameters as the flash models of the universe that we know about. It reasons very well. It just doesn’t have a lot of knowledge.

nicce 13d ago
They seem to have good enough general intelligence that missing knowledge is not that big thing. If you are able to have a proper [free search engine], they can do almost anything. Having own local search index about relevant stuff can help a lof if you don’t want to pay for search API.
- bitexploder 13d ago
  But running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)
  - nicce 13d ago
    You can run it with 2x r9700 with 150-200 tokens per second. It is intelligent enough if you just point the docs / whatever for it.
    - bitexploder 12d ago
      I believe. I run it on my mac M5 pro at like 30t/s with some RAGs and let it work on stuff overnight and it's great. It isn't the same as the big models where things can be more unbounded, but if local models keep progressing there is a universe where a 200-300B model is all most of us will need to stay out of the big tech moats.

codazoda 13d ago

Really an aside, but yesterday I got the Gemma-4-12b (128k context) to build it's first web app in the minimal Dark Software Factory I've been building for myself. https://joeldare.com/a-local-open-weight-model-builds-its-first-web-app https://joeldare.com/a-local-open-weight-model-builds-its-fi...

blaesus 13d ago

May I ask what you used Gemma 31B for? Last time I try it wasn't bad but then it wasn't particularly good either.

freehorse 12d ago
It has been used in kagi's "quick assistant", so basically summarisation tasks (gets context from a bunch of webpages and finds the answer to a question in them, spits all the relevant to the query information, or similar). It is great UX when you are in a search results page, but I don't use it in the assistant directly because usually this kind of speed is less relevant there.