7 ms·

Oh yes, I am firmly on Team China here because US companies got too greedy. Meta is an exception here though and they also propelled AI development massively.

by raxxor 2y ago

Oh yes, I am firmly on Team China here because US companies got too greedy. Meta is an exception here though and they also propelled AI development massively. DeepSeek is awesome. Any AI task yet implemented in our business can be run from my local PC with just the smaller models. And my PC is fairly crappy to begin with. OpenAI looks quite silly with their "we have to close everything".

manmal 2y ago
Can you elaborate which models you are using? I‘m running an R1 distilled Qwen coder with 32B Q4, and while it’s giving useful answers, it‘s quite slow on my M1 Max. Slow enough that I keep reaching for cloud models.
- raxxor 2y ago
  Not on my machine currently, I use the 14b Q4 model I think, which delivers very good answers. I run a 4060 with 16gb memory and performance is quite good. I used the largest model that was recommended with this amount of VRAM, I think it was the 14b one. I do have some applications that process images, text and pdf files and I use smaller models for extracting embeddings. I think my system wouldn't be able to handle it with decent speed otherwise. I do run LLM on a M1 16gb macbook air and performance is surprisingly good. Not for image synthesis though and a PC with a dedicated GPU is still significantly faster with LLM responses as well. Haven't tried to run deepseek on the macbook yet.
  - manmal 2y ago
    Interesting, I didn’t like the quality of the output of the 14B models. Could be the quantization though, apparently some are a bit broken.
blackeyeblitzar 2y ago
The US companies got too greedy? How? They invented this entire space, literally. DeepSeek built their base models off Llama releases and OpenAI outputs (or so it’s thought), and while they added some optimizations on top, it seems like they’ve lied about the costs to produce their models by simply being vague about their base model and training data, and quoting the cost of their final training run. And then there’s all the dystopian propaganda baked into these models, which threatens to misinform users at scale based on a government driven agenda. Hard to be on that team, let alone firmly, knowing that it’s giving power to a dictatorial regime.
- wkat4242 2y ago
  The US models are also full of censorship. For example the US is much more sensitive to anything related to sexuality and here in Europe it's quite frustrating to deal with that censorship.
  - infecto 2y ago
    I think we will find that each region will have their own flair of censorship. The only reason it stands out more from a Chinese perspective is the requirement to have alignment with PRC/CCP rhetoric.
    - wkat4242 2y ago
      Yes that's what I mean. I wish all models were uncensored and it would just be up to the implementer to decide how to finetune on top of that. Save for the super crazy stuff of course.
- timeon 2y ago
  > DeepSeek built their base models off Llama releases and OpenAI outputs Those models are also trained on data that was ignoring licenses / copyrighted content.
  - ithkuil 2y ago
    That's a problem for sure. But why would that argument play in favour china which would be even less constrained by licenses / copyright ?
    - timeon 2y ago
      I didn't mean it as favour for China. Just that this industry is unfortunately like that.
- raxxor 2y ago
  That was perhaps a bit too general, but aside from meta and Google they didn't share their research and tried to sell AI products as fast as possible and tried to lobby legislation to keep their head start. I would also include nvidia here, that has some moat through software integrations. I haven't tested deepseek for censorship yet, but they shared their release and even their input data. And in this case you could correct its shortcomings, so propaganda would be difficult.
nejsjsjsbsb 2y ago
I'm on team open source. To me the exciting thing was ollama downloading the 7B and running it on a 5yo cheap lonovo and getting a token rate similar to the first release of ChatGPT. Running local on CPU opens so much possibilities for smart and privacy focused home devices that serve you. In my test it hallucinated confidently but my interest is in simple second brain like rag. "Hey thingy, what is my schedule today?" Need it to be a bit faster though as the thinking part adds a lot of latency.
- raxxor 2y ago
  The thinking is quite fascinating though, I love reading it. Especially when it notices something must be wrong. It will probably be very helpful to refine answer for itself and other models. It does add latency of course, but I still think that I could provide all AI needs of my company (industrial production) with a simple older off the shelf PC. My GPU is decently recent, but the smallest model of the series and otherwise the machine is a rusty bucket. I didn't test it thoroughly yet, but I have some invoices where I need to extract info and it did a perfect job until now. But I don't think there is any LLM yet that can do that without someone checking the output.

by raxxor 2y ago

manmal 2y ago

Can you elaborate which models you are using? I‘m running an R1 distilled Qwen coder with 32B Q4, and while it’s giving useful answers, it‘s quite slow on my M1 Max. Slow enough that I keep reaching for cloud models.

raxxor 2y ago
Not on my machine currently, I use the 14b Q4 model I think, which delivers very good answers. I run a 4060 with 16gb memory and performance is quite good. I used the largest model that was recommended with this amount of VRAM, I think it was the 14b one. I do have some applications that process images, text and pdf files and I use smaller models for extracting embeddings. I think my system wouldn't be able to handle it with decent speed otherwise. I do run LLM on a M1 16gb macbook air and performance is surprisingly good. Not for image synthesis though and a PC with a dedicated GPU is still significantly faster with LLM responses as well. Haven't tried to run deepseek on the macbook yet.
- manmal 2y ago
  Interesting, I didn’t like the quality of the output of the 14B models. Could be the quantization though, apparently some are a bit broken.

blackeyeblitzar 2y ago

The US companies got too greedy? How? They invented this entire space, literally. DeepSeek built their base models off Llama releases and OpenAI outputs (or so it’s thought), and while they added some optimizations on top, it seems like they’ve lied about the costs to produce their models by simply being vague about their base model and training data, and quoting the cost of their final training run. And then there’s all the dystopian propaganda baked into these models, which threatens to misinform users at scale based on a government driven agenda. Hard to be on that team, let alone firmly, knowing that it’s giving power to a dictatorial regime.

wkat4242 2y ago
The US models are also full of censorship. For example the US is much more sensitive to anything related to sexuality and here in Europe it's quite frustrating to deal with that censorship.
- infecto 2y ago
  I think we will find that each region will have their own flair of censorship. The only reason it stands out more from a Chinese perspective is the requirement to have alignment with PRC/CCP rhetoric.
  - wkat4242 2y ago
    Yes that's what I mean. I wish all models were uncensored and it would just be up to the implementer to decide how to finetune on top of that. Save for the super crazy stuff of course.
timeon 2y ago
> DeepSeek built their base models off Llama releases and OpenAI outputs Those models are also trained on data that was ignoring licenses / copyrighted content.
- ithkuil 2y ago
  That's a problem for sure. But why would that argument play in favour china which would be even less constrained by licenses / copyright ?
  - timeon 2y ago
    I didn't mean it as favour for China. Just that this industry is unfortunately like that.
raxxor 2y ago
That was perhaps a bit too general, but aside from meta and Google they didn't share their research and tried to sell AI products as fast as possible and tried to lobby legislation to keep their head start. I would also include nvidia here, that has some moat through software integrations. I haven't tested deepseek for censorship yet, but they shared their release and even their input data. And in this case you could correct its shortcomings, so propaganda would be difficult.

nejsjsjsbsb 2y ago

I'm on team open source. To me the exciting thing was ollama downloading the 7B and running it on a 5yo cheap lonovo and getting a token rate similar to the first release of ChatGPT. Running local on CPU opens so much possibilities for smart and privacy focused home devices that serve you. In my test it hallucinated confidently but my interest is in simple second brain like rag. "Hey thingy, what is my schedule today?" Need it to be a bit faster though as the thinking part adds a lot of latency.

raxxor 2y ago
The thinking is quite fascinating though, I love reading it. Especially when it notices something must be wrong. It will probably be very helpful to refine answer for itself and other models. It does add latency of course, but I still think that I could provide all AI needs of my company (industrial production) with a simple older off the shelf PC. My GPU is decently recent, but the smallest model of the series and otherwise the machine is a rusty bucket. I didn't test it thoroughly yet, but I have some invoices where I need to extract info and it did a perfect job until now. But I don't think there is any LLM yet that can do that without someone checking the output.