9 ms·

I think this is a game changer, because data privacy is a legitimate concern for many enterprise users. Btw, you can also run Mistral locally within the Docker

I think this is a game changer, because data privacy is a legitimate concern for many enterprise users. Btw, you can also run Mistral locally within the Docker model runner on a Mac.

v3ss0n 1y ago
What's the point when we can run much powerful models now? Qwen3 , Deepseek
- _bin_ 1y ago
  It would be short-termist for Americans or euros to use chinese-made models. Increasing their popularity has an indirect but significant cost in the long term. china "winning AI" should be an unacceptable outcome for America or europe by any means necessary.
  - atwrk 1y ago
    Why would that be? I can see why Americans wouldn't want to do that, but Europeans? In the current political climate, where the US openly claims their desire to annex European territory and so on? I'd rather see them prefer a locally hostable open source solution like DeepSeek.
    - tigroferoce 1y ago
      My two cents, as European, is that since we are more and more asking to LLMs for information, it wouldn't be wise to let a foreign country, not even truly democratic, to choose the information we get.
      - jamesblonde 1y ago
        The Chinese don't get any of information if we use self-hosted DeepSeek or Qwen. They are open-source. You can run them in an air-gapped environment that can't phone home.
        fennecbutt 1y ago
        But their models are gimped by bad censoring. At least I can still ask chatgpt how many innocent civilians America has bombed.
kergonath 1y ago
> I think this is a game changer, because data privacy is a legitimate concern for many enterprise users. Indeed. At work, we are experimenting with this. Using a cloud platform is a non-starter for data confidentiality reasons. On-premise is the way to go. Also, they’re not American, which helps. > Btw, you can also run Mistral locally within the Docker model runner on a Mac. True, but you can do that only with their open-weight models, right? They are very useful and work well, but their commercial models are bigger and hopefully better (I use some of their free models every day, but none of their commercial ones).
- distances 1y ago
  I also kind of don't understand how it seems everyone is using AI for coding. I haven't had a client yet which would have approved any external AI usage. So I basically use them as search engines on steroids, but code can't go directly in or out.
  - fhd2 1y ago
    You might be able to get your clients to sign something to allow usage, but if you don't, as you say, it doesn't seem wise to vibe code for them. For two reasons: 1. A typical contract transfers the rights to the work. The ownership of AI generated code is legally a wee bit disputed. If you modify and refactor generated code heavily it's probably fine, but if you just accept AI generated code en masse, making your client think that you wrote it and it is therefore their copyright, that seems dangerous. 2. A typical contract or NDA also contains non disclosure, i.e. you can't share confidential information, e.g. code (including code you _just_ wrote, due to #1) with external parties or the general public willy nilly. Whether any terms of service assurances from OpenAI or Anthropic that your model inputs and outputs will probably not be used for training are legally sufficient, I have doubts. IANAL, and _perhaps_ I'm wrong about one or both of these, in one or more countries, but by and large I'd say the risk is not worth the benefit. I mostly use third party LLMs like I would StackOverflow: Don't post company code there verbatim, make an isolated example. And also don't paste from SO verbatim. I tried other ways of using LLMs for programming a few times in personal projects and can't say I worry about lower productivity with these limitations. YMMV. (All this also generally goes for employees with typical employment contracts: It's probably a contract violation.)
burnte 1y ago
I have an M4 Mac Mini with 24GB of RAM. I loaded Studio.LM on it 2 days ago and had Mistral NeMo running in ten minutes. It's a great model, I need to figure out how to add my own writing to it, I want it to generate some starter letters for me. Impressive model.
ulnarkressty 1y ago
I think many in this thread are underestimating the desire of VPs and CTOs to just offload the risk somewhere else. Quite a lot of companies handling sensitive data are already using various services in the cloud and it hasn't been a problem before - even in Europe with its GDPR laws. Just sign an NDA or whatever with OpenAI/Google/etc. and if any data gets leaked they are on the hook.
- boringg 1y ago
  Good luck ever winning that one. How are you going to prove out a data leak with an AI model without deploying excessive amounts of legal spend? You might be talking about small tech companies that have no other options.
nicce 1y ago
> Btw, you can also run Mistral locally within the Docker model runner on a Mac. Efficiently? I thought macOS does not have API so that Docker could use GPU.
- jt_b 1y ago
  I haven't/wouldn't use it because I have a decent K8S ollama/open-webui setup, but docker announced this a month ago: https://www.docker.com/blog/introducing-docker-model-runner https://www.docker.com/blog/introducing-docker-model-runner
  - nicce 1y ago
    Hmm, I guess that is not actually running inside container/ there is no isolation. Some kind of new way that mixes llama.cpp , OCI format and docker CLI.
simonw 1y ago
There are plenty of other ways to run Mistral models on a Mac. I'm a big fan of Mistral Small 3.1. I've run that using both Ollama (easiest) and MLX. Here are the Ollama models: https://ollama.com/library/mistral-small3.1/tags https://ollama.com/library/mistral-small3.1/tags - the 15GB one works fine. For MLX https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-Instruct-2503-8bit https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-I... and https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-Instruct-2503-4bit https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-I... should work, I use the 8bit one like this: llm install llm-mlx llm mlx download-model mlx-community/Mistral-Small-3.1-Text-24B-Instruct-2503-8bit -a mistral-small-3.1 llm chat -m mistral-small-3.1 The Ollama one supports image inputs too: llm install llm-ollama ollama pull mistral-small3.1 llm -m mistral-small3.1 'describe this image' \ -a https://static.simonwillison.net/static/2025/Mpaboundrycdfw-1.png Output here: https://gist.github.com/simonw/89005e8aa2daef82c53c2c2c62207f6a#response https://gist.github.com/simonw/89005e8aa2daef82c53c2c2c62207...
- indigodaddy 1y ago
  Simon, can you recommend some small models that would be usable for coding on a standard M4 Mac Mini (only 16G ram) ?
  - the_other_mac 1y ago
    Run Mistral 7b in under 4gb ram: https://github.com/garagesteve1155/Overload https://github.com/garagesteve1155/Overload (As announced this morning in the FB group "Dull Men's Club!)
  - simonw 1y ago
    That's pretty tough - the problem is that you need to have RAM left over to run actual applications! Qwen 3 8B on MLX runs in just 5GB of RAM and can write basic code but I don't know if it would be good enough for anything interesting: https://simonwillison.net/2025/May/2/qwen3-8b/ https://simonwillison.net/2025/May/2/qwen3-8b/ Honestly though with that little memory I'd stick to running against hosted LLMs - Claude 3.7 Sonnet, Gemini 2.5 Pro, o4-mini are all cheap enough that it's hard to spend much money with them for most coding workflows.
ATechGuy 1y ago
Why not use confidential computing based offerings like Azure's private inference for privacy concerns?
lolinder 1y ago
Game changer feels a bit strong. This is a new entry in a field that's already pretty crowded with open source tooling that's already available to anyone with the time and desire to wire it all up. It's likely that they execute this better than the community-run projects have so far and make it more approachable and Enterprise friendly, but just for reference I have most of the features that they've listed here already set up on my desktop at home with Ollama, Open WebUI, and a collection of small hand-rolled apps that plug into them. I can't run very big models on mine, obviously, but if I were an Enterprise I would. The key thing they'd need to nail to make this better than what's already out there is the integrations. If they can make it seamless to integrate with all the key third-party enterprise systems then they'll have something strong here, otherwise it's not obvious how much they're adding over Open WebUI, LibreChat, and the other self-hosted AI agent tooling that's already available.
- troyvit 1y ago
  > crowded with open source tooling that's already available to anyone with the time and desire to wire it all up. Those who don't have the time and desire to wire it all up probably make up a larger part of the market than those who do. It's a long-tail proposition, and that might be a problem. > I have most of the features that they've listed here already set up on my desktop at home I think your boss and your boss' boss are the audience they are going for. In my org there's concern over the democratization of locally run LLMs and the loss of data control that comes with it. Mistral's product would allow IT or Ops or whatever department to set guardrails for the organization. The selling point that it's turn-key means that a small organization doesn't have to invest a ton of time into all the tooling needed to run it and maintain it. Edit: I just re-read your comment and I do have to agree though. "game-changer" is a bit strong of a word.
abujazar 1y ago
Actually you shouldn't be running LLMs in Docker on Mac because it doesn't have GPU support. So the larger models will be extremely slow if they'll even produce a single token.
raxxorraxor 1y ago
I think the the standard setup for vscode continue for ollama is already 99% of ai coding support I need. I think it is even better than commercial offerings like cursor, at least in the projects and languages I use and have tested it. We had a Mac Studio here nobody was using and it we now use it as a tiny AI station. If we like, we could even embed our codebases, but it wasn't necessary yet. Otherwise it should be easy to just buy a decent consumer PC with a stronger GPU, but performance isn't too bad even for autocomplete.
- thepill 1y ago
  Which models are you using?
dzhiurgis 1y ago
How many is many? Literally all of them use cloud services.
Palmik 1y ago
I really don't see the big deal. Gemini also allows on-prem in similar fashion: https://cloud.google.com/blog/products/ai-machine-learning/run-gemini-and-ai-on-prem-with-google-distributed-cloud https://cloud.google.com/blog/products/ai-machine-learning/r...

I think this is a game changer, because data privacy is a legitimate concern for many enterprise users. Btw, you can also run Mistral locally within the Docker model runner on a Mac.

v3ss0n 1y ago

What's the point when we can run much powerful models now? Qwen3 , Deepseek

_bin_ 1y ago
It would be short-termist for Americans or euros to use chinese-made models. Increasing their popularity has an indirect but significant cost in the long term. china "winning AI" should be an unacceptable outcome for America or europe by any means necessary.
- atwrk 1y ago
  Why would that be? I can see why Americans wouldn't want to do that, but Europeans? In the current political climate, where the US openly claims their desire to annex European territory and so on? I'd rather see them prefer a locally hostable open source solution like DeepSeek.
  - tigroferoce 1y ago
    My two cents, as European, is that since we are more and more asking to LLMs for information, it wouldn't be wise to let a foreign country, not even truly democratic, to choose the information we get.
    - jamesblonde 1y ago
      The Chinese don't get any of information if we use self-hosted DeepSeek or Qwen. They are open-source. You can run them in an air-gapped environment that can't phone home.
      - fennecbutt 1y ago
        But their models are gimped by bad censoring. At least I can still ask chatgpt how many innocent civilians America has bombed.

kergonath 1y ago

> I think this is a game changer, because data privacy is a legitimate concern for many enterprise users. Indeed. At work, we are experimenting with this. Using a cloud platform is a non-starter for data confidentiality reasons. On-premise is the way to go. Also, they’re not American, which helps. > Btw, you can also run Mistral locally within the Docker model runner on a Mac. True, but you can do that only with their open-weight models, right? They are very useful and work well, but their commercial models are bigger and hopefully better (I use some of their free models every day, but none of their commercial ones).

distances 1y ago
I also kind of don't understand how it seems everyone is using AI for coding. I haven't had a client yet which would have approved any external AI usage. So I basically use them as search engines on steroids, but code can't go directly in or out.
- fhd2 1y ago
  You might be able to get your clients to sign something to allow usage, but if you don't, as you say, it doesn't seem wise to vibe code for them. For two reasons: 1. A typical contract transfers the rights to the work. The ownership of AI generated code is legally a wee bit disputed. If you modify and refactor generated code heavily it's probably fine, but if you just accept AI generated code en masse, making your client think that you wrote it and it is therefore their copyright, that seems dangerous. 2. A typical contract or NDA also contains non disclosure, i.e. you can't share confidential information, e.g. code (including code you _just_ wrote, due to #1) with external parties or the general public willy nilly. Whether any terms of service assurances from OpenAI or Anthropic that your model inputs and outputs will probably not be used for training are legally sufficient, I have doubts. IANAL, and _perhaps_ I'm wrong about one or both of these, in one or more countries, but by and large I'd say the risk is not worth the benefit. I mostly use third party LLMs like I would StackOverflow: Don't post company code there verbatim, make an isolated example. And also don't paste from SO verbatim. I tried other ways of using LLMs for programming a few times in personal projects and can't say I worry about lower productivity with these limitations. YMMV. (All this also generally goes for employees with typical employment contracts: It's probably a contract violation.)

burnte 1y ago

I have an M4 Mac Mini with 24GB of RAM. I loaded Studio.LM on it 2 days ago and had Mistral NeMo running in ten minutes. It's a great model, I need to figure out how to add my own writing to it, I want it to generate some starter letters for me. Impressive model.

ulnarkressty 1y ago

I think many in this thread are underestimating the desire of VPs and CTOs to just offload the risk somewhere else. Quite a lot of companies handling sensitive data are already using various services in the cloud and it hasn't been a problem before - even in Europe with its GDPR laws. Just sign an NDA or whatever with OpenAI/Google/etc. and if any data gets leaked they are on the hook.

boringg 1y ago
Good luck ever winning that one. How are you going to prove out a data leak with an AI model without deploying excessive amounts of legal spend? You might be talking about small tech companies that have no other options.

nicce 1y ago

> Btw, you can also run Mistral locally within the Docker model runner on a Mac. Efficiently? I thought macOS does not have API so that Docker could use GPU.

jt_b 1y ago
I haven't/wouldn't use it because I have a decent K8S ollama/open-webui setup, but docker announced this a month ago: https://www.docker.com/blog/introducing-docker-model-runner https://www.docker.com/blog/introducing-docker-model-runner
- nicce 1y ago
  Hmm, I guess that is not actually running inside container/ there is no isolation. Some kind of new way that mixes llama.cpp , OCI format and docker CLI.

simonw 1y ago

There are plenty of other ways to run Mistral models on a Mac. I'm a big fan of Mistral Small 3.1. I've run that using both Ollama (easiest) and MLX. Here are the Ollama models: https://ollama.com/library/mistral-small3.1/tags https://ollama.com/library/mistral-small3.1/tags - the 15GB one works fine. For MLX https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-Instruct-2503-8bit https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-I... and https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-Instruct-2503-4bit https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-I... should work, I use the 8bit one like this: llm install llm-mlx llm mlx download-model mlx-community/Mistral-Small-3.1-Text-24B-Instruct-2503-8bit -a mistral-small-3.1 llm chat -m mistral-small-3.1 The Ollama one supports image inputs too: llm install llm-ollama ollama pull mistral-small3.1 llm -m mistral-small3.1 'describe this image' \ -a https://static.simonwillison.net/static/2025/Mpaboundrycdfw-1.png Output here: https://gist.github.com/simonw/89005e8aa2daef82c53c2c2c62207f6a#response https://gist.github.com/simonw/89005e8aa2daef82c53c2c2c62207...

indigodaddy 1y ago
Simon, can you recommend some small models that would be usable for coding on a standard M4 Mac Mini (only 16G ram) ?
- the_other_mac 1y ago
  Run Mistral 7b in under 4gb ram: https://github.com/garagesteve1155/Overload https://github.com/garagesteve1155/Overload (As announced this morning in the FB group "Dull Men's Club!)
- simonw 1y ago
  That's pretty tough - the problem is that you need to have RAM left over to run actual applications! Qwen 3 8B on MLX runs in just 5GB of RAM and can write basic code but I don't know if it would be good enough for anything interesting: https://simonwillison.net/2025/May/2/qwen3-8b/ https://simonwillison.net/2025/May/2/qwen3-8b/ Honestly though with that little memory I'd stick to running against hosted LLMs - Claude 3.7 Sonnet, Gemini 2.5 Pro, o4-mini are all cheap enough that it's hard to spend much money with them for most coding workflows.

ATechGuy 1y ago

Why not use confidential computing based offerings like Azure's private inference for privacy concerns?

lolinder 1y ago

Game changer feels a bit strong. This is a new entry in a field that's already pretty crowded with open source tooling that's already available to anyone with the time and desire to wire it all up. It's likely that they execute this better than the community-run projects have so far and make it more approachable and Enterprise friendly, but just for reference I have most of the features that they've listed here already set up on my desktop at home with Ollama, Open WebUI, and a collection of small hand-rolled apps that plug into them. I can't run very big models on mine, obviously, but if I were an Enterprise I would. The key thing they'd need to nail to make this better than what's already out there is the integrations. If they can make it seamless to integrate with all the key third-party enterprise systems then they'll have something strong here, otherwise it's not obvious how much they're adding over Open WebUI, LibreChat, and the other self-hosted AI agent tooling that's already available.

troyvit 1y ago
> crowded with open source tooling that's already available to anyone with the time and desire to wire it all up. Those who don't have the time and desire to wire it all up probably make up a larger part of the market than those who do. It's a long-tail proposition, and that might be a problem. > I have most of the features that they've listed here already set up on my desktop at home I think your boss and your boss' boss are the audience they are going for. In my org there's concern over the democratization of locally run LLMs and the loss of data control that comes with it. Mistral's product would allow IT or Ops or whatever department to set guardrails for the organization. The selling point that it's turn-key means that a small organization doesn't have to invest a ton of time into all the tooling needed to run it and maintain it. Edit: I just re-read your comment and I do have to agree though. "game-changer" is a bit strong of a word.

abujazar 1y ago

Actually you shouldn't be running LLMs in Docker on Mac because it doesn't have GPU support. So the larger models will be extremely slow if they'll even produce a single token.

raxxorraxor 1y ago

I think the the standard setup for vscode continue for ollama is already 99% of ai coding support I need. I think it is even better than commercial offerings like cursor, at least in the projects and languages I use and have tested it. We had a Mac Studio here nobody was using and it we now use it as a tiny AI station. If we like, we could even embed our codebases, but it wasn't necessary yet. Otherwise it should be easy to just buy a decent consumer PC with a stronger GPU, but performance isn't too bad even for autocomplete.

thepill 1y ago
Which models are you using?

dzhiurgis 1y ago

How many is many? Literally all of them use cloud services.

Palmik 1y ago

I really don't see the big deal. Gemini also allows on-prem in similar fashion: https://cloud.google.com/blog/products/ai-machine-learning/run-gemini-and-ai-on-prem-with-google-distributed-cloud https://cloud.google.com/blog/products/ai-machine-learning/r...