10 ms·
Mistral Small 3
- Havoc 2y agoUsed it a bit today on coding tasks and overall very pleasant. The combination of fast and fits into 24gb is also appreciated Wouldn’t be surprised if this gets used a fair bit given open license
- rvz 2y agoThe AI race to zero continues to accelerate and Mistral has shown one card to just stay in the race. (And released for free) OpenAI's reaction to DeepSeek looked more like cope and panic after they realized they're getting squeezed at their own game. Notice how Google hasn't said anything with these announcements and didn't rush out a model nor did they do any price cuts? They are not in panic and have something up their sleeve. I'd expect Google to release a new reasoning model that is competitive with DeepSeek and o1 (or matches o3). Would be even more interesting if they release it for free.
- jiraiya0 2y agoAlready tried it. It’s called gemini-2.0-flash-thinking-exp-01-21. Looks better than DeepSeek.
- beAbU 2y agoGoogle has been consistently found with their finger up their nose during this entire AI bubble. The reason why they are so silent is because they are still reacting to ChatGPT 3.5
- christianqchung 2y agoThe Gemini launch was a complete disaster, but technically speaking since February 2024, Gemini 1.5 pro and the ensuing lineup have been very impressive.
- rvz 2y agoA lot can change in a year.
- jug 2y agoGemini 2.0 Experimental is now a leading LLM. They started out poorly, but after a more reasonable 1.5 Pro, 2.0 is in another class entirely and a direct competitor to o1 (or o1-mini as for Gemini 2.0 Flash). They've made quick strides forward as the DeepMind team is kicking into gear, and I feel like they're neglected a bit too often these days, especially now while usage cost on AI Studio is a nice $0.
- Arthur_ODC 2y agoDefinitely. The 1206 is probably my favorite out of any I've ever used.
- upbeat_general 2y agoimo gemini-exp-1206 is the best public LLM that exists right now.
- staticman2 2y agoGemini-exp-1206 seems significantly dumber than 1.5 pro on reading and comprehending large context documents.
- staticman2 2y agoGemini 1.5 pro is extremely impressive at reading 2 million tokens of a document and answering questions about it. And at least for the time being it's offered for free on AI studio.
- fuegoio 2y agoFinally something from them
- azinman2 2y agoThey released codestral on Jan 13. What do you mean by “finally”?
- timestretch 2y agoTheir models have been great, but I wish they'd include the number of parameters in the model name, like every other model.
- jbentley1 2y agoIt's 24B parameters
- deleted 2y ago[deleted]
- cptcobalt 2y agoThis is really exciting—the 12-32b size range has my favorite model size on my home computer, and the mistrals have been historically great and embraced for various fine-tuning. At 24b, I think this has a good chance of fitting on my more memory constrained work computer.
- deleted 2y ago[deleted]
- msp26 2y agoFinally, all the recent MoE model releases make me depressed with my mere 24GB VRAM. > Note that Mistral Small 3 is neither trained with RL nor synthetic data Not using synthetic data at all is a little strange
- bloopernova 2y agoI'm surprised no GPU cards are available with like a TB of older/cheaper RAM.
- gr3ml1n 2y agoNot surprising at all: Nvidia doesn't want to compete with their own datacenter cards.
- papichulo2023 2y agoNvidia upcoming 'minipc' has shared ram up to 128gb for around 3k. No a competitor but pretty good for enthusiast. Hopefully is at least quadchannel.
- wongarsu 2y agoAMD could arguably do it. But they have to focus to stay above water at all, and "put 128GB or more of DDR5 ram on any previous-gen GPU" is probably not in their focus. With the state of their software it's not even certain if the community could pick up the slack and turn that into a popular solution.
- hnuser123456 2y agoTheir next generation of APUs will have a lot more memory bandwidth and there will probably be lots of AMD APU laptops with 64GB+ of RAM that can use HW acceleration and not be artificially segmented the way Nvidia can do it with VRAM being soldered.
- aurareturn 2y agoBecause memory bandwidth is the #1 bottleneck for inference, even more than capacity. What good is 1TB RAM if the bandwidth is fed through a straw? Models would run very slow. You can see this effect on 128GB MacBook Pros. Yes, the model will fit but it’s slow. 500GB/s of memory bandwidth feeds 128GB RAM at a maximum rate of 3.9x per second. This means if your model is 128GB large, your max tokens/s is 3.9. In the real world, it’s more like 2-3 tokens/s after overhead and compute. That’s too slow to use comfortably. You’re probably wondering why not increase memory bandwidth too. Well, you need faster memory chips such as HBM and/or more memory channels. These changes will result in drastically more power consumption and bigger memory controllers. Great, you’ll pay for those. Now you’re bottlenecked by compute. Just add more compute? Ok, you just recreated the Nvidia H100 GPU. That’ll be $20k please. Some people have tried to use AMD Epyc CPUs with 8 channel memory for inference but those are also painfully slow in most cases.
- asb 2y agoNote the announcement at the end, that they're moving away from the non-commercial only license used in some of their models in favour of Apache: We’re renewing our commitment to using Apache 2.0 license for our general purpose models, as we progressively move away from MRL-licensed models
- diggan 2y agoNote that this seems to be about the weights themselves, AFAIK, the actual training code and datasets (for example) aren't actually publicly available. It's a bit like developing a binary application and slapping a FOSS license on the binary while keeping the code proprietary. Not saying that's wrong or anything, but people reading these announcements tend to misunderstand what actually got FOSS licensed when the companies write stuff like this.
- mcraiha 2y agoThe binary comparison is a bit bad, since binary can have copyrights. Weights cannot.
- diggan 2y agoHas that actually been tried in court, or is that your guess? Because you seem confident, but I don't think this has been tried (yet)
- badsectoracula 2y agoIt is a guess (not the same author) but it'd make sense: weights are machine output so if the output of AI is not under copyright because it is machine output (which seems to be something that is pretty much universally agreed upon), then the same would apply for the weights themselves. I'm not sure how someone would argue (in good faith) that training on copyrighted materials does not cause the weights to be a derivative of those materials and the output of their AI is not protected under copyright but the part in the middle, the weights, does fall under copyright. Note that this would be about the weights (i.e. the numbers), not their container.
- fvv 2y agogiven new USA ai diffusion rules will mistral be able to survive and attract new capitals ? , I mean, given that france is top tier country
- beAbU 2y agoThis sounds like a USA problem, rather than a Mistral problem.
- solomatov 2y agoWhat are these ai diffusion rules?
- Beretta_Vexee 2y ago"Those destinations, which are listed in paragraph (a) to Supplement No. 5 to Part 740, are Australia, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, Italy, Japan, the Netherlands, New Zealand, Norway, Republic of Korea, Spain, Sweden, Taiwan, the United Kingdom, and the United States. For these destinations, this IFR makes minimal changes: companies in these destinations generally will be able to obtain the most advanced ICs without a license as long as they certify compliance with specific requirements provided in § 740.27." [0] France seems clearly exempt from most of the requirements. The main requirement of 740.27 is to sign a license under U.S. law, under which customers are prohibited from re-exporting ICs to non-Third 1 countries without U.S. approval. What's more, the text refers to AIs, which can have dual uses. The concept of dual civil-military use concerns a large number of technologies, and dates back to the first nuclear technologies. The text gives a few examples of dual-use models, such as models that simulate or facilitate the production of chemical compounds that could be used for chemical weapon creation, non conventional weapon creation or that could simplify or replace already identified dual-use goods or technologies. These uses are already covered by existing legislation on dual-use goods, and US export control. The American legislator is therefore potentially thinking of other uses, such as satellite and radar image analysis, and electronic warfare. As France is a nuclear-armed country with its own version of thoses technologies, it makes little sense to place it under embargo. But France isn't going to like being obliged once again to be forced to apply American law and regulation on its soil. As a European, I hope that alternatives to American dependence will soon appear. [0] https://www.federalregister.gov/documents/2025/01/15/2025-00636/framework-for-artificial-intelligence-diffusion https://www.federalregister.gov/documents/2025/01/15/2025-00...
- netdur 2y agoseems on par or better than gpt4 mini
- bugglebeetle 2y agoInterested to see what folks do with putting DeepSeek-style RL methods on top of this. The smaller Mistral models have always punched above their weight and been the best for fine-tuning.
- petercooper 2y agoIt's not RL, but you can get a long way with a thorough system prompt to encourage it to engage in 'thinking' behavior on its own without extra training. Just playing with it myself now with promising results - Mistral Small seems very receptive to this approach (not all models are - cough, Llama). Update: This is such a prompt: https://gist.github.com/peterc/955d797ee35b3c777d76a2d881d2fb63 https://gist.github.com/peterc/955d797ee35b3c777d76a2d881d2f...
- Terretta 2y ago"When quantized, Mistral Small 3 can be run privately on a single RTX 4090 or a Macbook with 32GB RAM."
- jszymborski 2y agoThe trouble now is finding an RTX 4090.
- hnuser123456 2y agoRTX 3090s are easy to find and work just as well.
- petercooper 2y agoRunning the Q4 quant (14GB or so in size) at 46 tok/sec on a 3090 Ti right now if anyone's curious to performance. Want the headroom to try and max out the context.
- earleybird 2y agoInteresting - _q4 on a pair of 12Gb 3060s it runs at 20 tok/sec. _q8 (25Gb) on same is about 4 tok/sec.
- petercooper 2y ago~360GB/s memory bandwidth on the 3060, versus ~1008GB/s on the 3090 Ti probably accounts for that. Given that, I'd expect a single 3060 (if a large enough one existed) to run at about 16 tok/s so 20 tok/s on two isn't bad not being NVLinked.
- benkaiser 2y agoRuns on an AMD 7900 XTX at about ~20 tokens per second using LM Studio + Vulkan.
- yodsanklai 2y agoI'm curious, what people do with these smaller models?
- ignoramous 2y agoMistral repeatedly emphasize on "accuracy" and "latency" for this Small (24b) model; which to me means (and as they also point out): - Local virtual assistants. - Local automated workflows. Also from TFA: Our customers are evaluating Mistral Small 3 across multiple industries, including: - Financial services customers for fraud detection - Healthcare providers for customer triaging - Robotics, automotive, and manufacturing companies for on-device command and control - Horizontal use cases across customers include virtual customer service, and sentiment and feedback analysis.
- _boffin_ 2y agoCleaning messy assessor data. Email draft generation.
- frankfrank13 2y agoThey're fast, I used 4o mini to run the final synthesis in a CoT app and to do initial entity/value extraction in an ETL. Mistral is pretty good for code completions too, if I was in the Cursor business I would consider a model like this for small code-block level completions, and let the bigger models handle chat, large requests, etc.
- Beretta_Vexee 2y agoRAG mainly, Feature extraction, tagging, Document and e-mail classification. You don't need a 24B parameter to know whether the e-mail should go to accounting or customer support.
- pheeney 2y agoWhat models would you recommend for basic classification if you don't need a 24B parameter one?
- 2y ago
- resource_waste 2y agoCurious how it actually compares to LLaMa. Last year Mistral was garbage compared to LLaMa. I needed a permissive license, so I was forced to use Mistral, but I had LLaMa that I could compare it to. I was always extremely jealous of LLaMa since the Berkley Sterling finetune was so amazing. I ended up giving up on the project because Mistral was so unusable. My conspiracy was that there was some European patriotism that gave Mistral a bit more hype than was merited.
- maven29 2y agoThey're both European. Look at the author names on the llama paper.
- resource_waste 2y agoThat is a very European thing to say/do/claim.
- cpldcpu 2y agoa goof part of team is actually located in europe
- resource_waste 2y agoI said it was a very European thing to say, because only a European would stretch that hard. Merikan company with Merikan investment get the credit. No one cares except Europeans about the interchangable workers residency is. I'm trying to remember the other case where people lol'd at Europe/Italy for taking credit for something that was clearly invented in the US. I think the person was born there, and moved to the US, but Italy still took credit. lol no. Its probably even more embarrassing that they left Europe.
- cpldcpu 2y agoI thought now everything is about meritocracy? Have we been duped?
- unraveller 2y agoWhat's this stuff about the model catering to ‘80%’ of generative AI tasks? What model do they expect me to use for the other 20% of the time when my question needs reasoning smarts.
- xnx 2y agoMistral Large
- deleted 2y ago[deleted]
- zamadatix 2y agoTake your pick based on your use cases and needs?
- abdullahkhalids 2y agoCrazy idea: a small super fast model whose only job is to decide which model to send your task to.
- sneak 2y agoThere are APIs that use a very small model to determine the complexity of the request then route it to different apis or models based on the result of that classifier model. This way you can do cheap/local automatically without the api client having to know anything about it, and the proxy will send the requests out to an expensive big model only when necessary.
- GaggiX 2y agoHopefully they will finetuning it using RL like DeepSeek did, it would be great to have more open reasoning models.
- simonw 2y agoI'm excited about this one - they seem to be directly targeting the "best model to run on a decent laptop" category, hence the comparison with Llama 3.3 70B and Qwen 2.5 32B. I'm running it on a M2 64GB MacBook Pro now via Ollama and it's fast and appears to be very capable. This downloads 14GB of model weights: ollama run mistral-small:24b Then using my https://llm.datasette.io/ https://llm.datasette.io/ tool (so I can log my prompts to SQLite): llm install llm-ollama llm -m mistral-small:24b "say hi" More notes here: https://simonwillison.net/2025/Jan/30/mistral-small-3/ https://simonwillison.net/2025/Jan/30/mistral-small-3/
- isoprophlex 2y agoI make very heavy use of structured output (to convert unstructured data into something processable, eg for process mining on customer service mailboxes) Is it any good for this, if you tested it? I'm looking for something that hits the sweet spot of runs locally & follows prescribed output structure, but I've been quite underwhelmed so far
- rkwz 2y agoWhat local models are you currently using and what issues are you facing?
- the_mitsuhiko 2y agoI get decent JSON from it quite well with the "assistant: {" trick. I'm not sure how well trained it is to do JSON. The template on ollama has tools calls so I assume they made sure JSON works: https://ollama.com/library/mistral-small:24b/blobs/6db27cd4e277 https://ollama.com/library/mistral-small:24b/blobs/6db27cd4e...
- a_wild_dandan 2y agoAnd for anyone looking to dig deeper, check out "grammar-based sampling."
- azinman2 2y ago
- butz 2y agoIs there a gguf version that could be used with llamafile?
- simonw 2y agoA bunch have started showing up here: https://huggingface.co/models?other=base_model:quantized:mistralai/Mistral-Small-24B-Instruct-2501 https://huggingface.co/models?other=base_model:quantized:mis... The lmstudio-community ones tend to work well in my experience.
- strobe 2y agonot sure how much worse it than original but mistral-small:22b-instruct-2409-q2_K seems works on 16GB VRAM GPU
- Havoc 2y agoHow does that fit into a 4090? The files on the repo look way too large. Do they mean a quant?
- cbg0 2y ago> Mistral Small can be deployed locally and is exceptionally "knowledge-dense", fitting in a single RTX 4090 or a 32GB RAM MacBook once quantized.
- m3kw9 2y agoSorry to dampen the news but 4o-mini level isn’t really a useful model other than talk to me for fun type of applications.
- rcarmo 2y agoThere's also a 22b model that I appreciate, since it _almost_ fits into my 12GB 3060. But, alas, I might need to get a new GPU if this trend of fatter smaller models continues.
- aargh_aargh 2y agoThat's the older version (4 months old), check the release date.
- rcarmo 2y agoAh. I need to find a tighter quantization then, if it exists at all.
- mohsen1 2y agoNot so subtle in function calling example[1] "role": "assistant", "content": "---\n\nOpenAI is a FOR-profit company.", [1] https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501#function-calling https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-...
- picografix 2y agoTried running locally, gone were the days where you get broken responses on local models (i know this happened earlier but I tried after so many days)
- freehorse 2y agoI tried just a few of the code generating prompts I have used last days, and it looks quite good and promising. It seems at least on par with qwen2.5-coder-32b which was the first local model i would actually use for code. I am also surprised how far we went with small models producing such more polished output in the last year. On another note, I also wish they would follow up with a new version of the 8x7B mixtral. It was one of my favourite models, but at the time it could barely fit in my ram, and now that I have more ram it is rather outdated. But I don't complain, this model anyway is great and it is great that they are one of the companies which actually publish such models targeted to edge computing.
- spwa4 2y agoSo the point of this release is 1) code + weights Apache 2.0 licensed (enough to run locally, enough to train, not enough to reproduce this version) 2) Low latency, meaning 11ms per token (so ~90 tokens/sec on 4xH100) 3) Performance, according to mistral, somewhere between Qwen 2.5 32B and Llama 3.3 70B, roughly equal with GPT4o-mini 4) ollama run mistral-small (14G download) 9 tokens/sec on the question "who is the president of the US?" (also to enjoy that the answer ISN'T orange idiot)
- mrbonner 2y agoIs there a chance for me to get a eGPU (external GPU dock) for my M1 16GB laptop to plunge thru this model?
- hnfong 2y agoThe smaller IQ2/Q3 GGUF quants should run "fine" on your existing 16GB. (also, I don't know that M1 supports any eGPU...)
- rahimnathwani 2y agoUntil today, no language model I've run locally on a 32GB M1 has been able to answer this question correctly: "What was Mary J Blige's first album?" Today, a 4-bit quantized version of Mistral Small (14GB model size) answered correctly :) https://ollama.com/library/mistral-small:24b-instruct-2501-q4_K_M https://ollama.com/library/mistral-small:24b-instruct-2501-q...
- kamranjon 2y agoI just tried your question against Gemma 2 27b llamafile on my M1 Macbook with 32gb of ram, here is the transcript: >>> What was Mary J Blige's first album? Mary J. Blige's first album was titled *"What's the 411?"*. It was released on July 28, 1992, by Uptown Records and became a critical and commercial success, establishing her as the "Queen of Hip-Hop Soul." Would you like to know more about the album, like its tracklist or its impact on music?
- rahimnathwani 2y agoAh! I had not tried any gemma models locally. It worked: % llm -m gemma2:27b-instruct-q4_0 "What was Mary J Blige's first album?" Mary J. Blige's first album was **"What's the 411?"** It was released in July 1992. Let me know if you have any other questions about Mary J. Blige!
- svachalek 2y agoGemma2 seems to be the best small model for trivia. Even the 9b model really surprises me sometimes with things it can answer, that seem ridiculous for a small model to know.
- tadamcz 2y agoHi! I'm Tom, a machine learning engineer at the nonprofit research institute Epoch AI [0]. I've been working on building infrastructure to: * run LLM evaluations systematically and at scale * share the data with the public in a rigorous and transparent way We use the UK government's Inspect [1] library to run the evaluations. As soon as I saw this news on HN, I evaluated Mistral Small 3 on MATH [2] level 5 (hardest subset, 1,324 questions). I get an accuracy of 0.45 (± 0.011). We sample the LLM 8 times for each question, which lets us obtain less noisy estimates of mean accuracy, and measure the consistency of the LLM's answers. The 1,324*8=10,584 samples represent 8.5M tokens (2M in, 6.5M out). You can see the full transcripts here in Inspect’s interactive interface: https://epoch.ai/inspect-viewer/484131e0/viewer?log_file=https%3A%2F%2Fepoch-benchmarks-production-public.s3.us-east-2.amazonaws.com%2Finspect_ai_logs%2FNbsnvBsMoMizozbPZY8LLb.eval https://epoch.ai/inspect-viewer/484131e0/viewer?log_file=htt... Note that MATH is a different benchmark from the MathInstruct [3] mentioned in the OP. It's still early days for Epoch AI's benchmarking work. I'm developing a systematic database of evaluations run directly by us (so we can share the full details transparently), which we hope to release very soon. [0]: https://epoch.ai/ https://epoch.ai/ [1]: https://github.com/UKGovernmentBEIS/inspect_ai https://github.com/UKGovernmentBEIS/inspect_ai [2]: https://arxiv.org/abs/2103.03874 https://arxiv.org/abs/2103.03874 [3]: https://huggingface.co/datasets/TIGER-Lab/MathInstruct https://huggingface.co/datasets/TIGER-Lab/MathInstruct
- coalteddy 2y agoThanks a lot for this eval! One question i have regarding evals is, what sampling temperature and/or method do you use? As far as i understand temperature/ method can impact model output alot. Would love to here you're thoughts on how these different settings of the same model can impact output and how to go about evaluating models when its not clear how to use the to their fullest
- tadamcz 2y agoGenerally, we'll use the API provider's defaults. For models we run ourselves from the weights, at the moment we'd use vLLM's defaults, but this may warrant more thought and adjustment. Other things being equal, I prefer to use an AI lab's API, with settings as vanilla as possible, so that we essentially defer to them on these judgments. For example, this is why we ran this Mistral model from Mistral's API instead of from the weights. I believe the `temperature` parameter, for example, has different implementations across architectures/models, so it's not as simple as picking a single temperature number for all models. However, I'm curious if you have further thoughts on how we should approach this. By the way, in the log viewer UI, for any model call, you can click on the "API" button to see the payloads that were sent. In this case, you can see that we do not send any values to Mistral for `top_p`, `temperature`, etc.
- adt 2y agohttps://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/
- mike31fr 2y agoRunning it on a MacBook with M1 Pro chip and 32 GB of RAM is quite slow. I expected to be as fast as phi4 but it's much slower.
- mike31fr 2y agoWith eval rate numbers: - phi4: 12 tokens/s - mistral-small: 9 tokens/s On Nvidia RTX 4090 laptop: - phi4: 36 tokens/s - mistral-small: 16 tokens/s
- Alifatisk 2y agoIs there a good benchmark one can look at that shows the best performing llm in terms of instruction following or overall score? The only ones I am aware of is benchmarks on Twitter, Chatbot Arena [1] and Aider benchmark [2] 1. https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leaderboard https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leade... 2. https://aider.chat/docs/leaderboards https://aider.chat/docs/leaderboards
- mariconrobot 2y agoi cunt get past the name