6 ms·
Honest question: if they're only beating GPT 3.5 with their latest model (not GPT 4) and OpenAI/Google have infrastructure on tap and a huge distribution advant
by dizzydes 3y ago
Honest question: if they're only beating GPT 3.5 with their latest model (not GPT 4) and OpenAI/Google have infrastructure on tap and a huge distribution advantage via existing products - what chance do they stand?
How do people see things going in the future?
- spacebanana7 3y agoPerhaps they’re hoping some enterprises will be willing to pay extra for a 3.5 grade model that can run on prem? A niche market but I can imagine some demand there. Biggest challenge would be Llama models.
- anentropic 3y agoand according to the article this model behaves like a 12B model in terms of speed and cost while matching or outperforming Llama 2 70B in output
- viraptor 3y agoIn terms of speed per token. What they don't say explicitly is that choosing the mix per token means you may need to reload the active model multiple times in a single sentence. If you don't have memory available for all the experts at the same time, that's a lot of memory swapping time.
- anon373839 3y agoTim Dettmers stated that he thinks this one could be compressed down to a 4GB memory footprint, due to the ability of MoE layers to be sparsified with almost no loss of quality.
- jlokier 3y agoIf your motivation is to be able to run the model on-prem, with parallelism for API service throughput (rather than on a single device), you don't need large memory GPUs or intensive memory swapping. You can architect it as cheaper, low-memory GPUs, one expert submodel per GPU, transferring state over the network between the GPUs for each token. They run in parallel by overlapping API calls (and in future by other model architecture changes). Th MoE model reduces inter-GPU communication requirements for splitting the model, in an addition to reducing GPU processing requirements, compared with a non-MoE model with the same number of weights. There are pros and cons to this splitting, but you can see the general trend.
- v4dok 3y agoNiche market?? You have no idea how big that market is!
- mepiethree 3y agoYeah I would venture to say it’s closer to “the majority of the market” than “niche”
- visarga 3y agoAlmost no serious user - private or company - wants to slurp their private data to cloud providers. Sometimes it is ethically or contractually impossible.
- michaelt 3y agoThe success of AWS and Gmail and Google docs and Azure and Github and Cloudflare make me think this... probably not an up-to-date opinion. By and large, companies actually seem perfectly happy to hand pretty much all their private data over to cloud providers.
- b4ke 3y agoyet they don't provide access to their children, there may be something in that.
- evantbyrne 3y agoWe can't use LLMs at work at all right now because of IP leakage, copyright, and regulatory concerns. Hosting locally would solve one of those issues for us.
- joelthelion 3y agoCompete on price (open-source model, cheap hosted inference) probably? Also, they are probably well-placed to answer some proposals from European governments, who won't want to depend on US-companies too much.
- dataking 3y ago> they are probably well-placed to answer some proposals from European governments That's true but I wonder how they stack up against Aleph Alpha and Kyutai? Genuinely curious as I haven't found a lot of concrete info on their offerings.
- dataking 3y agoMicrosoft, Apple, and Google also have more resources at their disposal yet Linux is doing just fine (to put it mildly). As long as Mistral delivers something unique, they'll be fine.
- mhh__ 3y agoLinux is funded by big tech companies. IBM probably put a billion into Linux and that was 20 years ago now.
- smcleod 3y agoMistral and its hybrids are a lot better than GPT3.5, and while not as good as GPT4 in general tasks - they’re extremely fast and powerful with specific tasks. In the time it takes GPT4 to apologise that it’s not allowed to do something I can be three iterations deep getting highly targeted responses from mistral - and best yet - I can run it 100% offline, locally and on my laptop.
- stavros 3y agoAre they a lot better than 3.5? I see wildly varying opinions.
- MacsHeadroom 3y agoMistral-Medium, the one announced here which beats GPT-3.5 on every benchmark, isn't even available yet. Those opinions are referencing Mistral-Tiny (aka Mistral-7B). However, Mistral-Tiny beats the latest GPT-3.5 in human ratings on the chatbot-arena-leaderboard, in the form of OpenHermes-2.5-Mistral-7B. Mixtral 8x7B aka (Mistral-Small) was released a couple of days ago and will likely come close to GPT-4, and well above GPT-3.5, on the leaderboards once it has gone through some finetuning.
- whimsicalism 3y agoThey could be. It is an open question whether the driving force will be OSS improving or OAI continuing to try to distill their model.
- sorokod 3y agoThere is an attempt to quantify subjective evaluation of models here[1] - the "Arena Elo rating". According to popular vote, Mistral chat is nowhere near GPT 3.5 [1] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- sva_ 3y agoDoesn't seem like that's the mixture of experts model in the list? Or am I blind
- raincole 3y ago> How do people see things going in the future? A niche thing that thrives in its own niche. Just like most open source apps without big corperations behind them.
- nuz 3y agoThey might be willing to do things like crawl libgen which google possibly isn't, giving them an advantage. They might be more skilled at generating useful synthetic data which is a bit of an art and subject to taste, which other competitors might not be as good at.
- raincole 3y ago> They might be willing to do things like crawl libgen which google possibly isn't Are you implying big companies don't crawl libgen? Or google specifically? I would be very surprised if OpenAI (MS) didn't crawl libgen.
- nuz 3y agoOpenAI probably does. Not sure about google, possibly not
- happycube 3y agoGoogle has a ton of scanned books and magazines from libraries etc, on top of their own web crawls. If they don't have the equivalent of libgen tucked away something's gone wrong.
- antirez 3y ago1. This is an open source model that can run on people's hardware at a fraction of the cost of GPT. No cloud services in the middle. 2. This model is not censored as GPT. 3. This model has a lot less latency than GPT. 4. In their endpoint this model is called mistral-small. Probably they are training something much larger than can compete with GPT4. 5. This model can be fine tuned.
- yawnxyz 3y agohow does this work in their favor as a business? Don't get me wrong I love how all of it's free, but that doesn't seem to be helpful towards a $2b valuation. At least WeWork charged for access
- pulse7 3y agoMaybe they will charge for accessing the future Mixtral 8x70B ...
- antirez 3y ago* Open source models: give you all the attention (pun intended) you can get, away from OpenAI. At the same time do a great service to the world. * Maybe in the future, bigger closed models? Make money with the state-of-art of what you can provide.
- supriyo-biswas 3y agoMany VC funded businesses do not have an initial business model involving direct monetization. The first step is probably gaining mindshare with free, open source models, and then they can extend into model training services, consultation for ML model construction, and paid access to proprietary models, similar to OpenAI.
- jddj 3y agoEven in the public markets this happens all the time, eg. Biotech, new battery chemistries, etc. In trends people pay for a seat at the table with a good team and worry about the details later. The 2B headline number is a distraction.
- pbmonster 3y agoThey are focusing hard on small models. Sooner or later, you'll be able to run their product offline, even on mobile devices. Google was criticized [0] for offloading pretty much all generative AI tasks onto the cloud - instead of running it on the Tensor G3 built into its Pixel Phones specifically for that purpose. The reason being, of course, that the Tensor G3 is much to small for almost all modern generative models. So Mistral is focusing specifically on an area the big players are failing right now. [0] https://news.ycombinator.com/item?id=37966569 https://news.ycombinator.com/item?id=37966569
- jstummbillig 3y agoPretty much as with OSS in general: Lagging behind the cutting edge in terms of functionality/ux/performance, in areas where and as long as big tech is feeling combative, but eventually, probably, good enough across all axis to be useable. There could be a close-ish future where OpenAI tech will simply solve most business problems and there is no need for anything dramatically better in terms of AI tech. Think of word/google docs: It's doing what most businesses need well enough. For the most part people are not longing for anything else and happy with it just staying familiar. This is where Open Source can catch up relatively easily.
- jeswin 3y ago> Pretty much as with OSS in general That's not how I feel about OSS - from Operating Systems, to Databases, to Browsers, to IDEs, to tools like Blender etc. Of course there are certain areas where Commercial offerings are better, but can't generalize.
- jstummbillig 3y agoOh well, it's an evaluation, but I feel you may have glossed over the "in areas where and as long as big tech is feeling combative" part. > to tools like Blender "Tools like" needs a little more content to not be filled massive amounts of magical OSS thinking. Blender has in recent years gained an interesting amount of pro-adoption, but, in general, as for the industries that I have good insight into, inkscape, gimp, ardour or penpot are not winning. This is mostly debated by people who are not actually mainly and professionally using these tools. There are exceptions, of course (nextcloud might be used over google workspace when compliance is critical) but businesses will on average use the best tool, because the perceived value is still high enough and the cost is not, specificially when contrasted with labor cost and training someone to use a different tool.
- mola 3y agoAre you seriously claiming most oss is irrelevant? Maybe in consumer facing products such as libre office. But oss powers most of commercial products. I wouldn't be surprised if most functionality in all of current software is built from a thin layer over open source software.
- sgt101 3y agoAs I read it they are doing this with 8 * 7Bn parameter models. So, their model should run pretty well as fast as a 7Bn model and at the cost of a 56bn parameter model. That a lot quicker and cheaper than GPT-4 Also this is kinda a promissory note, they've been able to do this in a few months and create a service on top of it. Does this intimate that they have the capability to create and run SoA models? Possibly. If I were a VC I could see a few ways for this bet to go well. The big killer is moat - maybe this just demonstrates that there is no LLM moat.
- ankit219 3y agoYou could broadly segregate the market into three groups - general purpose, specialized-instructions, and local tasks. For general purpose, look at the uses of GPT4. Gemini might give them competition lately, and I dont think OSS would in the near future. They are trained on open internet and are going to be excellent at various tasks like answering basic questions, coding, generating content for marketing or website. Where they do badly is when you introduce a totally new concept which is likely outside of their training data. Dont think mistral is even trying to compete with them. local tasks is a mix of automation and machine level tasks. A small mistral like model would work superbly well because it does not require as much expertise. Usecases like locating a file by semantic search, generating answers to reply to email/text within context, summarize a webpage. Specialized instructions though is key for OSS. From two angles. One is security and compliance. Open AI uses a huge system prompt to get their model to perform in a particular manner, and for different companies, policies and compliance requirements may result in a specific system prompt for guardrails. This is ever changing and better to have an open source model that can be customized than depending on Open AI. From the blog post. > Note: Mixtral can be gracefully prompted to ban some outputs from constructing applications that require a strong level of moderation, as exemplified here. A proper preference tuning can also serve this purpose. Bear in mind that without such a prompt, the model will just follow whatever instructions are given. I think moderation is one such issue. Could be many and it is an evolving space as we go forward. (though this is likely to be an exposed functionality in future Open AI models). There is also the data governance bit - which is easier to do with an oss model than just depending on Open AI apis, just architectural reasons. The second is training a model on domain knowledge of the company. We at Clio AI[1] (sorry, shameless plug) have had seven requests in the last one month about companies wanting their own private models pretrained on their own domain knowledge. These datasets are not on open internet and so no model is good at answering based them. A catalyst was Open AI dev day[2] which asked for proposals for custom models trained on enterprise domain knowledge. and their price start at $2M. Finetuning works, but on small datasets, not the bigger ones. Large Companies are likely to approach Open AI and all these OSS models to train a custom instruction following model. Cos there are a handful of people who have done it, and that is the way they can get most out of a LLM deployment. [1]: https://www.clioapp.ai/custom-llm-model https://www.clioapp.ai/custom-llm-model Sorry for the shameless plug. Still working on website so it wont be as clear. [2]:https://openai.com/form/custom-models https://openai.com/form/custom-models
- jillesvangurp 3y agoThe demand for using AI models for whatever is going through the roof. Right now it's mostly people typing things manually in chat gpt, bard, or wherever. But that's not going to stay like that. Models being queried as part of all sorts of services is going to be a thing. The problem with this is that running these models at scale is still really expensive. So, instead of using the best possible model at any cost for absolutely everything, the game is actually good enough models that can run cheaply at scale that do a particular job. Not everything is going to require models trained on the accumulated volume of human knowledge on the internet. It's overkill for a lot of use cases. Model runtime cost is a showstopper for a lot of use cases. I saw a nice demo of a big ecommerce company in Berlin that had built a nice integration with openai's APIs to provide a shopping assistent. Great demo. Then somebody asked them when this was launching. And the answer was that token cost was prohibitively expensive. It just doesn't make any sense until that comes down a few orders of magnitudes. Companies this size already have quite sizable budgets that they use on AI model training and inference.
- akbarnama 3y agoIf possible, please share, how was the shopping assistant helping out a consumer in the buying process? What were the features?
- jillesvangurp 3y agoFeatures I saw demoed were about comparing products based on descriptions, images, and pricing. So, it was able to find products based on a question that was about something suitable for X costing less than Y where X can be some kind of situation or event. Or find me things similar to this but more like so. And so on.
- aunty_helen 3y agoI can agree with this, I’m currently building a system that pulls data from a series of pdfs that are semi-structured. Just testing alone is taking up 10s of $ in api costs. We have 60k PDFs to do. I can’t deliver a system to a client that costs more in api costs than it does in development costs for their expected input size. Using the most naive approach the ai would be beaten on a cost basis by a mechanical Turk.
- masa331 3y agoAnother advantage over Google or OpenAI for me would be that it is not from Google or OpenAI
- ekianjo 3y agoYou do understand that you cant run GPT4 on your own right?
- Shrezzing 3y ago>How do people see things going in the future? The EU and other European governments will throw absolute boatloads of money at Mistral, even if that only keeps them at a level on par with the last generation. AI is too big of a technological leap for the bloc to ride America's coattails on. Mistral doesn't just exist to make competitive AI products, it's an existential issue for Europe that someone on the continent is near the vanguard on this tech, and as such, they'll get enormous support.
- arlort 3y agoYou are vastly overestimating both the EU's budget and the willingness of countries to throw money at other countries' companies I doubt mistral will get any direct EU funding
- yodsanklai 3y agoEU is good at fostering free market, but not at funding strategic efforts. Some people (Piketty, Stiglitz) say that companies like Airbus couldn't emerge today for that reason.
- Culonavirus 3y ago> EU is good at fostering free market Uuuuuh... you could call the EU a lot of things, but "fostering free market" is a hot take. I'm sorry. When you look at the amount of regulation the EU brings to the table (EU basically is the poster child of market regulation), I would go as far as to say that your claim is objectively not true. We can debate how regulation is a good thing because this and that, but regulation - by definition - limits the free market. And there is an argument to be made, backed up literally thousands of regulations the EU has come up with, that the EU limits the free market a lot. When you factor in the regulations that are imposed on its member countries (I mean directly on the goverments) one could easily claim that it is the most harsh regulator on the planet. I could go into detail about the so called green deal, etc. but all of these things are easy to look up on the net / or official sources from the EU portal.
- Palmik 3y agoBeyond what others said, I think this is an extremely impressive showing. Consider that their efforts started years behind Google's, and yet their relatively small model (they call is mistral-small, and also offer mistral-medium) is beating or on par with Gemini Pro on many benchmarks (Google's best currently available model). On top of that Mixtral is truly open source (Apache 2.0), and extremely easy to self host or run on a cloud provider of your choice -- this unlocks many possibilities, and will definitely attract some business customers. EDIT: The just announced mistral-medium (larger version of the just open sourced mixtral 8x7b) is beating GPT3.5 with significant margin, and also Gemini Pro (on available benchmarks).
- HlessClaudesman 3y agoAI based on LLMs comes with several sets of inherent trade-offs, as such I don't predict that one winner will take all.
- yodsanklai 3y agoAlso, considering mistral is open source, what will prevent their competitor to integrate any innovation they make? Another thing I don't understand, how a 20 people company can provide a similar system as OpenAI (1000 employees)? what do they do themselves, and what do they re-use?
- lossolo 3y ago> Also, considering mistral is open source, what will prevent their competitor to integrate any innovation they make? Their small and tiny models are open source, it seems like a marketing strategy, and bigger models will not be open source. Their medium model is not open source. > Another thing I don't understand, how a 20 people company can provide a similar system as OpenAI (1000 employees)? what do they do themselves, and what do they re-use? They do not provide the scale of OpenAI or a model comparable to GPT-4 (yet).
- war321 3y agoCompanies move slow, especially as they get bigger. Just because a google engineer wants to yoink some open source inferencing innovation for example, doesn't mean they can just jam it into Gemini and have it rolled out immediately.
- HarHarVeryFunny 3y agoGoogle started late with any serious LLM effort. It takes time to iterate on something so complex and slow to train. I expect Google will match OpenAI in next iteration or two, or at worst stay one step behind, but it takes time. OTOH Google seem to be the Xerox Parc of our time (who were famous for state of the art research and failure to productize). Microsoft, and hence Microsoft-OpenAI, seem much better positioned to actually benefit from this type of generative AI.
- intellectronica 3y agoIf you're purely looking for capabilities and not especially interested in running an open model, this might not be that interesting. But even so, this positions Mistral as currently the most promising company in the open models camp, having released the first thing that not only competes well with GPT-3.5 but also competes with other open models like Llama-2 on cost/performance and presents the most technological innovation in the open models space so far. Now that they raised $400MM the question to ask is - what happens if they continue innovating and scale their next model sufficiently to compete with GPT-4 / Gemini? The prospects have never seemed better than they do today after this release.
- throwaway4aday 3y agoa lot of wordy answers to this but all you need to do is read the blog post to the end and notice this line: > We’re currently using Mixtral 8x7B behind our endpoint *mistral-small* emphasis on the name of the endpoint
- data-ottawa 3y agoGoogle BARD/AI isn’t available in Canada or the EU, so there’s one big competitive advantage. OpenAI is of course the big incumbent to beat and is in those markets. They only started this year, so beating ChatGPT3.5 is I think a great milestone for 6 months of work. Plus they will get a strategic investment as the EU’s answer to AI, which may become incredibly valuable to control and regulate. Edit: I fact checked myself and bard is available in the EU, I was working off outdated information. https://support.google.com/bard/answer/13575153?hl=en https://support.google.com/bard/answer/13575153?hl=en
- wrsh07 3y ago1) as a developer or founder looking to experiment quickly and cheaply with llm ideas, this (and llama etc) are huge gifts 2) for the research community, making this work available helps everyone (even OpenAI and Google, insofar as they've done something not yet tried at those larger orgs) 3) Mistral is well positioned to get money from investors or as consultants for large companies looking to fine tune or build models for super custom use cases The world is big and there's plenty of room for everyone!! Google and OpenAI haven't tried all permutations of research ideas - most researchers at the cutting edge have dozens of ideas they still want to try, so having smaller orgs trying things at smaller scales is really great for pushing the frontier! Of course it's always possible that some major tech co playing from behind (ahem, apple) might acquire some LLM expertise too
- deleted 3y ago[deleted]