7 ms·
Llama 3 is tuned very nicely for English answers. What is most surprising to me is that the 8B model is performing similarly to Mistral's large model and the or
by davej 2y ago
Llama 3 is tuned very nicely for English answers. What is most surprising to me is that the 8B model is performing similarly to Mistral's large model and the original GPT4 model (in English answers). Easily the most efficient model currently available.
- swalsh 2y agoParameter count seems to only matter for range of skills, but these smaller models can be tuned to be more than competitive with far larger models. I suspect the future is going to be owned by lots of smaller more specific models, possibly trained by much larger models. These smaller models have the advantage of faster and cheaper inference.
- theLiminator 2y agoProbably why MoE models are so competitive now. Basically that idea within a single model.
- elevaet 2y agoPlease, what is an MoE model?
- CuriouslyC 2y agoI don't think MoE is the way forward. The bottleneck is memory, and MoE trades MORE memory consumption for lower inference times at a given performance level. Before too long we're going to see architectures where a model decomposes a prompt into a DAG of LLM calls based on expertise, fans out sub-prompts then reconstitutes the answer from the embeddings they return.