6 ms·
It’s great that this can run on a laptop but FWIW, llama 70B model is no where near “GPT-4 class” in my own use cases. 405B might be, though I haven’t tested it
by stevedekorte 2y ago
It’s great that this can run on a laptop but FWIW, llama 70B model is no where near “GPT-4 class” in my own use cases. 405B might be, though I haven’t tested it.
- buildbot 2y agoThe new 3.3 70B model has comparable benchmarks to the 405B model, which is probably what people mean by GPT-4 class.
- simonw 2y agoAre you sure about that? When I say GPT-4 class I'm talking about being comparable to the GPT-4 that was released in March 2023. The Llama 3.3 70B model is clearly no way near as good as today's GPT-4o family of models, or the other top-ranking models today like Gemini 1.5 Pro and Claude 3.5 Sonnet. To my surprise, Llama 3.3 70B is ranking higher than Claude 3 Opus on https://livebench.ai/ https://livebench.ai/ - I'm suspicious of that result, personally. I think Opus was the best available model for a few months earlier this year.
- mmiyer 2y agoI guess it's because it has the highest score of all models in instruction following, 20 points higher then Opus, which compensates for shortcomings elsewhere (e.g. in language), and which wouldn't necessarily translate to human evaluation of usefulness.
- simonw 2y agoWow, yeah I think you're right - 3.3 somehow gets top position on the entire leaderboard for that category, I bet that skews the average up a lot.
- MichaelZuo 2y agoHow do you reliably compare it with the GPT-4 released in March 2023?
- simonw 2y agoVibes, based on what I can remember using that model for. There's still a gpt-4 model available via the OpenAI API, but it's gpt-4-0613 from June 2023 - the March 2023 snapshot gpt-4-0314 is no longer available. I ran one of my test prompts against that old June 2023 GPT-4 model here: https://gist.github.com/simonw/de4951452df2677f2a1a3cd415168b60 https://gist.github.com/simonw/de4951452df2677f2a1a3cd415168... I'm not going to try for an extensive evaluation comparing it with Llama 3.3 though, life's too short and that's already been done better than I could by https://livebench.ai/ https://livebench.ai/
- MichaelZuo 2y agoWhy not ask it to solve math questions? The bar for GPT-4 was so low that unambiguously clearing that threshold should be pretty easy.
- simonw 2y agoI am not particularly interested in those benchmarks that deliberately expose weaknesses in models: I know that models have weaknesses already! What I care about is the things that they're proven to be good at - can I do those kinds of things (RAG, summarization, code generation, language translation) directly on my laptop?
- ac29 2y agoThe model you are running isnt the one used in the benchmarks you link. The default llama3.3 model in ollama is heavily quantized (~4 bit). Running the full fp16 model, or even an 8-bit quant wouldnt be possible on your laptop with 64G RAM.
- simonw 2y agoThanks - yeah, I should have mentioned that. I just added a note directly above this heading https://simonwillison.net/2024/Dec/9/llama-33-70b/#honorable-mentions https://simonwillison.net/2024/Dec/9/llama-33-70b/#honorable...
- zamadatix 2y ago> when I ran Llama 3.3 70B on the same laptop for the first time. There is no llama 3.3 405B to test, 3.3 only comes in 70B. Are you sure you aren't thinking of llama 3 or 3.1?
- simonw 2y agoNo, I meant Llama 3.3 70B.