5 ms·

Beats Gemini 1.5 Pro at all but two of the listed benchmarks. Google DeepMind is starting to get their bearings in the LLM era. These are the minds behind Alpha

Beats Gemini 1.5 Pro at all but two of the listed benchmarks. Google DeepMind is starting to get their bearings in the LLM era. These are the minds behind AlphaGo/Zero/Fold. They control their own hardware destiny with TPUs. Bullish.

p1esk 2y ago
Are these benchmarks still meaningful?
- maeil 2y ago
  No, and they haven't been for at least half a year. Utterly optimized for by the providers. Nowadays if a model would be SotA for general use but not #1 on any of these benchmarks, I doubt they'd even release it.
- CamperBob2 2y ago
  I've started keeping an eye out for original brainteasers, just for that reason. GCHQ's Christmas puzzle just came out [1], and o1-pro got 6 out of 7 of them right. It took about 20 minutes in total. I wasn't going to bother trying those because I was pretty sure it wouldn't get any of them, but decided to give it an easy one (#4) and was impressed at the CoT. Meanwhile, Google's newest 2.0 Flash model went 0 for 7. 1: https://metro.co.uk/2024/12/11/gchq-christmas-puzzle-2024-released-can-solve-22161098 https://metro.co.uk/2024/12/11/gchq-christmas-puzzle-2024-re...
  - p1esk 2y ago
    Wow! That’s all I need to know about Google’s model.
    - danpalmer 2y ago
      That's a comparison of multiple GPT-4 models working together... against a single GPT-4 mini style model.
      - p1esk 2y ago
        multiple GPT-4 models working together What do you mean? Is o1 not a single model?
    - Workaccount2 2y ago
      What is impressive about this new model is that it is the lightweight version (flash). There will probably be a 2.0 pro (which will be 4o/sonnet class) and maybe an ultra (o1(?)/Opus).
  - nrvn 2y ago
    Did it get the 8 right? The linked article provides the wrong answer btw.
    - CamperBob2 2y ago
      I didn't see a straightforward way to submit the final problem, because I used different contexts for each of the 7 subproblems. Given the right prompt, though, I'm sure it could handle the 'find the corresponding letter from the landmarks to form an anagram' part. That's easier than most of the other problems. You're saying the ultimate answer isn't 'PROTECTING THE UNITED KINGDOM'?
      - nrvn 2y ago
        if you follow the sleigh morse path starting from the robin it will be 'united in protecting the kingdom'.
  - iamdelirium 2y ago
    Why are you comparing flash vs o1-pro, wouldn't a more fair comparison be flash vs mini?
    - iamdelirium 2y ago
      I just ask o1-mini the first two questions and it got it wrong.
    - deleted 2y ago
      [deleted]
    - CamperBob2 2y ago
      It's the only Google model that my account has access to that accepts .PNG files. I assumed it was the latest/greatest experimental 2.0 release. If they want a rematch, they'll need to bring their 'A' game next time, because o1-pro is crazy good.
dagmx 2y ago
Regarding TPU’s, sure for the stuff that’s running on the cloud. However their on device TPUs lag behind the competition and Google still seem to struggle to move significant parts of Gemini to run on device as a result. Of course, Gemini is provided as a subscription service as well so perhaps they’re not incentivized to move things locally. I am curious if they’ll introduce something like Apple’s private cloud compute.
- YetAnotherNick 2y ago
  If the model weights is not open, you can't run it on device anyways.
  - kridsdale1 2y ago
    The Pixel 9 runs many small proprietary Gemini models on the internal TPU.
    - griomnib 2y ago
      And yet these new models still haven’t reached feature parity with Google Assistant, which can turn my flashlight on, but with all the power of burning down a rainforest, Gemini still cannot interact with my actual phone.
      - lern_too_spel 2y ago
        I just tried asking my phone to turn on the flashlight using Gemini. It worked. https://9to5google.com/2024/11/07/gemini-utilities-extension/ https://9to5google.com/2024/11/07/gemini-utilities-extension...
        griomnib 2y ago
        Ok I tried literally last week on Pixel 7a and it didn’t work. What model do you have? Maybe it requires a phone that can do on-device models?
JeremyNT 2y ago
Yeah they've been slow to release end-user facing stuff but it's obvious that they're just grinding away internally. They've ceded the fast mover advantage, but with a massive installed base of Android devices, a team of experts who basically created the entire field, a huge hardware presence (that THEY own), massive legal expertise, existing content deals, and a suite of vertically integrated services, I feel like the game is theirs to lose at this point. The only caution is regulation / anti-trust action, but with a Trump administration that seems far less likely.
VirusNewbie 2y ago
If you look at where talent is going, it's Anthropic that is the real competitor to Google, not OpenAI.

p1esk 2y ago

Are these benchmarks still meaningful?

maeil 2y ago
No, and they haven't been for at least half a year. Utterly optimized for by the providers. Nowadays if a model would be SotA for general use but not #1 on any of these benchmarks, I doubt they'd even release it.
CamperBob2 2y ago
I've started keeping an eye out for original brainteasers, just for that reason. GCHQ's Christmas puzzle just came out [1], and o1-pro got 6 out of 7 of them right. It took about 20 minutes in total. I wasn't going to bother trying those because I was pretty sure it wouldn't get any of them, but decided to give it an easy one (#4) and was impressed at the CoT. Meanwhile, Google's newest 2.0 Flash model went 0 for 7. 1: https://metro.co.uk/2024/12/11/gchq-christmas-puzzle-2024-released-can-solve-22161098 https://metro.co.uk/2024/12/11/gchq-christmas-puzzle-2024-re...
- p1esk 2y ago
  Wow! That’s all I need to know about Google’s model.
  - danpalmer 2y ago
    That's a comparison of multiple GPT-4 models working together... against a single GPT-4 mini style model.
    - p1esk 2y ago
      multiple GPT-4 models working together What do you mean? Is o1 not a single model?
  - Workaccount2 2y ago
    What is impressive about this new model is that it is the lightweight version (flash). There will probably be a 2.0 pro (which will be 4o/sonnet class) and maybe an ultra (o1(?)/Opus).
- nrvn 2y ago
  Did it get the 8 right? The linked article provides the wrong answer btw.
  - CamperBob2 2y ago
    I didn't see a straightforward way to submit the final problem, because I used different contexts for each of the 7 subproblems. Given the right prompt, though, I'm sure it could handle the 'find the corresponding letter from the landmarks to form an anagram' part. That's easier than most of the other problems. You're saying the ultimate answer isn't 'PROTECTING THE UNITED KINGDOM'?
    - nrvn 2y ago
      if you follow the sleigh morse path starting from the robin it will be 'united in protecting the kingdom'.
- iamdelirium 2y ago
  Why are you comparing flash vs o1-pro, wouldn't a more fair comparison be flash vs mini?
  - iamdelirium 2y ago
    I just ask o1-mini the first two questions and it got it wrong.
  - deleted 2y ago
    [deleted]
  - CamperBob2 2y ago
    It's the only Google model that my account has access to that accepts .PNG files. I assumed it was the latest/greatest experimental 2.0 release. If they want a rematch, they'll need to bring their 'A' game next time, because o1-pro is crazy good.

dagmx 2y ago

Regarding TPU’s, sure for the stuff that’s running on the cloud. However their on device TPUs lag behind the competition and Google still seem to struggle to move significant parts of Gemini to run on device as a result. Of course, Gemini is provided as a subscription service as well so perhaps they’re not incentivized to move things locally. I am curious if they’ll introduce something like Apple’s private cloud compute.

YetAnotherNick 2y ago
If the model weights is not open, you can't run it on device anyways.
- kridsdale1 2y ago
  The Pixel 9 runs many small proprietary Gemini models on the internal TPU.
  - griomnib 2y ago
    And yet these new models still haven’t reached feature parity with Google Assistant, which can turn my flashlight on, but with all the power of burning down a rainforest, Gemini still cannot interact with my actual phone.
    - lern_too_spel 2y ago
      I just tried asking my phone to turn on the flashlight using Gemini. It worked. https://9to5google.com/2024/11/07/gemini-utilities-extension/ https://9to5google.com/2024/11/07/gemini-utilities-extension...
      - griomnib 2y ago
        Ok I tried literally last week on Pixel 7a and it didn’t work. What model do you have? Maybe it requires a phone that can do on-device models?

JeremyNT 2y ago

Yeah they've been slow to release end-user facing stuff but it's obvious that they're just grinding away internally. They've ceded the fast mover advantage, but with a massive installed base of Android devices, a team of experts who basically created the entire field, a huge hardware presence (that THEY own), massive legal expertise, existing content deals, and a suite of vertically integrated services, I feel like the game is theirs to lose at this point. The only caution is regulation / anti-trust action, but with a Trump administration that seems far less likely.

VirusNewbie 2y ago

If you look at where talent is going, it's Anthropic that is the real competitor to Google, not OpenAI.