7 ms·

Gemini-2.5-pro-preview-06-05

349 points by jcuenod 1y ago

jbellis 1y ago
Did it get upgraded in-place again or do you need to opt in to the new model?
unpwn 1y ago
I feel like instead of constantly releasing these preview versions with different dates attached they should just add a patch version and bump that.
- impulser_ 1y ago
  They can't because if someone has built something around that version they don't want to replace that model with a new model that could provide different results.
  - nsriv 1y ago
    Looking at you Anthropic. 4.0 markedly different from 3.7 in my experience.
    - Aeolun 1y ago
      The model name is completely different? How do you accidentally switch from 3.7 to 4.0?
      - nsriv 1y ago
        Sorry, I meant Sonnet 3.7 to Sonnet 4.0.
  - jfoster 1y ago
    In what way are dates better than integers at preventing that kind of mistake?
  - dist-epoch 1y ago
    Except google did exactly that with the previous release, where they silently redirect 03-25 requests to 05-06.
op00to 1y ago
I found Gemini 2.5 Pro highly useful for text summaries, and even reasoning in long conversations... UP TO the last 2 weeks or month. Recently, it seems to totally forget what I'm talking about after 4-5 messages of a paragraph of text each. We're not talking huge amounts of context, but conversational braindeadness. Between ChatGPT's sycophancy, Gemini's forgetfulness and poor attention, I'm just sticking with whatever local model du jour fits my needs and whatever crap my company is paying for today. It's super annoying, hopefully Gemini gets its memory back!
- energy123 1y ago
  I believe it's intentionally nerfed if you use it through the app. Once you use Gemini for a long time you realize they have a number of dark patterns to deter heavy users but maintain the experience for light users. These dark patterns are: - "Something went wrong error" after too many prompts in a day. This was an undocumented rate limit because it never occurs earlier in the day and will immediately disappear if you subscribe for and use a new paid account, but it won't disappear if you make a new free account, and the error going away is strictly tied to how long you wait. Users complained about this for over a year. Of course they lied about the real reasons for this error, and it was never fixed until a few days ago when they rug pulled paying users by introducing actual documented tight rate limits. - "You've been signed out" error if the model has exceeded its output token budget (or runtime duration) for a single inference, so you can't do things like what Anthropic recommends where you coax the model to think longer. - I have less definitive evidence for this but I would not be surprised if they programmatically nerf the reasoning effort parameter for multiturn conversations. I have no other explanation for why the chain of thought fails to generate for small context multiturn chats but will consistently generate for ultra long context singleturn chats.
  - op00to 1y ago
    Right! I feel like it will sail through MBs of text data, but remembering what I said two turns ago is just too much.
- harrisoned 1y ago
  I noticed that same behavior across older Gemini models. I build a chatbot at work around 1.5 Flash, and one day suddenly it was behaving like that. it was perfect before, but after it always saluted the user like it was their first chat, despite me sending the history. And i didn't found any changelog regarding that at the time. After that i moved to OpenAI, Gemini models just seem unreliable on that regard.
jcuenod 1y ago
82.2 on Aider Still actually falling behind the official scores for o3 high. https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- sottol 1y ago
  Does 82.2 correspond to the "Percent correct" of the other models? Not sure if OpenAI has updated O3, but it looks like "pure" o3 (high) has a score of 79.6% in the linked table, "o3 (high) + gpt-4.1" combo has a the highest score of 82.7%. The previous Gemini 2.5 Pro Preview 05-06 (yea, not current 06-05!) was at 76.9%. That looks like a pretty nice bump! But either way, these Aider benchmarks seem to be most useful/trustworthy benchmarks currently and really the only ones I'm paying attention to.
- hobofan 1y ago
  That's the older 05-06 preview, not the new one from today.
  - energy123 1y ago
    They knew that. The 82.2 comes from the new benchmarks in the OP not from the aider url. The aider url was supplied for comparison.
    - hobofan 1y ago
      Ah, thanks for clearing that up!
- vessenes 1y ago
  But so.much.cheaper.and.faster. Pretty amazing.
vthallam 1y ago
As if 3 different preview versions of the same model is not confusing enough, the last two dates are 05-06 and 06-05. They could have held off for a day:)
- tomComb 1y ago
  Since those days are ambiguous anyway, they would have had to hold off until the 13th. In Canada, a third of the dates we see are British, and another third are American, so it’s really confusing. Thankfully y-m-d is now a legal format and seems to be gaining ground.
  - layer8 1y ago
    > they would have had to hold off until the 13th. 06-06 is unambiguously after 05-06 regardless of date format.
    - Sammi 1y ago
      The problems is that I mentally just panic and abort without even trying when I see 06-06 and 05-06. The ambiguity just flips my brain off.
- deleted 1y ago
  [deleted]
- declan_roberts 1y ago
  Engineers are surprisingly bad at naming things!
  - jacob019 1y ago
    I rather like date codes as versions.
    - atom058 1y ago
      But it's not clear how to interpret the date code: 05-06 could be 5th June or 6th May; same sorry for 06-05. Very confusing due to American-style date formatting. Versions number are at least sequential, with a bigger number being a later version.
unsupp0rted 1y ago
Curious to see how this compares to Claude 4 Sonnet in code. This table seems to indicate it's markedly worse? https://blog.google/products/gemini/gemini-2-5-pro-latest-preview/ https://blog.google/products/gemini/gemini-2-5-pro-latest-pr...
- gundmc 1y ago
  Almost all of those benchmarks are coding related. It looks like SWE-Bench is the only one where Claude is higher. Hard to say which benchmark is most representative of actual work. The community seems to like Aider Polyglot from what I've seen
energy123 1y ago
So there's both a 05-06 model and a 06-05 model, and the launch page for 06-05 has some graphs with benchmarks for the 05-06 model but without the 06-05 model?
sergiotapia 1y ago
In Cursor this is called "gemini-2.5-pro-preview-06-05" you have to enable it manually.
johnfn 1y ago
Impressive seeing Google notch up another ~25 ELO on lmarena, on top of the previous #1, which was also Gemini! That being said, I'm starting to doubt the leaderboards as an accurate representation of model ability. While I do think Gemini is a good model, having used both Gemini and Claude Opus 4 extensively in the last couple of weeks I think Opus is in another league entirely. I've been dealing with a number of gnarly TypeScript issues, and after a bit Gemini would spin in circles or actually (I've never seen this before!) give up and say it can't do it. Opus solved the same problems with no sweat. I know that that's a fairly isolated anecdote and not necessarily fully indicative of overall performance, but my experience with Gemini is that it would really want to kludge on code in order to make things work, where I found Opus would tend to find cleaner approaches to the problem. Additionally, Opus just seemed to have a greater imagination? Or perhaps it has been tailored to work better in agentic scenarios? I saw it do things like dump the DOM and inspect it for issues after a particular interaction by writing a one-off playwright script, which I found particularly remarkable. My experience with Gemini is that it tries to solve bugs by reading the code really really hard, which is naturally more limited. Again, I think Gemini is a great model, I'm very impressed with what Google has put out, and until 4.0 came out I would have said it was the best.
- tempusalaria 1y ago
  I agree I find claude easily the best model, at least for programming which is the only thing I use LLMs for
- varunneal 1y ago
  Have you tried o3 on those problems? I've found o3 to be much more impressive than Opus 4 for all of my use cases.
  - johnfn 1y ago
    To be honest, I haven't, because the "This model is extremely expensive" popup on Cursor makes me a bit anxious - but given the accolades here I'll have to give it a shot.
- joshmlewis 1y ago
emehex 1y ago
Is this "kingfall"?
- paisanashapyaar 1y ago
  No, Kingfall is a separate model which is supposed to deliver slightly better performance, around 2.5% to 5% improvement over this.
- Workaccount2 1y ago
  Sundar tweeted a lion so it's probably goldmane. Kingfall is probably their deep think model, and they might wait for O3 pro to drop so they can swing back.
pelorat 1y ago
Why not call it Gemini 2.6?
- MallocVoidstar 1y ago
  Beta, beta, release candidate (this version)
- laweijfmvo 1y ago
  because the plethora of models and versions is getting ridiculous, and for anyone who's not following LLM news daily, you have no clue what to use. There was never a "Google Search 2.6.4 04-13". You just went to google.com and searched.
  - johnfn 1y ago
    Well, Google Search never released an API that millions of people depended on.
    - ZeroTalent 1y ago
      Yes, they did on Google Cloud: "Custom Search JSON API: The primary solution offered by Google is the Custom Search JSON API. This API allows you to create a customized search engine that can search a collection of specified websites. While it's not a direct equivalent to a full-fledged Google Search API, it can be configured to search the entire web." In my experience it's essentially the same as Google Search if configured properly.
  - AISnakeOil 1y ago
    These api models are for developers. Gemini is for consumers.
- Szpadel 1y ago
  next year maybe? they they so not have year in version so they will need to bump the number make sure you can just sort by name
Workaccount2 1y ago
Apparently 06-05 bridges the gap that people were feeling between the 03-25 and 05-06 release[1] [1]https://nitter.net/OfficialLoganK/status/1930657743251349854#m https://nitter.net/OfficialLoganK/status/1930657743251349854...
hu3 1y ago
I pay for both ChatGPT Plus and Gemini Pro. I'm thinking of cancelling my ChatGPT subscription because I keep hitting rate limits. Meanwhile I have yet to hit any rate limit with Gemini/AI Studio.
- oofbaroomf 1y ago
  I think AI Studio uses the API, so rate limits are extremely high and almost impossible for a normal human to reach if using the paid preview model.
  - staticman2 1y ago
    As far as I know AI Studio is always free, even on pay accounts, and you can definetly hit the rate limit.
- Squarex 1y ago
  I much prefer Gemini over chapgpt, but they recently introduced a limit of 100 messages a day on a pro plan :( aistudio is probably still fine
  - MisterPea 1y ago
    I've heard it's only on mobile? I was using gemini for work on desktop for at least 6 hours yesterday (definitely over 100 back and forths) for work and did not get hit with any rate limits Either way, Google's transparency with this is very poor - I saw the limits from a VP's tweet
- fermentation 1y ago
  Is there a reason not to just use the API through openrouter or something?
- HenriNext 1y ago
  AI Studio uses your API account behind the scenes, and it is subject to normal API limits. When you signup for AI Studio, it creates a Google Cloud free tier project with "gen-lang-client-" prefix behind the scenes. You can link a billing account at the bottom of the "get an api key page". Also note that AI studio via default free tier API access doesn't seem to fall within "commercial use" in Google's terms of service, which would mean that your prompts can be reviewed by humans and used for training. All info AFAIK.
tibbar 1y ago
Interesting, I just learned about matharena.ai. Google cherry-picks one result where they're the best here, but in the overall results, it's still O3 and o4-mini-high who are in the lead.
pu_pe 1y ago
I just checked and it looks like the limits for Jules has been bumped from 5 free daily tasks to 60. Not sure it uses the latest model, but I would assume it does
- kristianp 1y ago
  https://jules.google/ https://jules.google/
ChrisArchitect 1y ago
Blog post: https://blog.google/products/gemini/gemini-2-5-pro-latest-preview/ https://blog.google/products/gemini/gemini-2-5-pro-latest-pr... (https://news.ycombinator.com/item?id=44192954 https://news.ycombinator.com/item?id=44192954)
- xnx 1y ago
  That's a much better link
abraxas 1y ago
I found all the previous Gemini models somewhat inferior even compared to Claude 3.7 Sonnet (and much worse than 4) as my coding assistants. I'm keeping an open mind but also not rushing to try this one until some evaluations roll in. I'm actually baffled that the internet at large seems to be very pumped about Gemini but it's not reflective of my personal experience. Not to be that tinfoil hat guy but I smell at least a bit of astroturf activity around Gemini.
- bachmeier 1y ago
  > I'm actually baffled that the internet at large seems to be very pumped about Gemini but it's not reflective of my personal experience. Not to be that tinfoil hat guy but I smell at least a bit of astroturf activity around Gemini. I haven't used Claude, but Gemini has always returned better answers to general questions relative to ChatGPT or Copilot. My impression, which could be wrong, is that Gemini is better in situations that are a substitute for search. How do I do this on the command line, tell me about this product, etc. all give better results, sometimes much better, on Gemini.
  - dist-epoch 1y ago
    You should try Grok then. It's by far the best when searching is required, especially if you enable DeepSearch.
    - Take8435 1y ago
      I don't really want to use the X platform. What's the best alternative? Claude?
    - morgannewman 1y ago
      [dead]
  - praveer13 1y ago
    I’ve honestly had consistently the opposite experiences for general questions. Also for images, Gemini just hallucinates crazily. ChatGPT even on free tier is giving perfectly correct answers, and I’m on Gemini pro. I canceled it yesterday because of this
wiradikusuma 1y ago
I have two issues with Gemini that I don't experience with Claude: 1. It RENAMES VARIABLE NAMES even in places where I don't tell it to change (I pass them just as context). and 2. Sometimes it's missing closing square brackets. Sure I'm a lazy bum, I call the variable "json" instead of "jsonStringForX", but it's contextual (within a closure or function), and I appreciate the feedback, but it makes reviewing the changes difficult (too much noise).
- 93po 1y ago
  i've noticed with ChatGPT is will 100% ignore certain instructions and I wonder if it's just an LLM thing. For example, I can scream and yell in caps at ChatGPT to not use em or en dashes and if anything it makes it use them even more. I've literally never once made it successfully not use them, even when it ignored it the first time, and my follow up is "output the same thing again but NO EM or EN DASHES!" i've not tested this thoroughly, it's just my ancedotal experience over like a dozen attempts.
  - creesch 1y ago
    There are some things so ubiquitous in the training data that it is really difficult to tell models to not so them. Simply because it is so ingrained in their core training. Em dashes are apparently one of those things. It's something I read a lottle while ago in a larger article but can't remember which article it was.
  - tacotime 1y ago
    I wonder if using the character itself in the directions, instead of the name for the character, might help with this. Something like, "Forbidden character list: [—, –]" or "Do NOT use the characters '—' or '–' in any of your output"
  - EnPissant 1y ago
    I have had 95% success rate telling it not to use emdash or semicolon.
- danielbln 1y ago
carbocation 1y ago
Is it possible to know which model version their chat app ( https://gemini.google.com/app https://gemini.google.com/app ) is using?
chollida1 1y ago
I'd start to worry about OpenAI, from a valuation standpoint. The company has some serious competition now and is arguably no longer the leader. its going to be interesting to see how easily they can raise more money. Their valuation is already in the $300B range. How much larger can it get given their relatively paltry revenue at the moment and increasingly rising costs for hardware and electricity. If the next generation of llms needs new data sources, then Facebook and Google seem well positioned there, OpenAI on the other hand seems like its going to lose such race for proprietary data sets as unlike those other two, they don't have another business that generates such data. When they were the leader in both research and in user facing applications they certainly deserved their lofty valuation. What is new money coming into OpenAI getting now? At even a $300B valuation a typical wall street analysts would want to value them at 2x sales which would mean they'd expect OpenAI to have $600B in annual sales to account for this valuation when they go public. Or at an extremely lofty P/E ratio of say 100 that would be $3B in annual earnings, that analysts would have to expect you to double each year for the next 10ish years looking out, ala AMZN in the 2000s, to justify this valuation. They seem to have boxed themselves into a corner where it will be painful to go public, assuming they can ever figure out the nonprofit/profit issue their company has. Congrats to Google here, they have done great work and look like they'll be one of the biggest winners of the AI race.
- ketzo 1y ago
  OpenAI has already forecast $12B in revenue by the end of this year. I agree that Google is well-positioned, but the mindshare/product advantage OpenAI has gives them a stupendous amount of leeway
  - chollida1 1y ago
    Agreed, its the doubling of that each year for the next 4-5 years that I see as being difficult.
  - Workaccount2 1y ago
    The hurdle for OpenAI is going to be on the profit side. Google has their own hardware acceleration and their own data centers. OpenAI has to pay a monopolist for hardware acceleration and beholden to another tech giant for data centers. Never mind that Google can customize it's hardware specifically for it's models. The only way for OpenAI to really get ahead on solid ground is to discover some sort of absolute game changer (new architecture, new algorithm) and manage to keep it bottled away.
lxe 1y ago
Gemini is a good and fast model, but I think the style of code it writes is... amateur / inexperienced. It doesn't make a lot of mistakes typical of an LLM, but rather chooses approaches that are typical of someone who just learned programming. I have to always nudge it to avoid verbosity, keep structure less repetitive, optimize async code, etc. With claude, I rarely have this problem -- it feels more like working with a more experienced developer.
- PantaloonFlames 1y ago
  > I have to always nudge it to avoid verbosity, keep structure less repetitive, optimize async code, etc. Isn’t this what you can do with system instructions?
jdmoreira 1y ago
Is there a no brainer alternative to Claude Code where I can try other models?
- ketzo 1y ago
  People quite like aider! I’m not as much of a fan of the CLI workflow but it’s quite comparable, I think.
  - jdmoreira 1y ago
    I've heard about it but is the outcome as good as claude code?
    - hensybex 1y ago
      In short - it's like comparing Ubuntu with MacOS/Windows Open source power vs corp - if you are eager to do some stuff yourself, aider is probably a better pick (or even a strategic investment); otherwise you'll simply find much more tools that don't require setting them up in Cline/Claude Code But man, if you're on ycombinator.com, how not to stick with open source?
  - kristianp 1y ago
    I enjoy using Aider, but it's not agentic: it cant run your tests for you, for example.
    - geoka9 1y ago
      It sort of can: https://aider.chat/docs/usage/lint-test.html#testing https://aider.chat/docs/usage/lint-test.html#testing
- rubslopes 1y ago
  Roo Code, or Cline. You can allow it to run everything by itself and just watch. I've been preferring to use Copilot agent mode with Sonnet 4, but it asks you to intervene a lot.
- 1y ago
simianwords 1y ago
I feel stupid for asking but how do I enable deepthink?
- koakuma-chan 1y ago
  They added a thinking section in AI studio
  - simianwords 1y ago
    True but it’s greyed out. Not sure if this is “deep think”
    - koakuma-chan 1y ago
      I can set its thinking budget to 32k, that probably is deep think
      - koakuma-chan 1y ago
        Oh nevermind, you need to buy Gemini Ultra to access that https://one.google.com/ai?g1_landing_page=75 https://one.google.com/ai?g1_landing_page=75
johnnyApplePRNG 1y ago
General first impressions are that it's not as capable as 05-06, although it's technically testing better on the leaderboards... interesting.
Alifatisk 1y ago
Finally Google is advertising their ai studio, it's a shame they didn't push that beautiful app before.
consumer451 1y ago
Man, if the benchmarks are to be believed, this is a lifeline for Windsurf as Anthropic becomes less and less friendly. However, in my personal experience Sonnet 3.x has still been king so far. Will be interesting to watch this unfold. At this point, it's still looking grim for Windsurf.
- lexandstuff 1y ago
  Well, they just had a $3B exit, so not that grim, all things considered.
  - consumer451 1y ago
    Yeah, true.. but I just meant for users/user growth. Even if not completely warranted, users in their subreddit are upset that they don't have access to Sonnet 4. With the Claude Max development, non-vibing users seem to be going to Claude Code. This makes me think that maybe Cursor should have taken an exit, cause Claude Code is gonna eat everyone's lunch?
_pdp_ 1y ago
Is it still rate limited though?
zone411 1y ago
Omproves on the Extended NYT Connections benchmark compared to both Gemini 2.5 Pro Exp (03-25) and Gemini 2.5 Pro Preview (05-06), scoring 58.7. The decline observed between 03-25 and 05-06 has been reversed - https://github.com/lechmazur/nyt-connections/ https://github.com/lechmazur/nyt-connections/.
feelingsonice 1y ago
I'm confused by the naming. It advertises itself as "Thinking" so is this the release of the new "Deep Think" model or not?
fallinditch 1y ago
As a Windsurf user I was happy with Claude 3.7 but then switched to Google Gemini 2.5 when Claude started glitching on a particularly large file. It's a bummer that 3.7 has gone from Windsurf - I considered cancelling my Windsurf subscription, but decided not to because it is still good value for money.
- sumedh 1y ago
  No models have gone from WindSurf. Are you talking about Sonnet 4 which never came to Windsurf because Anthropic does not want to support OpenAI?
InTheArena 1y ago
RIght now, the claude code tooling and ChatGPT codex are far better then anything else I have seen for massive code development. Is there a better option out there with Gemini at the heart of it? I noticed the command line codex might support it.
excerionsforte 1y ago
Ok Google, I was deflated after you guys took away 03-25, but now I am happy again with 06-05. Hell yes, we are back baby!
kisamoto 1y ago
Amateur question, how are people using this for coding? Direct chat and copy pasting code? Seems clunky. Or manually switching in cursor? Although is extra cost and not required for a lot of tasks where Cursor tab is faster and good enough. So need to opt in on demand. Cline + open router in VSCode? Something else?
- 4d66ba06 1y ago
  Consider taking a look at Zed, it can use Gemini with an API key and has an agentic “write” mode if you don’t want to copy and paste.
BDivyesh 1y ago
It depends on where and how you use it, I only use the gemini pro model on aistudio, and set the temperature to 0.05 or 0.1 in rare cases I bump it to 0.3 if I need some frontend creativity, it still isn't impressive, I see that claude is still far better, o4-mini-high too. When it comes to o3 I despise it, despite being ranked very high on benchmarks, the best version of it is only available through api.
aienjoyer 1y ago
The truth is that Gemini 2.5 6-05 is a fraud in coding; before, out of 10 codes you wrote, 1 or 2 might not work, meaning they had errors. Now, out of 10 codes, 9 or 10 are wrong. Why does it have so many errors???
- aienjoyer 1y ago
  it have more skill in coding but have a lot of errors, i can't code anything

349 points by jcuenod 1y ago

Did it get upgraded in-place again or do you need to opt in to the new model?

I feel like instead of constantly releasing these preview versions with different dates attached they should just add a patch version and bump that.

impulser_ 1y ago
They can't because if someone has built something around that version they don't want to replace that model with a new model that could provide different results.
- nsriv 1y ago
  Looking at you Anthropic. 4.0 markedly different from 3.7 in my experience.
  - Aeolun 1y ago
    The model name is completely different? How do you accidentally switch from 3.7 to 4.0?
    - nsriv 1y ago
      Sorry, I meant Sonnet 3.7 to Sonnet 4.0.
- jfoster 1y ago
  In what way are dates better than integers at preventing that kind of mistake?
- dist-epoch 1y ago
  Except google did exactly that with the previous release, where they silently redirect 03-25 requests to 05-06.

op00to 1y ago

I found Gemini 2.5 Pro highly useful for text summaries, and even reasoning in long conversations... UP TO the last 2 weeks or month. Recently, it seems to totally forget what I'm talking about after 4-5 messages of a paragraph of text each. We're not talking huge amounts of context, but conversational braindeadness. Between ChatGPT's sycophancy, Gemini's forgetfulness and poor attention, I'm just sticking with whatever local model du jour fits my needs and whatever crap my company is paying for today. It's super annoying, hopefully Gemini gets its memory back!

energy123 1y ago
I believe it's intentionally nerfed if you use it through the app. Once you use Gemini for a long time you realize they have a number of dark patterns to deter heavy users but maintain the experience for light users. These dark patterns are: - "Something went wrong error" after too many prompts in a day. This was an undocumented rate limit because it never occurs earlier in the day and will immediately disappear if you subscribe for and use a new paid account, but it won't disappear if you make a new free account, and the error going away is strictly tied to how long you wait. Users complained about this for over a year. Of course they lied about the real reasons for this error, and it was never fixed until a few days ago when they rug pulled paying users by introducing actual documented tight rate limits. - "You've been signed out" error if the model has exceeded its output token budget (or runtime duration) for a single inference, so you can't do things like what Anthropic recommends where you coax the model to think longer. - I have less definitive evidence for this but I would not be surprised if they programmatically nerf the reasoning effort parameter for multiturn conversations. I have no other explanation for why the chain of thought fails to generate for small context multiturn chats but will consistently generate for ultra long context singleturn chats.
- op00to 1y ago
  Right! I feel like it will sail through MBs of text data, but remembering what I said two turns ago is just too much.
harrisoned 1y ago
I noticed that same behavior across older Gemini models. I build a chatbot at work around 1.5 Flash, and one day suddenly it was behaving like that. it was perfect before, but after it always saluted the user like it was their first chat, despite me sending the history. And i didn't found any changelog regarding that at the time. After that i moved to OpenAI, Gemini models just seem unreliable on that regard.

jcuenod 1y ago

82.2 on Aider Still actually falling behind the official scores for o3 high. https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/

sottol 1y ago
Does 82.2 correspond to the "Percent correct" of the other models? Not sure if OpenAI has updated O3, but it looks like "pure" o3 (high) has a score of 79.6% in the linked table, "o3 (high) + gpt-4.1" combo has a the highest score of 82.7%. The previous Gemini 2.5 Pro Preview 05-06 (yea, not current 06-05!) was at 76.9%. That looks like a pretty nice bump! But either way, these Aider benchmarks seem to be most useful/trustworthy benchmarks currently and really the only ones I'm paying attention to.
hobofan 1y ago
That's the older 05-06 preview, not the new one from today.
- energy123 1y ago
  They knew that. The 82.2 comes from the new benchmarks in the OP not from the aider url. The aider url was supplied for comparison.
  - hobofan 1y ago
    Ah, thanks for clearing that up!
vessenes 1y ago
But so.much.cheaper.and.faster. Pretty amazing.

vthallam 1y ago

As if 3 different preview versions of the same model is not confusing enough, the last two dates are 05-06 and 06-05. They could have held off for a day:)

tomComb 1y ago
Since those days are ambiguous anyway, they would have had to hold off until the 13th. In Canada, a third of the dates we see are British, and another third are American, so it’s really confusing. Thankfully y-m-d is now a legal format and seems to be gaining ground.
- layer8 1y ago
  > they would have had to hold off until the 13th. 06-06 is unambiguously after 05-06 regardless of date format.
  - Sammi 1y ago
    The problems is that I mentally just panic and abort without even trying when I see 06-06 and 05-06. The ambiguity just flips my brain off.
deleted 1y ago
[deleted]
declan_roberts 1y ago
Engineers are surprisingly bad at naming things!
- jacob019 1y ago
  I rather like date codes as versions.
  - atom058 1y ago
    But it's not clear how to interpret the date code: 05-06 could be 5th June or 6th May; same sorry for 06-05. Very confusing due to American-style date formatting. Versions number are at least sequential, with a bigger number being a later version.

unsupp0rted 1y ago

Curious to see how this compares to Claude 4 Sonnet in code. This table seems to indicate it's markedly worse? https://blog.google/products/gemini/gemini-2-5-pro-latest-preview/ https://blog.google/products/gemini/gemini-2-5-pro-latest-pr...

gundmc 1y ago
Almost all of those benchmarks are coding related. It looks like SWE-Bench is the only one where Claude is higher. Hard to say which benchmark is most representative of actual work. The community seems to like Aider Polyglot from what I've seen

energy123 1y ago

So there's both a 05-06 model and a 06-05 model, and the launch page for 06-05 has some graphs with benchmarks for the 05-06 model but without the 06-05 model?

sergiotapia 1y ago

In Cursor this is called "gemini-2.5-pro-preview-06-05" you have to enable it manually.

johnfn 1y ago

Impressive seeing Google notch up another ~25 ELO on lmarena, on top of the previous #1, which was also Gemini! That being said, I'm starting to doubt the leaderboards as an accurate representation of model ability. While I do think Gemini is a good model, having used both Gemini and Claude Opus 4 extensively in the last couple of weeks I think Opus is in another league entirely. I've been dealing with a number of gnarly TypeScript issues, and after a bit Gemini would spin in circles or actually (I've never seen this before!) give up and say it can't do it. Opus solved the same problems with no sweat. I know that that's a fairly isolated anecdote and not necessarily fully indicative of overall performance, but my experience with Gemini is that it would really want to kludge on code in order to make things work, where I found Opus would tend to find cleaner approaches to the problem. Additionally, Opus just seemed to have a greater imagination? Or perhaps it has been tailored to work better in agentic scenarios? I saw it do things like dump the DOM and inspect it for issues after a particular interaction by writing a one-off playwright script, which I found particularly remarkable. My experience with Gemini is that it tries to solve bugs by reading the code really really hard, which is naturally more limited. Again, I think Gemini is a great model, I'm very impressed with what Google has put out, and until 4.0 came out I would have said it was the best.

tempusalaria 1y ago
I agree I find claude easily the best model, at least for programming which is the only thing I use LLMs for
varunneal 1y ago
Have you tried o3 on those problems? I've found o3 to be much more impressive than Opus 4 for all of my use cases.
- johnfn 1y ago
  To be honest, I haven't, because the "This model is extremely expensive" popup on Cursor makes me a bit anxious - but given the accolades here I'll have to give it a shot.
joshmlewis 1y ago

emehex 1y ago

Is this "kingfall"?

paisanashapyaar 1y ago
No, Kingfall is a separate model which is supposed to deliver slightly better performance, around 2.5% to 5% improvement over this.
Workaccount2 1y ago
Sundar tweeted a lion so it's probably goldmane. Kingfall is probably their deep think model, and they might wait for O3 pro to drop so they can swing back.

pelorat 1y ago

Why not call it Gemini 2.6?

MallocVoidstar 1y ago
Beta, beta, release candidate (this version)
laweijfmvo 1y ago
because the plethora of models and versions is getting ridiculous, and for anyone who's not following LLM news daily, you have no clue what to use. There was never a "Google Search 2.6.4 04-13". You just went to google.com and searched.
- johnfn 1y ago
  Well, Google Search never released an API that millions of people depended on.
  - ZeroTalent 1y ago
    Yes, they did on Google Cloud: "Custom Search JSON API: The primary solution offered by Google is the Custom Search JSON API. This API allows you to create a customized search engine that can search a collection of specified websites. While it's not a direct equivalent to a full-fledged Google Search API, it can be configured to search the entire web." In my experience it's essentially the same as Google Search if configured properly.
- AISnakeOil 1y ago
  These api models are for developers. Gemini is for consumers.
Szpadel 1y ago
next year maybe? they they so not have year in version so they will need to bump the number make sure you can just sort by name

Workaccount2 1y ago

Apparently 06-05 bridges the gap that people were feeling between the 03-25 and 05-06 release[1] [1]https://nitter.net/OfficialLoganK/status/1930657743251349854#m https://nitter.net/OfficialLoganK/status/1930657743251349854...

hu3 1y ago

I pay for both ChatGPT Plus and Gemini Pro. I'm thinking of cancelling my ChatGPT subscription because I keep hitting rate limits. Meanwhile I have yet to hit any rate limit with Gemini/AI Studio.

oofbaroomf 1y ago
I think AI Studio uses the API, so rate limits are extremely high and almost impossible for a normal human to reach if using the paid preview model.
- staticman2 1y ago
  As far as I know AI Studio is always free, even on pay accounts, and you can definetly hit the rate limit.
Squarex 1y ago
I much prefer Gemini over chapgpt, but they recently introduced a limit of 100 messages a day on a pro plan :( aistudio is probably still fine
- MisterPea 1y ago
  I've heard it's only on mobile? I was using gemini for work on desktop for at least 6 hours yesterday (definitely over 100 back and forths) for work and did not get hit with any rate limits Either way, Google's transparency with this is very poor - I saw the limits from a VP's tweet
fermentation 1y ago
Is there a reason not to just use the API through openrouter or something?
HenriNext 1y ago
AI Studio uses your API account behind the scenes, and it is subject to normal API limits. When you signup for AI Studio, it creates a Google Cloud free tier project with "gen-lang-client-" prefix behind the scenes. You can link a billing account at the bottom of the "get an api key page". Also note that AI studio via default free tier API access doesn't seem to fall within "commercial use" in Google's terms of service, which would mean that your prompts can be reviewed by humans and used for training. All info AFAIK.

tibbar 1y ago

Interesting, I just learned about matharena.ai. Google cherry-picks one result where they're the best here, but in the overall results, it's still O3 and o4-mini-high who are in the lead.

pu_pe 1y ago

I just checked and it looks like the limits for Jules has been bumped from 5 free daily tasks to 60. Not sure it uses the latest model, but I would assume it does

kristianp 1y ago
https://jules.google/ https://jules.google/

ChrisArchitect 1y ago

Blog post: https://blog.google/products/gemini/gemini-2-5-pro-latest-preview/ https://blog.google/products/gemini/gemini-2-5-pro-latest-pr... (https://news.ycombinator.com/item?id=44192954 https://news.ycombinator.com/item?id=44192954)

xnx 1y ago
That's a much better link

abraxas 1y ago

I found all the previous Gemini models somewhat inferior even compared to Claude 3.7 Sonnet (and much worse than 4) as my coding assistants. I'm keeping an open mind but also not rushing to try this one until some evaluations roll in. I'm actually baffled that the internet at large seems to be very pumped about Gemini but it's not reflective of my personal experience. Not to be that tinfoil hat guy but I smell at least a bit of astroturf activity around Gemini.

bachmeier 1y ago
> I'm actually baffled that the internet at large seems to be very pumped about Gemini but it's not reflective of my personal experience. Not to be that tinfoil hat guy but I smell at least a bit of astroturf activity around Gemini. I haven't used Claude, but Gemini has always returned better answers to general questions relative to ChatGPT or Copilot. My impression, which could be wrong, is that Gemini is better in situations that are a substitute for search. How do I do this on the command line, tell me about this product, etc. all give better results, sometimes much better, on Gemini.
- dist-epoch 1y ago
  You should try Grok then. It's by far the best when searching is required, especially if you enable DeepSearch.
  - Take8435 1y ago
    I don't really want to use the X platform. What's the best alternative? Claude?
  - morgannewman 1y ago
    [dead]
- praveer13 1y ago
  I’ve honestly had consistently the opposite experiences for general questions. Also for images, Gemini just hallucinates crazily. ChatGPT even on free tier is giving perfectly correct answers, and I’m on Gemini pro. I canceled it yesterday because of this

wiradikusuma 1y ago

I have two issues with Gemini that I don't experience with Claude: 1. It RENAMES VARIABLE NAMES even in places where I don't tell it to change (I pass them just as context). and 2. Sometimes it's missing closing square brackets. Sure I'm a lazy bum, I call the variable "json" instead of "jsonStringForX", but it's contextual (within a closure or function), and I appreciate the feedback, but it makes reviewing the changes difficult (too much noise).

93po 1y ago
i've noticed with ChatGPT is will 100% ignore certain instructions and I wonder if it's just an LLM thing. For example, I can scream and yell in caps at ChatGPT to not use em or en dashes and if anything it makes it use them even more. I've literally never once made it successfully not use them, even when it ignored it the first time, and my follow up is "output the same thing again but NO EM or EN DASHES!" i've not tested this thoroughly, it's just my ancedotal experience over like a dozen attempts.
- creesch 1y ago
  There are some things so ubiquitous in the training data that it is really difficult to tell models to not so them. Simply because it is so ingrained in their core training. Em dashes are apparently one of those things. It's something I read a lottle while ago in a larger article but can't remember which article it was.
- tacotime 1y ago
  I wonder if using the character itself in the directions, instead of the name for the character, might help with this. Something like, "Forbidden character list: [—, –]" or "Do NOT use the characters '—' or '–' in any of your output"
- EnPissant 1y ago
  I have had 95% success rate telling it not to use emdash or semicolon.
danielbln 1y ago

carbocation 1y ago

Is it possible to know which model version their chat app ( https://gemini.google.com/app https://gemini.google.com/app ) is using?

chollida1 1y ago

I'd start to worry about OpenAI, from a valuation standpoint. The company has some serious competition now and is arguably no longer the leader. its going to be interesting to see how easily they can raise more money. Their valuation is already in the $300B range. How much larger can it get given their relatively paltry revenue at the moment and increasingly rising costs for hardware and electricity. If the next generation of llms needs new data sources, then Facebook and Google seem well positioned there, OpenAI on the other hand seems like its going to lose such race for proprietary data sets as unlike those other two, they don't have another business that generates such data. When they were the leader in both research and in user facing applications they certainly deserved their lofty valuation. What is new money coming into OpenAI getting now? At even a $300B valuation a typical wall street analysts would want to value them at 2x sales which would mean they'd expect OpenAI to have $600B in annual sales to account for this valuation when they go public. Or at an extremely lofty P/E ratio of say 100 that would be $3B in annual earnings, that analysts would have to expect you to double each year for the next 10ish years looking out, ala AMZN in the 2000s, to justify this valuation. They seem to have boxed themselves into a corner where it will be painful to go public, assuming they can ever figure out the nonprofit/profit issue their company has. Congrats to Google here, they have done great work and look like they'll be one of the biggest winners of the AI race.

ketzo 1y ago
OpenAI has already forecast $12B in revenue by the end of this year. I agree that Google is well-positioned, but the mindshare/product advantage OpenAI has gives them a stupendous amount of leeway
- chollida1 1y ago
  Agreed, its the doubling of that each year for the next 4-5 years that I see as being difficult.
- Workaccount2 1y ago
  The hurdle for OpenAI is going to be on the profit side. Google has their own hardware acceleration and their own data centers. OpenAI has to pay a monopolist for hardware acceleration and beholden to another tech giant for data centers. Never mind that Google can customize it's hardware specifically for it's models. The only way for OpenAI to really get ahead on solid ground is to discover some sort of absolute game changer (new architecture, new algorithm) and manage to keep it bottled away.

lxe 1y ago

Gemini is a good and fast model, but I think the style of code it writes is... amateur / inexperienced. It doesn't make a lot of mistakes typical of an LLM, but rather chooses approaches that are typical of someone who just learned programming. I have to always nudge it to avoid verbosity, keep structure less repetitive, optimize async code, etc. With claude, I rarely have this problem -- it feels more like working with a more experienced developer.

PantaloonFlames 1y ago
> I have to always nudge it to avoid verbosity, keep structure less repetitive, optimize async code, etc. Isn’t this what you can do with system instructions?

jdmoreira 1y ago

Is there a no brainer alternative to Claude Code where I can try other models?

ketzo 1y ago
People quite like aider! I’m not as much of a fan of the CLI workflow but it’s quite comparable, I think.
- jdmoreira 1y ago
  I've heard about it but is the outcome as good as claude code?
  - hensybex 1y ago
    In short - it's like comparing Ubuntu with MacOS/Windows Open source power vs corp - if you are eager to do some stuff yourself, aider is probably a better pick (or even a strategic investment); otherwise you'll simply find much more tools that don't require setting them up in Cline/Claude Code But man, if you're on ycombinator.com, how not to stick with open source?
- kristianp 1y ago
  I enjoy using Aider, but it's not agentic: it cant run your tests for you, for example.
  - geoka9 1y ago
    It sort of can: https://aider.chat/docs/usage/lint-test.html#testing https://aider.chat/docs/usage/lint-test.html#testing
rubslopes 1y ago
Roo Code, or Cline. You can allow it to run everything by itself and just watch. I've been preferring to use Copilot agent mode with Sonnet 4, but it asks you to intervene a lot.
1y ago

simianwords 1y ago

I feel stupid for asking but how do I enable deepthink?

koakuma-chan 1y ago
They added a thinking section in AI studio
- simianwords 1y ago
  True but it’s greyed out. Not sure if this is “deep think”
  - koakuma-chan 1y ago
    I can set its thinking budget to 32k, that probably is deep think
    - koakuma-chan 1y ago
      Oh nevermind, you need to buy Gemini Ultra to access that https://one.google.com/ai?g1_landing_page=75 https://one.google.com/ai?g1_landing_page=75

johnnyApplePRNG 1y ago

General first impressions are that it's not as capable as 05-06, although it's technically testing better on the leaderboards... interesting.

Alifatisk 1y ago

Finally Google is advertising their ai studio, it's a shame they didn't push that beautiful app before.

consumer451 1y ago

Man, if the benchmarks are to be believed, this is a lifeline for Windsurf as Anthropic becomes less and less friendly. However, in my personal experience Sonnet 3.x has still been king so far. Will be interesting to watch this unfold. At this point, it's still looking grim for Windsurf.

lexandstuff 1y ago
Well, they just had a $3B exit, so not that grim, all things considered.
- consumer451 1y ago
  Yeah, true.. but I just meant for users/user growth. Even if not completely warranted, users in their subreddit are upset that they don't have access to Sonnet 4. With the Claude Max development, non-vibing users seem to be going to Claude Code. This makes me think that maybe Cursor should have taken an exit, cause Claude Code is gonna eat everyone's lunch?

_pdp_ 1y ago

Is it still rate limited though?

zone411 1y ago

Omproves on the Extended NYT Connections benchmark compared to both Gemini 2.5 Pro Exp (03-25) and Gemini 2.5 Pro Preview (05-06), scoring 58.7. The decline observed between 03-25 and 05-06 has been reversed - https://github.com/lechmazur/nyt-connections/ https://github.com/lechmazur/nyt-connections/.

feelingsonice 1y ago

I'm confused by the naming. It advertises itself as "Thinking" so is this the release of the new "Deep Think" model or not?

fallinditch 1y ago

As a Windsurf user I was happy with Claude 3.7 but then switched to Google Gemini 2.5 when Claude started glitching on a particularly large file. It's a bummer that 3.7 has gone from Windsurf - I considered cancelling my Windsurf subscription, but decided not to because it is still good value for money.

sumedh 1y ago
No models have gone from WindSurf. Are you talking about Sonnet 4 which never came to Windsurf because Anthropic does not want to support OpenAI?

InTheArena 1y ago

RIght now, the claude code tooling and ChatGPT codex are far better then anything else I have seen for massive code development. Is there a better option out there with Gemini at the heart of it? I noticed the command line codex might support it.

excerionsforte 1y ago

Ok Google, I was deflated after you guys took away 03-25, but now I am happy again with 06-05. Hell yes, we are back baby!

kisamoto 1y ago

Amateur question, how are people using this for coding? Direct chat and copy pasting code? Seems clunky. Or manually switching in cursor? Although is extra cost and not required for a lot of tasks where Cursor tab is faster and good enough. So need to opt in on demand. Cline + open router in VSCode? Something else?

4d66ba06 1y ago
Consider taking a look at Zed, it can use Gemini with an API key and has an agentic “write” mode if you don’t want to copy and paste.

BDivyesh 1y ago

It depends on where and how you use it, I only use the gemini pro model on aistudio, and set the temperature to 0.05 or 0.1 in rare cases I bump it to 0.3 if I need some frontend creativity, it still isn't impressive, I see that claude is still far better, o4-mini-high too. When it comes to o3 I despise it, despite being ranked very high on benchmarks, the best version of it is only available through api.

aienjoyer 1y ago

The truth is that Gemini 2.5 6-05 is a fraud in coding; before, out of 10 codes you wrote, 1 or 2 might not work, meaning they had errors. Now, out of 10 codes, 9 or 10 are wrong. Why does it have so many errors???

aienjoyer 1y ago
it have more skill in coding but have a lot of errors, i can't code anything