6 ms·

I was using gemini 2.5 pro yesterday and it does seem decent. I still think claude 3.5 is better at following instruction then the new 3.7 model which just goes

by neal_ 1y ago

I was using gemini 2.5 pro yesterday and it does seem decent. I still think claude 3.5 is better at following instruction then the new 3.7 model which just goes ham messing stuff up. Really disappointed by Cursor and the Claude CLI tool, for me they create more problems then fix. I cant figure out how to use them on any of my projects with out them ruining the project and creating terrible tech debt. I really like the way gemini shows how much context window is left, i think every company should have this. To be honest i think there has been no major improvement beyond the original models which gained popularity first. Its just marginal improvements 10% better or something, and the free models like deepseek are actually better imo then anything openai has. I dont think the market can withstand the valuations of the big ai companies. They have no advantage, there models suck worse then free open source ones, and they charge money??? Where is the benefit to there product?? People originally said the models are the moat and methods are top secret, but turns out its pretty easy to reproduce these models, and its the application layer built on top of the models that is much more specific and has the real moat. People said the models would engulf these applications built ontop and just integrate natively.

cjonas 1y ago
My only experience is via cursor but I'd agree in that context 3.7 is worse than 3.5. 3.7 goes crazy trying to fix any little linter errors and often gets confused and will just hammer away, making things worse until I stop generation. I think if I let it continue it would probably proposed rm -rf and start over at some point :). Again, this could just have to do with the way cursor is prompting it.
- travisgriggs 1y ago
  So glad to see this!! I thought it was just me! The latest updates, I’m often like “would you just hold the f#^^ on trigger?!? Take a chill pill already”
- runekaagaard 1y ago
  I'm getting great and stable results with 3.7 on Claude desktop and mcp servers. It feels like an upgrade from 3.5
- theshrike79 1y ago
  I asked claude 3.7 to move a perfectly working module to another location. What did it do? A COMPLETE FUCKING REWRITE OF THE MODULE. The result did work, because of unit tests etc. but still, it has a habit of going down the rabbit hole of fixing and changing 42 different things when you ask for one change.
- heed 1y ago
  believe it or not, i had cursor in yolo mode just for fun recently and 3.7 rm -rf'd my home folder :(
  - neal_ 1y ago
    thats crazy! I haven't heard of yolo mode?? dont they like restrict access to the project? but i guess the terminal is unrestricted? lol i wonder what it was trying to do
    - heed 1y ago
      it had created a config file in my home dir and i asked it to move it to the project folder and apparently it thought deleting the entire home dir first was necessary? not sure because after my home folder was gone things started disappearing lol
vlovich123 1y ago
Have you tried wind surf? I’ve been really enjoying it and wondering if they do something on top to make it work better. The AI definitely still gets into weird rabbit holes and sometimes even injects security bugs (kept trying to add sandbox permissions for an iframe), but at least for UI work it’s been an accelerant.
mountainriver 1y ago
My whole team feels like 3.7 is a letdown. It really struggles to follow instructions as others are mentioning. Makes me think they really just hacked the benchmarks on this one.
- ignoramous 1y ago
  Claude Sonnet 3.7 Thinking is also an unmitigated disaster for coding. I was mistaken that a "thinking" model would be better at logic. It turns out "thinking" is a marketing term, a euphemism for "hallucinating" ... though, not unsurprising when you actually take a look at the model cards for these "reasoning" / "thinking" LLMs; however, I've found these to work nicely for IR (information retrieval).
  - theshrike79 1y ago
    Overthinking without extra input is always bad. It's super bad for humans too. You start to spiral down a dark path when your thoughts run away and make up theories and base more theories on those etc.
- dimitri-vs 1y ago
  They definitely over-optimized it for agentic use - where the quality of the code doesn't matter as much as it's ability to run, even if just barely. When you view it from that perspective all that nested errors handling, excessive comments, 10 lines that can be done in 2, etc. start to make sense.
martin-t 1y ago
Whenever I read about LLMs or try to use them, I feel like I am asleep in a dream where two contradicting things can be true at the same time. On one hand, you have people claiming "AI" can now do SWE tasks which take humans 30 minutes or 2 hours and the time doubles every X months so by Y year, SW development will be completely automated. On the other hand, you have people saying exactly what you are saying. Usually that LLMs have issues even with small tasks and that repeated/prolonged use generates tech debt even if they succeed on the small tasks. These 2 views clearly can't both be true at the same time. My experience is the second category so I'd like to chalk up the first as marketing hype but it's confusing how many people who have seemingly nothing to gain from the hype contribute to it.
- aleph_minus_one 1y ago
  > Whenever I read about LLMs or try to use them, I feel like I am asleep in a dream where two contradicting things can be true at the same time. This is called "paraconsistent logic": * https://en.wikipedia.org/wiki/Paraconsistent_logic https://en.wikipedia.org/wiki/Paraconsistent_logic * https://plato.stanford.edu/entries/logic-paraconsistent/ https://plato.stanford.edu/entries/logic-paraconsistent/
- radicality 1y ago
  At first thought you are gonna talk about how various LLMs will gaslight you, and say something is true, then only change their mind once you provide a counter example and when challenged with it, will respond “I obviously meant it’s mostly true, in that specific case it’s false”.
- frankohn 1y ago
  > people claiming "AI" can now do SWE tasks which take humans 30 minutes or 2 hours Yes people claim that but everyone with a grain of salt in his mind know this is not true. Yes, in some cases an LLM can write from scratch a python or web demo-like application and that looks impressive but it is still far from really replacing a SWE. Real world is messy and requires to be careful. It requires to plan, do some modifications, get some feedback, proceed or go back to the previous step, think about it again. Even when a change works you still need to go back to the previous step, double check, make improvements, remove stuff, fix errors, treat corner cases. The LLM doesn't do this, it tries to do everything in one single step. Yes, even when it is in "thinking" mode, in thinks ahead and explore a few possibilities but it doesn't do several iterations as it would be needed in many cases. It does a first write like a brilliant programmers may do in one attempt but it doesn't review its work. The idea of feeding back the error to the LLM so that it will fix it works in simple cases but in most common cases, where things are more complex, that leads to catastrophes. Also when dealing with legacy code it is much more difficult for an LLM because it has to cope with the existing code with all its idiosincracies. One need in this case a deep understanding of what the code is doing and some well-thought planning to modify it without breaking everything and the LLM is usually bad as that. In short, LLM are a wonderful technology but they are not yet the silver bullet someone pretends it to be. Use it like an assistant to help you on specific tasks where the scope is small the the requirements well-defined, this is the domain where it does excel and is actually useful. You can also use it to give you a good starting point in a domain you are nor familiar or it can give you some good help when you are stuck on some problem. Attempt to give the LLM a stack to big or complex are doomed to failure and you will be frustrated and lose your time.

by neal_ 1y ago

cjonas 1y ago

My only experience is via cursor but I'd agree in that context 3.7 is worse than 3.5. 3.7 goes crazy trying to fix any little linter errors and often gets confused and will just hammer away, making things worse until I stop generation. I think if I let it continue it would probably proposed rm -rf and start over at some point :). Again, this could just have to do with the way cursor is prompting it.

travisgriggs 1y ago
So glad to see this!! I thought it was just me! The latest updates, I’m often like “would you just hold the f#^^ on trigger?!? Take a chill pill already”
runekaagaard 1y ago
I'm getting great and stable results with 3.7 on Claude desktop and mcp servers. It feels like an upgrade from 3.5
theshrike79 1y ago
I asked claude 3.7 to move a perfectly working module to another location. What did it do? A COMPLETE FUCKING REWRITE OF THE MODULE. The result did work, because of unit tests etc. but still, it has a habit of going down the rabbit hole of fixing and changing 42 different things when you ask for one change.
heed 1y ago
believe it or not, i had cursor in yolo mode just for fun recently and 3.7 rm -rf'd my home folder :(
- neal_ 1y ago
  thats crazy! I haven't heard of yolo mode?? dont they like restrict access to the project? but i guess the terminal is unrestricted? lol i wonder what it was trying to do
  - heed 1y ago
    it had created a config file in my home dir and i asked it to move it to the project folder and apparently it thought deleting the entire home dir first was necessary? not sure because after my home folder was gone things started disappearing lol

vlovich123 1y ago

Have you tried wind surf? I’ve been really enjoying it and wondering if they do something on top to make it work better. The AI definitely still gets into weird rabbit holes and sometimes even injects security bugs (kept trying to add sandbox permissions for an iframe), but at least for UI work it’s been an accelerant.

mountainriver 1y ago

My whole team feels like 3.7 is a letdown. It really struggles to follow instructions as others are mentioning. Makes me think they really just hacked the benchmarks on this one.

ignoramous 1y ago
Claude Sonnet 3.7 Thinking is also an unmitigated disaster for coding. I was mistaken that a "thinking" model would be better at logic. It turns out "thinking" is a marketing term, a euphemism for "hallucinating" ... though, not unsurprising when you actually take a look at the model cards for these "reasoning" / "thinking" LLMs; however, I've found these to work nicely for IR (information retrieval).
- theshrike79 1y ago
  Overthinking without extra input is always bad. It's super bad for humans too. You start to spiral down a dark path when your thoughts run away and make up theories and base more theories on those etc.
dimitri-vs 1y ago
They definitely over-optimized it for agentic use - where the quality of the code doesn't matter as much as it's ability to run, even if just barely. When you view it from that perspective all that nested errors handling, excessive comments, 10 lines that can be done in 2, etc. start to make sense.

martin-t 1y ago

Whenever I read about LLMs or try to use them, I feel like I am asleep in a dream where two contradicting things can be true at the same time. On one hand, you have people claiming "AI" can now do SWE tasks which take humans 30 minutes or 2 hours and the time doubles every X months so by Y year, SW development will be completely automated. On the other hand, you have people saying exactly what you are saying. Usually that LLMs have issues even with small tasks and that repeated/prolonged use generates tech debt even if they succeed on the small tasks. These 2 views clearly can't both be true at the same time. My experience is the second category so I'd like to chalk up the first as marketing hype but it's confusing how many people who have seemingly nothing to gain from the hype contribute to it.

aleph_minus_one 1y ago
> Whenever I read about LLMs or try to use them, I feel like I am asleep in a dream where two contradicting things can be true at the same time. This is called "paraconsistent logic": * https://en.wikipedia.org/wiki/Paraconsistent_logic https://en.wikipedia.org/wiki/Paraconsistent_logic * https://plato.stanford.edu/entries/logic-paraconsistent/ https://plato.stanford.edu/entries/logic-paraconsistent/
radicality 1y ago
At first thought you are gonna talk about how various LLMs will gaslight you, and say something is true, then only change their mind once you provide a counter example and when challenged with it, will respond “I obviously meant it’s mostly true, in that specific case it’s false”.
frankohn 1y ago
> people claiming "AI" can now do SWE tasks which take humans 30 minutes or 2 hours Yes people claim that but everyone with a grain of salt in his mind know this is not true. Yes, in some cases an LLM can write from scratch a python or web demo-like application and that looks impressive but it is still far from really replacing a SWE. Real world is messy and requires to be careful. It requires to plan, do some modifications, get some feedback, proceed or go back to the previous step, think about it again. Even when a change works you still need to go back to the previous step, double check, make improvements, remove stuff, fix errors, treat corner cases. The LLM doesn't do this, it tries to do everything in one single step. Yes, even when it is in "thinking" mode, in thinks ahead and explore a few possibilities but it doesn't do several iterations as it would be needed in many cases. It does a first write like a brilliant programmers may do in one attempt but it doesn't review its work. The idea of feeding back the error to the LLM so that it will fix it works in simple cases but in most common cases, where things are more complex, that leads to catastrophes. Also when dealing with legacy code it is much more difficult for an LLM because it has to cope with the existing code with all its idiosincracies. One need in this case a deep understanding of what the code is doing and some well-thought planning to modify it without breaking everything and the LLM is usually bad as that. In short, LLM are a wonderful technology but they are not yet the silver bullet someone pretends it to be. Use it like an assistant to help you on specific tasks where the scope is small the the requirements well-defined, this is the domain where it does excel and is actually useful. You can also use it to give you a good starting point in a domain you are nor familiar or it can give you some good help when you are stuck on some problem. Attempt to give the LLM a stack to big or complex are doomed to failure and you will be frustrated and lose your time.