7 ms·

This is my experience too. Many people in my offline, non-techy life seem to get almost upset when I argue that models are good at many things (and getting bett

by tao_oat 21d ago

This is my experience too. Many people in my offline, non-techy life seem to get almost upset when I argue that models are good at many things (and getting better fast). And I kind of get it! If your primary exposure to AI is Google search summaries (from their cheapest models) and shitty AI graphic design, it makes sense to think it's vastly overhyped.

simianwords 21d ago
Unfortunately it’s not just non technical people, it’s also technical people from my experience. They are usually too caught up in “how it always works” so they can’t think outside of it.
danofsteel32 21d ago
I like how Tyler Cowen puts it: "This is the worst they will ever be". And some people are still stuck on what AI was capable of when GPT-3.5 was SOTA. They don't realize how much they've improved in ~4 years. Even if the rate of improvement slows down, AI will be tremendously more capable in another ~4 years.
- Anamon 17d ago
  Not my experience, an I'm a heavy daily user of frontier models for software development. Have they improved at all in the past ~2 years? Not really, at least not noticeably. We've improved some of the tooling around it, developed some mitigations for the many systemic weaknesses, and wrapped some things in a for loop to iterate faster. The worst they'll ever be? Maybe, but maybe also almost as good as they'll ever be, looking at the sharply diminishing returns. Reactions to every new model seem to always be roughly evenly split between it being better or worse. And what do we have to show for it? We can localise some bug causes faster, and it takes some of the mindless boilerplate work off our hands. Which affects maybe 10-20% of our actual job, and even there occasionally eliminates the gains again right away because you have to babysit or manually redo it, without always knowing when and where that might happen. That's not what I'd call a revolutionary increase in efficiency. And this isn't just my impression, it's what I hear from pretty much all of my professional peers, which includes people waaay more experienced and smarter than me.
  - tao_oat 13d ago
    This is a surprising opinion to me! Subjectively, I very strongly feel that models improved significantly over the last two years, particularly around the time of Opus 4.6 and now Sol 5.6 and such. It's made a huge difference to how much autonomy I can give them. If you trust benchmarks (which is debateable) then the rate of improvement is speeding up -- definitely not diminishing returns.

by tao_oat 21d ago

simianwords 21d ago

Unfortunately it’s not just non technical people, it’s also technical people from my experience. They are usually too caught up in “how it always works” so they can’t think outside of it.

danofsteel32 21d ago

I like how Tyler Cowen puts it: "This is the worst they will ever be". And some people are still stuck on what AI was capable of when GPT-3.5 was SOTA. They don't realize how much they've improved in ~4 years. Even if the rate of improvement slows down, AI will be tremendously more capable in another ~4 years.

Anamon 17d ago
Not my experience, an I'm a heavy daily user of frontier models for software development. Have they improved at all in the past ~2 years? Not really, at least not noticeably. We've improved some of the tooling around it, developed some mitigations for the many systemic weaknesses, and wrapped some things in a for loop to iterate faster. The worst they'll ever be? Maybe, but maybe also almost as good as they'll ever be, looking at the sharply diminishing returns. Reactions to every new model seem to always be roughly evenly split between it being better or worse. And what do we have to show for it? We can localise some bug causes faster, and it takes some of the mindless boilerplate work off our hands. Which affects maybe 10-20% of our actual job, and even there occasionally eliminates the gains again right away because you have to babysit or manually redo it, without always knowing when and where that might happen. That's not what I'd call a revolutionary increase in efficiency. And this isn't just my impression, it's what I hear from pretty much all of my professional peers, which includes people waaay more experienced and smarter than me.
- tao_oat 13d ago
  This is a surprising opinion to me! Subjectively, I very strongly feel that models improved significantly over the last two years, particularly around the time of Opus 4.6 and now Sol 5.6 and such. It's made a huge difference to how much autonomy I can give them. If you trust benchmarks (which is debateable) then the rate of improvement is speeding up -- definitely not diminishing returns.