17 ms·

I don’t think average user _needs_ to solve frontier challenges. ”Call to Jane”, ”turn on the lights” and ”what’s the weather this afternoon” is more like it I

by fsloth 1mo ago

I don’t think average user _needs_ to solve frontier challenges. ”Call to Jane”, ”turn on the lights” and ”what’s the weather this afternoon” is more like it I would guess. Ofc if the model has some critical bugs that’s another matter.

dzhiurgis 1mo ago
Your examples worked on phones for over a decade. Maybe baking in a model that is "certified" to have some unconditioned truths + rest is pulled from external models/store could make sense. But AFAIK that doesn't exist and I'm not sure it can possibly be made. Perhaps society as a whole at least can work on an open corpus of training data, but I'm not holding my breath on this.
- logicallee 1mo ago
  >Your examples worked on phones for over a decade. Nope. And not only not a decade ago, right now. If you have an Android or iPhone, you can give it clear and easy to understand instructions that Gemma 4 could complete[1] if it had tool calls on it, and that 100.00% of Claude, ChatGPT, Grok, Kimi, you name it, could understand and all complete if they had the access. The phones will fail to complete it. I just tried Siri. I said "hey Siri", waited for Siri to come up, and then I asked one of the exact sentences you replied to: "what's the weather this afternoon?" It thought for around 20 seconds, and said "Something went wrong. Please try again."[2] I have Wifi, I have mobile Internet, I have free storage space, I have up to date software. What went wrong is that phones have never properly connected agents, not ten years ago, not last year, not this year, and probably not next year. But don't settle for what Google could do in 1999 by hotlinking the keyword "weather" in any query to the weather being shown in the results. Tell your phone (any phone): "Please call back the last number that called me that is not an unlisted number, regardless of who it came from." 0 out of any phone will complete that today, tomorrow, a year from now, five years from now, ever, because phone makers are not going to let them do that. Meanwhile, 100% of all frontier agents could complete it if they had tool calls on the phone. Which they don't, and won't ever, thanks to the duopoly. Okay, that's a bit dismissive, I would love to be wrong! [1] after any voice recognition to text - which does work really well on both Android and iPhone! [2] screenshot: https://ibb.co/21rtDnfV https://ibb.co/21rtDnfV
  - narism 1mo ago
    I’m on the IOS 27 beta and Siri did those two tasks (weather/phone) flawlessly. It’s a lot better than it used to be.
    - logicallee 1mo ago
      Thanks for trying that! Very interesting. Can you say this to it: "Hey Siri [wait for it to come up] - please send me an email with the temperature right now so I have it for my records." and see if it can complete the task without any backtalk or misunderstanding, and if you get exactly what you asked for. (It's a really clear request.) Should be 1 statement, no clarification, conversation, random search results, ("Here's what I found!"), etc. A normal frontier model can do that - or Siri can do it if it is properly connected to Claude, ChatGPT, Gemini, Grok, or any other frontier AI - but previously it was never properly connected. If it can do this task, I might have to look into this again. It counts as a success if it sends yourself any email with the current temperature and you actually get it (it can include whatever other text in the email), and a failure if it talks back, says "here's what I found", says it can't, asks you any question, sends you an email that doesn't actually contain the current temperature, just reads you the temperature and then asks if you want it to send an email, etc. Should be 1 shot. let me know if it works!
      - narism 1mo ago
        It brings up a preview of the email and you have to tap or tell it to send it from there but otherwise it worked for this as well. Subject: Current Temperature Body: The current temperature is 27°C in <my city>.
        logicallee 1mo ago
        thanks! useful.
  - fwip 1mo ago
    In my experience, they had a lot of stuff working well in the first few years they rolled out the home voice assistants - Alexa, google home, etc. But for whatever reason, they've spent the last eight(?) years silently breaking things that used to work. Stuff like audiobook playing, music alarms, or even messaging people. Once they started seeing useful (if niche) functionality as a cost center, there wasn't really a world in which these could usefully exist. Their big bet now seems to be that LLMs will lead them to profitability - but whether that's from increased data harvesting, cheaper integrations, or because it'll be useful enough to charge subscription fees, I couldn't tell you.
- gf000 1mo ago
  It barely works even today, like Siri is laughably bad.
  - dzhiurgis 1mo ago
    I mean the examples he gave definitely work. Mostly well I'd say as they are pretty primitive. What Siri is missing is more logical solutions and answers for recipes, etc (still suck even with chatgpt integration).
    - gf000 1mo ago
      Works, sometimes. But they can fail spectacularly and unexpectedly even on very basic questions/instructions, like so simple that a hand-coded word-matching style logic could get them right 20 years ago.
    - darkwater 1mo ago
      No, they don't work. Just asked Siri the other day "what's the weather tomorrow in $LOCATION" (where $LOCATION is a broader zone and not strictly a city) and the answer was the weather in a street called "$LOCATION Avenue" in a city 150km away.
- yunwal 1mo ago
  I just asked Siri "Hey Siri, what's the weather in <nearby town with a generic name> tomorrow" and it gave me a town with the same name ~800 miles from me.

by fsloth 1mo ago

dzhiurgis 1mo ago

Your examples worked on phones for over a decade. Maybe baking in a model that is "certified" to have some unconditioned truths + rest is pulled from external models/store could make sense. But AFAIK that doesn't exist and I'm not sure it can possibly be made. Perhaps society as a whole at least can work on an open corpus of training data, but I'm not holding my breath on this.

logicallee 1mo ago
>Your examples worked on phones for over a decade. Nope. And not only not a decade ago, right now. If you have an Android or iPhone, you can give it clear and easy to understand instructions that Gemma 4 could complete[1] if it had tool calls on it, and that 100.00% of Claude, ChatGPT, Grok, Kimi, you name it, could understand and all complete if they had the access. The phones will fail to complete it. I just tried Siri. I said "hey Siri", waited for Siri to come up, and then I asked one of the exact sentences you replied to: "what's the weather this afternoon?" It thought for around 20 seconds, and said "Something went wrong. Please try again."[2] I have Wifi, I have mobile Internet, I have free storage space, I have up to date software. What went wrong is that phones have never properly connected agents, not ten years ago, not last year, not this year, and probably not next year. But don't settle for what Google could do in 1999 by hotlinking the keyword "weather" in any query to the weather being shown in the results. Tell your phone (any phone): "Please call back the last number that called me that is not an unlisted number, regardless of who it came from." 0 out of any phone will complete that today, tomorrow, a year from now, five years from now, ever, because phone makers are not going to let them do that. Meanwhile, 100% of all frontier agents could complete it if they had tool calls on the phone. Which they don't, and won't ever, thanks to the duopoly. Okay, that's a bit dismissive, I would love to be wrong! [1] after any voice recognition to text - which does work really well on both Android and iPhone! [2] screenshot: https://ibb.co/21rtDnfV https://ibb.co/21rtDnfV
- narism 1mo ago
  I’m on the IOS 27 beta and Siri did those two tasks (weather/phone) flawlessly. It’s a lot better than it used to be.
  - logicallee 1mo ago
    Thanks for trying that! Very interesting. Can you say this to it: "Hey Siri [wait for it to come up] - please send me an email with the temperature right now so I have it for my records." and see if it can complete the task without any backtalk or misunderstanding, and if you get exactly what you asked for. (It's a really clear request.) Should be 1 statement, no clarification, conversation, random search results, ("Here's what I found!"), etc. A normal frontier model can do that - or Siri can do it if it is properly connected to Claude, ChatGPT, Gemini, Grok, or any other frontier AI - but previously it was never properly connected. If it can do this task, I might have to look into this again. It counts as a success if it sends yourself any email with the current temperature and you actually get it (it can include whatever other text in the email), and a failure if it talks back, says "here's what I found", says it can't, asks you any question, sends you an email that doesn't actually contain the current temperature, just reads you the temperature and then asks if you want it to send an email, etc. Should be 1 shot. let me know if it works!
    - narism 1mo ago
      It brings up a preview of the email and you have to tap or tell it to send it from there but otherwise it worked for this as well. Subject: Current Temperature Body: The current temperature is 27°C in <my city>.
      - logicallee 1mo ago
        thanks! useful.
- fwip 1mo ago
  In my experience, they had a lot of stuff working well in the first few years they rolled out the home voice assistants - Alexa, google home, etc. But for whatever reason, they've spent the last eight(?) years silently breaking things that used to work. Stuff like audiobook playing, music alarms, or even messaging people. Once they started seeing useful (if niche) functionality as a cost center, there wasn't really a world in which these could usefully exist. Their big bet now seems to be that LLMs will lead them to profitability - but whether that's from increased data harvesting, cheaper integrations, or because it'll be useful enough to charge subscription fees, I couldn't tell you.
gf000 1mo ago
It barely works even today, like Siri is laughably bad.
- dzhiurgis 1mo ago
  I mean the examples he gave definitely work. Mostly well I'd say as they are pretty primitive. What Siri is missing is more logical solutions and answers for recipes, etc (still suck even with chatgpt integration).
  - gf000 1mo ago
    Works, sometimes. But they can fail spectacularly and unexpectedly even on very basic questions/instructions, like so simple that a hand-coded word-matching style logic could get them right 20 years ago.
  - darkwater 1mo ago
    No, they don't work. Just asked Siri the other day "what's the weather tomorrow in $LOCATION" (where $LOCATION is a broader zone and not strictly a city) and the answer was the weather in a street called "$LOCATION Avenue" in a city 150km away.
yunwal 1mo ago
I just asked Siri "Hey Siri, what's the weather in <nearby town with a generic name> tomorrow" and it gave me a town with the same name ~800 miles from me.