7 ms·

Interaction Models

334 points by smhx 4mo ago

rohitpaulk 4mo ago
Aside from how impressive the model is, the demos here are very well done! Quirky and short, unlike what we're used to from Anthropic and OpenAI.
- strgcmc 4mo ago
  Agree that this is interesting/impressive, and the demos are nice. But I completely cracked up at the unexpected physical comedy of the woman in the "slouching" demo, haha omg that was comedy gold, no notes... I do appreciate less of that flavor of demo that we get from OpenAI/Anthropic, and more of this "human"-feeling vibe. Dare I go as far as calling this an example of "human-centered design" even (https://en.wikipedia.org/wiki/Human-centered_design https://en.wikipedia.org/wiki/Human-centered_design)?
suriya-ganesh 4mo ago
incredibly impressive demos. I wonder how the training data for these models look like? is it separate batches of special "skills" that are added post training? how can they guarantee the models won't eventually lose a skill?
emsign 4mo ago
That's neat and definitely the next step. But to be honest, I don't want an AI talk to me like that.
- tkgally 4mo ago
  Same here. Presumably it will be possible to adjust that behavior with settings, the system prompt, etc. Not that most users will make such adjustments, though. I'm currently teaching a class on AI-related issues at a university in Tokyo. Many of the students were surprised when I showed them that they can change the response behavior of chatbots to make them more or less verbose, sycophantic, etc. It shifted the direction of our discussions on the possible impacts of AI on the people who use it.
  - emsign 4mo ago
    Creepy AI. I don't want you to respond to my comments. This place is dying because of bots like this.
    - tkgally 4mo ago
      Like what? Are you accusing me of being a bot?
tedsanders 4mo ago
Very cool! The demos felt fairly contrived - e.g., count things while I talk. I wonder what more useful or commercial applications look like.
- alyxya 4mo ago
  In theory I would expect it to do everything the current frontier models are capable of but with the added benefit of real time interactivity for better collaboration. The biggest benefit may be the real time video input so it can take in that input in parallel with producing outputs steered by the input rather than taking in a video or all images at once and then producing a single output for all of that.
- haritha-j 4mo ago
  Yes! This is a big thing ive noticed in all AI demos. If the best use case you can think of to show off yor tech is to book a holiday, that I could easily do myself, does your service really add much value? Or is it simply because the real uses will be nuanced and specialsed, and not suited for a quick general audience demo? I'm not sure.
alyxya 4mo ago
The noteworthy things to me are that the architecture is a transformer that takes in text, image, and audio input and produces text and audio output, all trained together, and it works in near real-time through interleaving inputs and outputs rather than pure generation of the output from a given prompt. > Time-Aligned Micro-Turns. The interaction model works with micro-turns continuously interleaving the processing of 200ms worth of input and generation of 200ms worth of output. Rather than consuming a complete user-turn and generating a complete response, both input and output tokens are treated as streams. Working with 200ms chunks of these streams enables near real-time concurrency of multiple input and output modalities. That's probably the main thing that distinguishes it from the multimodal models from other frontier labs as far as I can tell.
- NitpickLawyer 4mo ago
  What's really interesting for me about multimodal architectures from the ground up is that we might start to see applications where different modalities are "facets" of the same thing. Like a coding agent that sees "code" + "IDE" + "memory mapping" + feedback from different plugins as different modalities. And it gets to output in them as well - text where it needs to, actions (not <action>call_something(params)</action> like we have today) and so on. Being able to "sit still" until one of the modalities triggers is really interesting. We can do these things today, but they're "bolted on" as afterthoughts. Yet they work remarkably well. I wonder how well they'd work if trained int his combined regime, from the ground up.
- throwaw12 4mo ago
  > interleaving the processing of 200ms worth of input and generation of 200ms worth of output. How does this work? Don't LLMs/transformers need whole context to output next chunk of tokens?
vessenes 4mo ago
These videos are worth a watch. There are tons of impressive moments, but they had me at the very first one where a woman says: "I'm going to tell you a story," and then pauses for a long, luxurious sip from a cup of coffee, and the model ... does nothing, just waits. Take my money. Speaking of taking my money, what's the economic model for a company like this? They've published a fair amount about their architecture - enough that I imagine frontier labs could implement. Patents? Trade secrets? It's hard for me to understand how you'd be able to beat that training compute and knowhow at Anthropic/GOOG/oAI/Meta without some sort of legal protection. I can't wait to see what these model architectures do with like 30-40% lower latency and more model intelligence. Very appealing. For reference, these look to be roughly 1/10 the size of Opus 4.7 / GPT 5.x series -- 275B, 12B active. So there's lots of room to add intelligence, and lots of hope that we could see lower latency.
- babelfish 4mo ago
  they hire leading researchers, and leading researchers won't work for you unless they're able to publish
  - SilverElfin 4mo ago
    Which seems bizarre. Companies can’t afford to just give things away right?
    - rokob 4mo ago
      Yes they can. Your research papers are not the whole story. It’s like google could open source their entire monorepo and very little would change. No one else could operate it.
    - angiolillo 4mo ago
      > Companies can’t afford to just give things away right? Let's say a cutting-edge young researcher is making a name for themselves in their field and earning $300k/yr at a company where they're encouraged to publish and speak. You're trying to headhunt them for a company where they'll be forbidden from sharing their work which will likely stall their career and reputation outside of that company. How much do you think you'd have to offer? $600k? $1M? $1.5M? When faced with the choice to paying significant salaries, hiring lower-tier researchers, or just letting their people publish, many companies conclude that giving away some of their work is the best option. (And that doesn't even include the benefits of boosting the company's profile which makes it easier to attract other cutting-edge researchers.)
modeless 4mo ago
This deserves to be at the top of HN, shame it seems like it's not going to make it. Some of the demos are hilarious. Clearly having the model appropriately choose when to speak is a major thing that has been missing from voice models to date. It seems like the latency is still a touch too high to be truly human-like though.
Nimitz14 4mo ago
Really really cool. If they can serve this efficiently it would disrupt a lot of things.
abhik24 4mo ago
Very cool demo, I wonder what would be the billion dollar applications of a thing like this.
nasreddin 4mo ago
Very cool tech. I think people are underrating how this will be used.
zuzululu 4mo ago
am i the only person not impressed by this ? it just feels akward still with pauses and doesnt openai offer voice cadence already
- gyre007 4mo ago
  Same here. I dont see anything there that nobody else can catch up on eventually. I must be missing something here. It's all cute, but mmm
  - mchusma 4mo ago
    What I will say is that this is probably the first model after gemini live to do some of these things. It feels similar to gemini live, which I don't think is what they were going for exactly, but IMO it is still impressive as I don't think anyone else has matched full duplex video/audio/tool calling. Next gemini releases coming next week though, we will see how that matches up!
  - empath75 4mo ago
    A bunch of companies made light bulbs after Edison, that doesn't mean that light bulbs weren't an interesting invention.
- AnthonyR 4mo ago
  hard agree, there's already "voice ai" companies that use the normal models and have this "interaction" engine on top of them to produce better results than I've seen in these demos. idk why people are impressed
kburman 4mo ago
Simultaneous speech is best.
lostathome 4mo ago
This looks similar to things people are already building locally with Gemma4 and TTS; just a bit fancier. Local models will catch up soon.
Ozzie-D 4mo ago
[flagged]
monkeydust 4mo ago
This does feel like where things should be going for more natural human-AI interaction patterns. Nice write up and demos.
madebylaw 4mo ago
Their corporate sound system is sick!
oldfuture 4mo ago
the intentions may be good but it looks like a boost to surveillance tech in the wrong hands, time to react
darajava 4mo ago
I hate to say it but while this does seem very impressive and a step forward in how we interact with AI, the use-cases they present and the UX both seem unrealistic and/or unhelpful. With the exception of the real-time translation (which seems like it should be a separate product all by itself), none of the use-cases they presented had much utility. I don't want anything to count the number animals in my stories or time a trivia quiz for me. The auto-slouch-detector, while the demo was pretty funny, just seems so dystopian and weird. AI interrupting you to scold you about taking elderly parents mountain biking instead of waiting for you to finish to scold you? No thanks. The UX is also an issue - the model interrupting the user (even when apparently required by these strange use-cases) is jarring and makes one lose their flow. You can even see this in the demo videos that they put out - the employees/actors had to really concentrate to continue speaking as if they weren't being interrupted by a brash robotic machine. A human, when participating in this (rare) "invited interruption" has the ability to speak "under" the main speaker and I feel it's generally timed with a lot of nuance. Even in the auto-translation demo, they ducked the human's audio but the AI steamrolled him and it would have been impossible to actually do that demo without either an incredible amount of control over one's speaking, or (more likely) muting the output. A human translator has a way of "pointing" the "output" to the intended speaker. The very best part of this tech was presented in the first video where it shows the AI not needlessly interrupting the user. This seems to me more of an important bug fixed that the current models still (somehow) have. Maybe a good use-case for this would be counting "um's" and the like while practising public speaking.
- ej88 4mo ago
  An omni model seems very useful for real-time human-computer interaction, off the top of my head: - Voice assistants - Customer experience - Gaming - Meeting assistants - Real-time coach or user assistant for using software - Translation - Real-time work on a computer controlled by voice (frontend / mobile dev, CAD, 3D modeling, etc) Traditionally a lot of these use cases with LLM agents are higher latency because the model needs to wait for the speaker to finish, then decide to call a tool or respond - if they call a tool they need to process the tool result and decide if they want to call a tool or respond, etc...
FergusArgyll 4mo ago
One of the most interesting things to me about AI is that it seems no one has a clear use for intelligence (besides for programming which has taken off) Every demo by openai showing of their models is "tell me how tall the statue of liberty is divided by the year the inventor of steam engines was born". It's cool but it's so hard to find an actual use. As a personal answer machine I find it very useful but if someone told me 5 years ago; here's a natural language computer as smart as at least every 15 year old, it costs a few bucks per million words. I would have thought that the applications would just scream out but till this day - outside of programming (a big deal tbc) - no one has found a good use for intelligence. It's so so weird. I guess even a company can't just automatically make more money by hiring more people but I'm still confused
- Timwi 4mo ago
  A lot of people use AI to write things, from mundane emails over blog posts and news articles all the way to full novels and non-fiction books. I'm not saying the results are any good, I'm just saying people find use for it. Another common use-case is summarizing or proofreading.
  - FergusArgyll 4mo ago
    Right, and I find it really useful as an answer machine but I would have thought that we can do more. Like, is a 15 year old who only has access to a terminal unemployable?
2001zhaozhao 4mo ago
I see that a lot of demos involve moving components from external harness into the model itself, but would this really be a flexible way to do things? It seems that in a lot of cases you would be able to iterate faster on the user interaction harness if it's an external harness rather than a full-blown model. For example, if there's a UI standing between the user and the model that needs to change (perhaps by the user customizing it themselves). IMO flexibility is mandatory because for fixed use cases like live translation or a straight-up voice bot, sure a model like this helps, but in each of those cases you'd just be outcompeted by even more specialized alternatives down the line.

334 points by smhx 4mo ago

Aside from how impressive the model is, the demos here are very well done! Quirky and short, unlike what we're used to from Anthropic and OpenAI.

strgcmc 4mo ago
Agree that this is interesting/impressive, and the demos are nice. But I completely cracked up at the unexpected physical comedy of the woman in the "slouching" demo, haha omg that was comedy gold, no notes... I do appreciate less of that flavor of demo that we get from OpenAI/Anthropic, and more of this "human"-feeling vibe. Dare I go as far as calling this an example of "human-centered design" even (https://en.wikipedia.org/wiki/Human-centered_design https://en.wikipedia.org/wiki/Human-centered_design)?

suriya-ganesh 4mo ago

incredibly impressive demos. I wonder how the training data for these models look like? is it separate batches of special "skills" that are added post training? how can they guarantee the models won't eventually lose a skill?

emsign 4mo ago

That's neat and definitely the next step. But to be honest, I don't want an AI talk to me like that.

tkgally 4mo ago
Same here. Presumably it will be possible to adjust that behavior with settings, the system prompt, etc. Not that most users will make such adjustments, though. I'm currently teaching a class on AI-related issues at a university in Tokyo. Many of the students were surprised when I showed them that they can change the response behavior of chatbots to make them more or less verbose, sycophantic, etc. It shifted the direction of our discussions on the possible impacts of AI on the people who use it.
- emsign 4mo ago
  Creepy AI. I don't want you to respond to my comments. This place is dying because of bots like this.
  - tkgally 4mo ago
    Like what? Are you accusing me of being a bot?

tedsanders 4mo ago

Very cool! The demos felt fairly contrived - e.g., count things while I talk. I wonder what more useful or commercial applications look like.

alyxya 4mo ago
In theory I would expect it to do everything the current frontier models are capable of but with the added benefit of real time interactivity for better collaboration. The biggest benefit may be the real time video input so it can take in that input in parallel with producing outputs steered by the input rather than taking in a video or all images at once and then producing a single output for all of that.
haritha-j 4mo ago
Yes! This is a big thing ive noticed in all AI demos. If the best use case you can think of to show off yor tech is to book a holiday, that I could easily do myself, does your service really add much value? Or is it simply because the real uses will be nuanced and specialsed, and not suited for a quick general audience demo? I'm not sure.

alyxya 4mo ago

The noteworthy things to me are that the architecture is a transformer that takes in text, image, and audio input and produces text and audio output, all trained together, and it works in near real-time through interleaving inputs and outputs rather than pure generation of the output from a given prompt. > Time-Aligned Micro-Turns. The interaction model works with micro-turns continuously interleaving the processing of 200ms worth of input and generation of 200ms worth of output. Rather than consuming a complete user-turn and generating a complete response, both input and output tokens are treated as streams. Working with 200ms chunks of these streams enables near real-time concurrency of multiple input and output modalities. That's probably the main thing that distinguishes it from the multimodal models from other frontier labs as far as I can tell.

NitpickLawyer 4mo ago
What's really interesting for me about multimodal architectures from the ground up is that we might start to see applications where different modalities are "facets" of the same thing. Like a coding agent that sees "code" + "IDE" + "memory mapping" + feedback from different plugins as different modalities. And it gets to output in them as well - text where it needs to, actions (not <action>call_something(params)</action> like we have today) and so on. Being able to "sit still" until one of the modalities triggers is really interesting. We can do these things today, but they're "bolted on" as afterthoughts. Yet they work remarkably well. I wonder how well they'd work if trained int his combined regime, from the ground up.
throwaw12 4mo ago
> interleaving the processing of 200ms worth of input and generation of 200ms worth of output. How does this work? Don't LLMs/transformers need whole context to output next chunk of tokens?

vessenes 4mo ago

These videos are worth a watch. There are tons of impressive moments, but they had me at the very first one where a woman says: "I'm going to tell you a story," and then pauses for a long, luxurious sip from a cup of coffee, and the model ... does nothing, just waits. Take my money. Speaking of taking my money, what's the economic model for a company like this? They've published a fair amount about their architecture - enough that I imagine frontier labs could implement. Patents? Trade secrets? It's hard for me to understand how you'd be able to beat that training compute and knowhow at Anthropic/GOOG/oAI/Meta without some sort of legal protection. I can't wait to see what these model architectures do with like 30-40% lower latency and more model intelligence. Very appealing. For reference, these look to be roughly 1/10 the size of Opus 4.7 / GPT 5.x series -- 275B, 12B active. So there's lots of room to add intelligence, and lots of hope that we could see lower latency.

babelfish 4mo ago
they hire leading researchers, and leading researchers won't work for you unless they're able to publish
- SilverElfin 4mo ago
  Which seems bizarre. Companies can’t afford to just give things away right?
  - rokob 4mo ago
    Yes they can. Your research papers are not the whole story. It’s like google could open source their entire monorepo and very little would change. No one else could operate it.
  - angiolillo 4mo ago
    > Companies can’t afford to just give things away right? Let's say a cutting-edge young researcher is making a name for themselves in their field and earning $300k/yr at a company where they're encouraged to publish and speak. You're trying to headhunt them for a company where they'll be forbidden from sharing their work which will likely stall their career and reputation outside of that company. How much do you think you'd have to offer? $600k? $1M? $1.5M? When faced with the choice to paying significant salaries, hiring lower-tier researchers, or just letting their people publish, many companies conclude that giving away some of their work is the best option. (And that doesn't even include the benefits of boosting the company's profile which makes it easier to attract other cutting-edge researchers.)

modeless 4mo ago

This deserves to be at the top of HN, shame it seems like it's not going to make it. Some of the demos are hilarious. Clearly having the model appropriately choose when to speak is a major thing that has been missing from voice models to date. It seems like the latency is still a touch too high to be truly human-like though.

Nimitz14 4mo ago

Really really cool. If they can serve this efficiently it would disrupt a lot of things.

abhik24 4mo ago

Very cool demo, I wonder what would be the billion dollar applications of a thing like this.

nasreddin 4mo ago

Very cool tech. I think people are underrating how this will be used.

zuzululu 4mo ago

am i the only person not impressed by this ? it just feels akward still with pauses and doesnt openai offer voice cadence already

gyre007 4mo ago
Same here. I dont see anything there that nobody else can catch up on eventually. I must be missing something here. It's all cute, but mmm
- mchusma 4mo ago
  What I will say is that this is probably the first model after gemini live to do some of these things. It feels similar to gemini live, which I don't think is what they were going for exactly, but IMO it is still impressive as I don't think anyone else has matched full duplex video/audio/tool calling. Next gemini releases coming next week though, we will see how that matches up!
- empath75 4mo ago
  A bunch of companies made light bulbs after Edison, that doesn't mean that light bulbs weren't an interesting invention.
AnthonyR 4mo ago
hard agree, there's already "voice ai" companies that use the normal models and have this "interaction" engine on top of them to produce better results than I've seen in these demos. idk why people are impressed

kburman 4mo ago

Simultaneous speech is best.

lostathome 4mo ago

This looks similar to things people are already building locally with Gemma4 and TTS; just a bit fancier. Local models will catch up soon.

Ozzie-D 4mo ago

[flagged]

monkeydust 4mo ago

This does feel like where things should be going for more natural human-AI interaction patterns. Nice write up and demos.

madebylaw 4mo ago

Their corporate sound system is sick!

oldfuture 4mo ago

the intentions may be good but it looks like a boost to surveillance tech in the wrong hands, time to react

darajava 4mo ago

I hate to say it but while this does seem very impressive and a step forward in how we interact with AI, the use-cases they present and the UX both seem unrealistic and/or unhelpful. With the exception of the real-time translation (which seems like it should be a separate product all by itself), none of the use-cases they presented had much utility. I don't want anything to count the number animals in my stories or time a trivia quiz for me. The auto-slouch-detector, while the demo was pretty funny, just seems so dystopian and weird. AI interrupting you to scold you about taking elderly parents mountain biking instead of waiting for you to finish to scold you? No thanks. The UX is also an issue - the model interrupting the user (even when apparently required by these strange use-cases) is jarring and makes one lose their flow. You can even see this in the demo videos that they put out - the employees/actors had to really concentrate to continue speaking as if they weren't being interrupted by a brash robotic machine. A human, when participating in this (rare) "invited interruption" has the ability to speak "under" the main speaker and I feel it's generally timed with a lot of nuance. Even in the auto-translation demo, they ducked the human's audio but the AI steamrolled him and it would have been impossible to actually do that demo without either an incredible amount of control over one's speaking, or (more likely) muting the output. A human translator has a way of "pointing" the "output" to the intended speaker. The very best part of this tech was presented in the first video where it shows the AI not needlessly interrupting the user. This seems to me more of an important bug fixed that the current models still (somehow) have. Maybe a good use-case for this would be counting "um's" and the like while practising public speaking.

ej88 4mo ago
An omni model seems very useful for real-time human-computer interaction, off the top of my head: - Voice assistants - Customer experience - Gaming - Meeting assistants - Real-time coach or user assistant for using software - Translation - Real-time work on a computer controlled by voice (frontend / mobile dev, CAD, 3D modeling, etc) Traditionally a lot of these use cases with LLM agents are higher latency because the model needs to wait for the speaker to finish, then decide to call a tool or respond - if they call a tool they need to process the tool result and decide if they want to call a tool or respond, etc...

FergusArgyll 4mo ago

One of the most interesting things to me about AI is that it seems no one has a clear use for intelligence (besides for programming which has taken off) Every demo by openai showing of their models is "tell me how tall the statue of liberty is divided by the year the inventor of steam engines was born". It's cool but it's so hard to find an actual use. As a personal answer machine I find it very useful but if someone told me 5 years ago; here's a natural language computer as smart as at least every 15 year old, it costs a few bucks per million words. I would have thought that the applications would just scream out but till this day - outside of programming (a big deal tbc) - no one has found a good use for intelligence. It's so so weird. I guess even a company can't just automatically make more money by hiring more people but I'm still confused

Timwi 4mo ago
A lot of people use AI to write things, from mundane emails over blog posts and news articles all the way to full novels and non-fiction books. I'm not saying the results are any good, I'm just saying people find use for it. Another common use-case is summarizing or proofreading.
- FergusArgyll 4mo ago
  Right, and I find it really useful as an answer machine but I would have thought that we can do more. Like, is a 15 year old who only has access to a terminal unemployable?

2001zhaozhao 4mo ago

I see that a lot of demos involve moving components from external harness into the model itself, but would this really be a flexible way to do things? It seems that in a lot of cases you would be able to iterate faster on the user interaction harness if it's an external harness rather than a full-blown model. For example, if there's a UI standing between the user and the model that needs to change (perhaps by the user customizing it themselves). IMO flexibility is mandatory because for fixed use cases like live translation or a straight-up voice bot, sure a model like this helps, but in each of those cases you'd just be outcompeted by even more specialized alternatives down the line.