9 ms·

There are two things that I think are needed and that I'm not sure if anyone provides yet to make this scenario work well: 1. Interruption - I need to be able

There are two things that I think are needed and that I'm not sure if anyone provides yet to make this scenario work well: 1. Interruption - I need to be able to say "hang on" and have the LLM pause. 2. Wait for a specific cue before responding. I like "What do you think?" That + low latency are crucial. It needs to feel like talking to another person.

zan2434 3y ago
I agree. Have been working on a 2 way interruptions system + streaming like this. It's not robust yet, but when it works it does feel magical.
philsnow 3y ago
> 2. Wait for a specific cue before responding. I like "What do you think?" "Over."
- dr_kiszonka 3y ago
  "Over and out" closes the app. ;) Saying "Go" to indicate it's the bot's turn would work for me. (Or maybe pressing a button.) The bot should always stop wherever I start speaking.
- m463 3y ago
  > I like "What do you think?" I like it too! And I can't help but think getting into the habit of saying it -- it would help us get along much better with other people in our lives.
bilsbie 3y ago
It would be cool if the Ai could interrupt too.
- andai 3y ago
  "Imma let you finish, but..."
plufz 3y ago
I agree, it is unnatural and a little stressful with current implementations. It feels like I first need to figure out what to say and than say it so I don’t pause and mess up my input. I hope the new improved Siri and Google assistant will be able to chain actions as well. “Ok Google, turn off the lights. Ok Google, stop music.” Feels a bit cumbersome.
- renus 3y ago
  A fast turnaround time is also super important; if the transcription is not correct, waiting multiple seconds for each turn would kill the application. E.g., ordering food using voice is only convenient if it gets me right all the time; if not, I will fall back to the app.
stiffler01 3y ago
Indeed a great point. Waiting for a specific cue, before responding, is an interesting idea. It would make the interaction more natural, especially in situations where the user is thinking aloud or formulating their thoughts before seeking the AI's input. Interruption is something that is already in the pipeline and we are working on it. You should see an update soon.
- localhost 3y ago
  Thanks! Really looking forward to interruptions. I think about the cue as kind of being like "Hey Siri/Alexa/Cortana" but in reverse.
Valgrim 3y ago
In order to feel like a human, cues should not be a pre-programmed phrase, the system should continuously listen to the conversation, and evaluate constantly if speaking is pertinent at that particular moment. Humans will cut a conversation if it's important, and such a system should be able to do the same.
- localhost 3y ago
  Totally agree with your take. But a pre-programmed phrase would work today and hopefully wouldn't be too difficult to implement. I would imagine that higher latency would be more tolerable as well. But in the fullness of time, your approach is better. When I'm listening to someone else talk, I'm already formulating responses or at least an outline of responses in my head. If the LLM could do a progressive summarization of the conversation in real-time as part of its context this would be super cool as well. It could also interrupt you if the LLM self-reflects on the summary and realizes that now would be a good time to interrupt.
pksebben 3y ago
I wrote a sort of toy version of this a little while ago using Vosk and a variety of TTS engines, and the solution that worked mostly-well was to have a buffer that waited for audio that filled until a pause of so many seconds, then it sent that to the LLM. With the implementation of tools for GPT, I could see a way to having the model check if it thinks it received a complete thought, and if it didn't, send back a signal to keep appending to the buffer until the next long pause. The addition of a longer "pregnant pause" timeout could have the model check in to see if you're done talking or whatever.
- renus 3y ago
  To streamline the experience we don't send the transcription to the LLM after the pause, since we are using the time we wait for the end of sentence trigger (pause) to generate the LLM and text-to-speech output. So ideally once we detected the pause, we already processed everything.
lambdaba 3y ago
> Interruption Well, today is your lucky day!: https://persona-webapp-beta.vercel.app/ https://persona-webapp-beta.vercel.app/ and the demo https://smarterchild.chat/ https://smarterchild.chat/
- dmw_ng 3y ago
  The latency on this (or lack thereof) is the best I've seen, would love to know more about how it's achieved. I asked the bot and it claimed you're using Google's speech recognition, which I know supports streaming, but this result seems much lower lag than I remember Google's stuff being capable of
  - selcuka 3y ago
    > I asked the bot and it claimed you're using Google's speech recognition That doesn't sound plausible. How can the LLM part know which speech recognition service is being used?
    - meandmycode 3y ago
      It's not entirely unlikely that the llm is informed exactly what it's source data is, with the hope that it can potentially make corrections to transcription errors
      - lolinder 3y ago
        Or just because it's interesting and people might ask. I could imagine it being a hallucination, but it could also be an easter egg of sorts.
        selcuka 3y ago
        Apparently it uses the Web Speech API [1], not a specific service. [1] https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecog...
irthomasthomas 3y ago
I did a video demo of this. Tell it to only respond only with OK to every message and only respond fully when I tell you I am finished. Ok? Ok.

zan2434 3y ago

I agree. Have been working on a 2 way interruptions system + streaming like this. It's not robust yet, but when it works it does feel magical.

philsnow 3y ago

> 2. Wait for a specific cue before responding. I like "What do you think?" "Over."

dr_kiszonka 3y ago
"Over and out" closes the app. ;) Saying "Go" to indicate it's the bot's turn would work for me. (Or maybe pressing a button.) The bot should always stop wherever I start speaking.
m463 3y ago
> I like "What do you think?" I like it too! And I can't help but think getting into the habit of saying it -- it would help us get along much better with other people in our lives.

bilsbie 3y ago

It would be cool if the Ai could interrupt too.

andai 3y ago
"Imma let you finish, but..."

plufz 3y ago

I agree, it is unnatural and a little stressful with current implementations. It feels like I first need to figure out what to say and than say it so I don’t pause and mess up my input. I hope the new improved Siri and Google assistant will be able to chain actions as well. “Ok Google, turn off the lights. Ok Google, stop music.” Feels a bit cumbersome.

renus 3y ago
A fast turnaround time is also super important; if the transcription is not correct, waiting multiple seconds for each turn would kill the application. E.g., ordering food using voice is only convenient if it gets me right all the time; if not, I will fall back to the app.

stiffler01 3y ago

Indeed a great point. Waiting for a specific cue, before responding, is an interesting idea. It would make the interaction more natural, especially in situations where the user is thinking aloud or formulating their thoughts before seeking the AI's input. Interruption is something that is already in the pipeline and we are working on it. You should see an update soon.

localhost 3y ago
Thanks! Really looking forward to interruptions. I think about the cue as kind of being like "Hey Siri/Alexa/Cortana" but in reverse.

Valgrim 3y ago

In order to feel like a human, cues should not be a pre-programmed phrase, the system should continuously listen to the conversation, and evaluate constantly if speaking is pertinent at that particular moment. Humans will cut a conversation if it's important, and such a system should be able to do the same.

localhost 3y ago
Totally agree with your take. But a pre-programmed phrase would work today and hopefully wouldn't be too difficult to implement. I would imagine that higher latency would be more tolerable as well. But in the fullness of time, your approach is better. When I'm listening to someone else talk, I'm already formulating responses or at least an outline of responses in my head. If the LLM could do a progressive summarization of the conversation in real-time as part of its context this would be super cool as well. It could also interrupt you if the LLM self-reflects on the summary and realizes that now would be a good time to interrupt.

pksebben 3y ago

I wrote a sort of toy version of this a little while ago using Vosk and a variety of TTS engines, and the solution that worked mostly-well was to have a buffer that waited for audio that filled until a pause of so many seconds, then it sent that to the LLM. With the implementation of tools for GPT, I could see a way to having the model check if it thinks it received a complete thought, and if it didn't, send back a signal to keep appending to the buffer until the next long pause. The addition of a longer "pregnant pause" timeout could have the model check in to see if you're done talking or whatever.

renus 3y ago
To streamline the experience we don't send the transcription to the LLM after the pause, since we are using the time we wait for the end of sentence trigger (pause) to generate the LLM and text-to-speech output. So ideally once we detected the pause, we already processed everything.

lambdaba 3y ago

> Interruption Well, today is your lucky day!: https://persona-webapp-beta.vercel.app/ https://persona-webapp-beta.vercel.app/ and the demo https://smarterchild.chat/ https://smarterchild.chat/

dmw_ng 3y ago
The latency on this (or lack thereof) is the best I've seen, would love to know more about how it's achieved. I asked the bot and it claimed you're using Google's speech recognition, which I know supports streaming, but this result seems much lower lag than I remember Google's stuff being capable of
- selcuka 3y ago
  > I asked the bot and it claimed you're using Google's speech recognition That doesn't sound plausible. How can the LLM part know which speech recognition service is being used?
  - meandmycode 3y ago
    It's not entirely unlikely that the llm is informed exactly what it's source data is, with the hope that it can potentially make corrections to transcription errors
    - lolinder 3y ago
      Or just because it's interesting and people might ask. I could imagine it being a hallucination, but it could also be an easter egg of sorts.
      - selcuka 3y ago
        Apparently it uses the Web Speech API [1], not a specific service. [1] https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecog...

irthomasthomas 3y ago

I did a video demo of this. Tell it to only respond only with OK to every message and only respond fully when I tell you I am finished. Ok? Ok.