11 ms·

Show HN: Real-time AI Voice Chat at ~500ms Latency

524 points by koljab 1y ago

purplezooey 1y ago
I had been working on something like it when I came across this. Excellent work. Love the demo.
koljab 1y ago
I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and RealtimeTTS (supporting engines like Coqui XTTSv2/Kokoro) to achieve around 500ms response latency, even when running larger local models like a 24B Mistral fine-tune via Ollama. Key aspects: Designed for local LLMs (Ollama primarily, OpenAI connector included). Interruptible conversation. Smart turn detection to avoid cutting the user off mid-thought. Dockerized setup available for easier dependency management. It requires a decent CUDA-enabled GPU for good performance due to the STT/TTS models. Would love to hear your feedback on the approach, performance, potential optimizations, or any features you think are essential for a good local voice AI experience. The code is here: https://github.com/KoljaB/RealtimeVoiceChat https://github.com/KoljaB/RealtimeVoiceChat
- pzo 1y ago
  This looks great will definitely have a look. I'm just wondering if you tested fastRTC from hugging face? I haven't done that curious about speed between this vs fastrtc vs pipecat.
  - koljab 1y ago
    Yes, I tested it. I'm not that sure what they created there. It adds some noticable latency compared towards using raw websockets. Imho it's not supposed to, but it did it nevertheless in my tests.
- ivape 1y ago
  Would you say you are using the best-in-class speech to text libs at the moment? I feel like this space is moving fast because the last time I was headed down this track, I was sure whisper-cpp was the best.
  - koljab 1y ago
    I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to evaluate these latest models, they look strong.
    - ivape 1y ago
      Yeah, I figured you would know. Thanks for that, bookmarking that asr leaderboard.
    - kristopolous 1y ago
      https://yummy-fir-7a4.notion.site/dia https://yummy-fir-7a4.notion.site/dia is the new hotness.
      - koljab 1y ago
        Tried that one. Quality is great but sometimes generations fail and it's rather slow. Also needs ~13 GB of VRAM, it's not my first choice for voice agents tbh.
        kristopolous 1y ago
        alright, dumb question. (1) I assume these things can do multiple languages (2) Given (1), can you strip all the languages you aren't using and speed things up?
        koljab 1y ago
        Actually good question. I'd say probably not. You can't easily "unlearn" things from the model weights (and even if this alone doesn't help). You could retrain/finetune the model heavily on a single language but again that alone does not speed up inference. To gain speed you'd have to bring the parameter count down and train the model from scratch with a single language only. That might work but it's also quite probable that it introduces other issues in the synthesis. In a perfect world the model would only use all that "free parameters" not used now for other languages for a better synthesis of that single trained language. Might be true to a certain degree, but it's not exactly how ai parameter scaling works.
- dotancohen 1y ago
  This looks great. What hardware do you use, or have you tested it on?
  - koljab 1y ago
    I only tested it on my 4090 so far
    - echelon 1y ago
      Are you using all local models, or does it also use cloud inference? Proprietary models? Which models are running in which places? Cool utility!
      - koljab 1y ago
        All local models: - VAD: Webrtcvad (first fast check) followed by SileroVAD (high compute verification) - Transcription: base.en whisper (CTranslate2) - Turn Detection: KoljaB/SentenceFinishedClassification (selftrained BERT-model) - LLM: hf.co/bartowski/huihui-ai_Mistral-Small-24B-Instruct-2501-abliterated-GGUF:Q4_K_M (easily switchable) - TTS: Coqui XTTSv2, switchable to Kokoro or Orpheus (this one is slower)
        echelon 1y ago
        That's excellent. Really amazing bringing all of these together like this. Hopefully we get an open weights version of Sesame [1] soon. Keep watching for it, because that'd make a killer addition to your app. [1] https://www.sesame.com/ https://www.sesame.com/
        koljab 1y ago
        That would be absolutely awesome. But I doubt it, since they released a shitty version of that amazing thing they put online. I feel they aren't planning to give us their top model soon.
- zaggynl 1y ago
  Neat! I'm already using openwebui/ollama with a 7900 xtx but the STT and TTS parts don't seem to work with it yet: 2025-05-05 20:53:15,808] [WARNING] [real_accelerator.py:194:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it. Error loading model for checkpoint ./models/Lasinya: This op had not been implemented on CPU backend.
  - dankwizard 1y ago
    I've given up trying to locally use LLMs on AMD
    - lhl 1y ago
      Basically anything llama.cpp (Vulkan backend) should work out of the box w/o much fuss (LM Studio, Ollama, etc). The HIP backend can have a big prefill speed boost on some architectures (high-end RDNA3 for example). For everything else, I keep notes here: https://llm-tracker.info/howto/AMD-GPUs https://llm-tracker.info/howto/AMD-GPUs
- dummydummy1234 1y ago
  Have you looked at pipecat, seems to be similar trying to do standardized backend/webrtc turn detection pipelines.
  - koljab 1y ago
    Did not look into that one. Looks quite good, I will try that soon.
- peterldowns 1y ago
  Can you explain more about the "Coqui XTTS Lasinya" models that the code is using? What are these, and how were they trained/finetuned? I'm assuming you're the one who uploaded them to huggingface, but there's no model card or README https://huggingface.co/KoljaB/XTTS_Models https://huggingface.co/KoljaB/XTTS_Models In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/audio_module.py#L106 https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...
  - Buckaroo9 1y ago
    https://huggingface.co/coqui/XTTS-v2 https://huggingface.co/coqui/XTTS-v2
  - optimog 1y ago
    Seems like they are out of business. Their homepage mentions "Coqui is schutting down"* That is probably the reason you can't find that much. *https://coqui.ai/ https://coqui.ai/
  - wkat4242 1y ago
    Yeah I really dislike the whisperiness of this voice "Lasinya". It sounds too much like an erotic phone service. I wonder if there's any alternative voice? I don't see Lasinya even mentioned in the public coqui models: https://github.com/coqui-ai/STT-models/releases https://github.com/coqui-ai/STT-models/releases . But I don't see a list of other model names I could use either. I tried to select kokoro in the python module but it says in the logs that only coqui is available. I do have to say the coqui models sound really good, it's just the type of voice that puts me off. The default prompt is also way too "girlfriendy" but that was easily fixed. But for the voice, I simply don't know what the other options are for this engine. PS: Forgive my criticism of the default voice but I'm really impressed with the responsiveness of this. It really responds so fast. Thanks for making this!
    - koljab 1y ago
      Yeah I know the voice polarizes, I trained it for myself, so it's not an official release. You can change the voice here: https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/audio_module.py#L108 https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a... Create a subfolder in the app container: ./models/some_folder_name Copy the files from your desired voice into that folder: config.json, model.pth, vocab.json and speakers_xtts.pth (you can copy the speakers_xtts.pth from Lasinya, it's the same for every voice) Then change the specific_model="Lasinya" line in audio_module.py into specific_model="some_folder_name". If you change TTS_START_ENGINE to "kokoro" in server.py it's supposed to work, what does happen then? Can you post the log message?
- karimf 1y ago
  Do you have any information on how long each step take? Like how many ms on each step of the pipeline? I'm curious how fast it will run if we can get this running on a Mac. Any ballpark guess?
  - koljab 1y ago
    LLM and TTS latency get's determined and logged at the start. It's around 220ms for the LLM returning the first synthesizable sentence fragment (depending on the length of the fragment, which is usually something between 3 and 10 words). Then around 80ms of TTS until the first audio chunk is delivered. STT with base.en you can neglect, it's under 5 ms, VAD same. Turn detection model also adds around 20 ms. I have zero clue if and how fast this runs on a Mac.
- riquito 1y ago
  Very cool, thanks for sharing. A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?
  - justlikereddit 1y ago
    Modify it with an ultra light LLM agent that always listens that uses a wake word to agentically call the paid API?
    - Dr4kn 1y ago
      You could use open wake word. Which Home Assistant developed for its own Voice Assistant
      - supermatt 1y ago
        It was developed by David Scripka: https://github.com/dscripka/openWakeWord https://github.com/dscripka/openWakeWord
  - TeMPOraL 1y ago
    FWIW, wake words are a stopgap; if we want to have a Star Trek level voice interfaces, where the computer responds only when you actually meant to call it, as opposed to using the wake word as a normal word in the conversation, the computer needs to be constantly listening. A good analogy here is to think of the computer (assistant) as another person in the room, busy with their own stuff but paying attention to the conversations happening around them, in case someone suddenly requests their assistance. This, of course, could be handled by a more lightweight LLM running locally and listening for explicit mentions/addressing the computer/assistant, as opposed to some context-free wake words.
    - Dr4kn 1y ago
      Home Assistant is much nearer to this than other solutions. You have a wake word, but it can also speak to you based on automations. You come home and it could tell you that the milk is empty, but with a holiday coming up you probably should go shopping.
- jokethrowaway 1y ago
  Neat! I build something almost identical last week (closed source, not my IP) and I recommend: NeMo Parakeet (even faster than insanely_fast_whisper), F5-TTS (fast + very good quality voice cloning), Qwen3-4B for LLM (amazing quality).
- tmaly 1y ago
  What is the min VRAM needed on the GPU to run this? I did not see that on the github
  - koljab 1y ago
    With the current 24b LLM model it's 24 GB. I have no clue how far down you can go with the GPU is using smaller models, you can set the model in server.py. Quite sure 16 GB will work but at some point it will probably fail.
smusamashah 1y ago
Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what he was saying. Length of pauses wasn't a problem. I don't how complex that problem is though. Probably another AI needs to analyse the input so far a decide if it's a pause or not.
- qwertox 1y ago
  Maybe we should settle on some special sound or word which officially signals that we're making a pause for whatever reason, but that we intend to continue with dictating in a couple of seconds. Like "Hmm, wait".
  - twodave 1y ago
    Alternatively we could pretend it’s a radio and follow those conventions.
  - ivape 1y ago
    Two input streams sounds like a good hacky solution. One input stream captures everything, the second is on the look out for your filler words like "um, aahh, waaiit, no nevermind, scratch that". The second stream can act as the veto-command and cut off the LLM. A third input stream can simply be on the lookout for long pauses. All this gets very resource intensive quickly. I been meaning to make this but since I haven't, I'm going to punish myself and just give the idea away. Hopefully I'll learn my lesson.
  - flippy_flops 1y ago
    Need some vocal version of “heredoc”
    - accrual 1y ago
      "Hello AI, over", "Hello human, over". :) Oh, wait: "How do I iterate over a list-", "Iteration is a process where..." :p
IshKebab 1y ago
Impressive! I guess the speech synthesis quality is the best available open source at the moment? The endgame of this is surely a continuously running wave to wave model with no text tokens at all? Or at least none in the main path.
- koljab 1y ago
  This is coqui xttsv2 because it can be tuned to deliver the first token in under 100 ms. Gives the best balance between quality and speed currently imho. If it's only about quality I'd say there are better models out there.
  - deleted 1y ago
    [deleted]
oldgregg 1y ago
Nice work, I like the lightweight web front end and your implementation of VAD.
breaker-kind 1y ago
why is your AI chatbot talking in a bizarre attempt at AAVE?
- PhunkyPhil 1y ago
  This is the system prompt https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/system_prompt.txt https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/s... My favorite line: "You ARE this charming, witty, wise girlfriend. Don't explain how you're talking or thinking; just be that person."
  - diggernet 1y ago
    I was hoping she'd let him have it for the way he kept interrupting her. But unfortunately it looks like he was just interrupting the TTS, so the LLM probably had no indication of the interuptions.
  - kevinsync 1y ago
    I still crack up at the idea of 'personality prompting', mostly because the most engaging and delightful IRL persons who knock us off our guard in a non-threatening way are super natural and possess that "It Factor" that's impossible to articulate lol -- probably because it's multimodal with humans and voice/cadence/vocab/timing/delivery isn't 100% of the attraction. That said, it's not like we have any better alternatives at the moment, but just something I think about when I try to digest a meaty personality prompt.
    - koljab 1y ago
      This character prompt has undergone so many iterations with LLMs it's not funny anymore. "Make her act more bold." - "She again talked about her character description, prevent that!"
    - varispeed 1y ago
      Aren't humans doing it as well? It's called affirmations. Many people do this as their morning "boot" time.
joshstrange 1y ago
This is very, very cool! The interrupting was a "wow" moment for me (I know it's not "new new" but to see it so well done in open source was awesome). Question about the Interrupt feature, how does it handle "Mmk", "Yes", "Of course", "cough", etc? Aside from the sycophancy from OpenAI's voice chat (no, not every question I ask is a "great question!") I dislike that a noise sometimes stops the AI from responding and there isn't a great way to get back on track, to pick up where you left off. It's a hard problem, how do you stop replying quickly AND make sure you are stopping for a good reason?
- koljab 1y ago
  That's a great question! My first implementation was interruption on voice activity after echo cancellation. It still had way too many false positives. I changed it to incoming realtime transcription as a trigger. That adds a bit of latency but that gets compensated by way better accuracy. Edit: just realized the irony but it's really a good question lol
  - joshstrange 1y ago
    That answer is even more than I could have hoped for. I worried doing that might be too slow. I wonder if it could be improved (without breaking something else) to "know" when to continue based on what it heard (active listening), maybe after a small pause. I'd put up with a chance of it continuing when I don't want it to as long as "Stop" would always work as a final fallback. Also, it took me longer than I care to admit to get your irony reference. Well done. Edit: Just to expand on that in case it was not clear, this would be the ideal case I think: LLM: You're going to want to start by installing XYZ, then you Human: Ahh, right LLM: Slight pause, makes sure that there is nothing more and checks if the reply is a follow up question/response or just active listening LLM: ...Then you will want to...
  - snet0 1y ago
    > That's a great question! Never forget what AI stole from us. This used to be a compliment, a genuine appreciation of a good question well-asked. Now it's tainted with the slimy, servile, sycophantic stink of AI chat models.
jedberg 1y ago
I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable and a 500ms delay as exceptional. - Every voice assistant up to that point (and probably still today) has a minimum delay of about 300ms, because they all use silence detection to decide when to start responding, and you need about 300ms of silence to reliably differentiate that from a speaker's normal pause - Alexa actually has a setting to increase this wait time for slower speakers. You'll notice in this demo video that the AI never interrupts him, which is what makes it feel like a not quite human interaction (plus the stilted intonations of the voice). Humans appear to process speech in a much more steaming why, constantly updating their parsing of the sentence until they have a high enough confidence level to respond, but using context clues and prior knowledge. For a voice assistant to get the "human" levels, it will have to work more like this, where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning.
- koljab 1y ago
  Thanks a lot, great insights. Exactly the kind of feedback that I need to improve things further.
  - jedberg 1y ago
    Love what you're doing, glad I could help!
- joshstrange 1y ago
  > where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning. I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multiple generations internally and starts generating after every word, or at least starts asking sub-LLM models "Do I have enough to reply?" and once it does it generates a reply and interrupts. I'm not sure how most apps handle the user interrupting, in regards to the conversation context. Do they stop generation but use what they have generated already in the context? Do they cut off where the LLM got interrupted? Something like "LLM: ..and then the horse walked... -USER INTERRUPTED-. User: ....". It's not a purely-voice-LLM issue but it comes up way more for that since rarely are you stopping generation (in the demo, that's been done for a while when he interrupts), just the TTS.
fintechie 1y ago
Quite good, it would sound much better with SOTA voices though: https://github.com/nari-labs/dia https://github.com/nari-labs/dia
- koljab 1y ago
  Dia is too slow, I need a time to first audio chunk of ~100 milliseconds. Also generations fail too often (artifacts etc)
- thamer 1y ago
  Does Dia support configuring voices now? I looked at it when it was first released, and you could only specify [S1] [S2] for the speakers, but not how they would sound. There was also a very prominent issue where the voices would be sped up if the text was over a few sentences long; the longer the text, the faster it was spoken. One suggestion was to split the conversation into chunks with only one or two "turns" per speaker, but then you'd hear two voices then two more, then two more… with no way to configure any of it. Dia looked cool on the surface when it was released, but it was only a demo for now and not at all usable for any real use case, even for a personal app. I'm sure they'll get to these issues eventually, but most comments I've seen so far recommending it are from people who have not actually used it or they would know of these major limitations.
dcreater 1y ago
Does the docker container work on Mac?
- koljab 1y ago
  I doubt TTS will be fast enough for realtime without a Nvidia GPU
cannonpr 1y ago
Kind of surprised nobody has brought up https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice#demo https://www.sesame.com/research/crossing_the_uncanny_valley_... It interacts nearly like a human, can and does interrupt me once it has enough context in many situations, and has exceedingly low levels of latency, using for the first time was a fairly shocking experience for me.
- varispeed 1y ago
  Didn't expect it to be that good! Nice.
  - ic4l 1y ago
    Yeah, thats one of the best ones I have seen, and it popped up a while ago.
briga 1y ago
I'm starting to feel like LLMs need to be tuned for shorter responses. For every short sentence you give them they outputs paragraphs of text. Sometimes it's even good text, but not every input sentence needs a mini-essay in response. Very cool project though. Maybe you can fine tune the prompt to change how chatty your AI is.
lacoolj 1y ago
Call me when the AI can interrupt YOU :)
- esafak 1y ago
  In Soviet Russia...
- alluro2 1y ago
  Apparently, based on other comments that mentioned it, this one can and will if it's confident enough it has sufficient context/information: https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice#demo https://www.sesame.com/research/crossing_the_uncanny_valley_...
- foobahhhhh 1y ago
  The next Turing test. Can you have a heated debate and not tell it was AI. Once it can emulate a 13 year old talking to their parent I will then worry about AGI
tintor 1y ago
After interrupt, unspoken words from LLM are still in the chat window. Is LLM even aware that it was interrupted and where exactly?
- koljab 1y ago
  It's not aware. The information that it had been interrupted would be something we can easily add to the next user chat request. Where exactly is harder, because at least for Coqui XTTSv2 we don't have TTS wordstamps (we do have them for Kokoro though). So adding the information where it had been interrupted would be easily possible when using Kokoro as TTS system. With Coqui we'd need to add another transcription on the tts output including word timestamps. That would cost more compute than a normal transcription and word timestamps aren't perfectly accurate. Yet directly after an interruption there's not that much concurrent need for compute (like in the end of turn detection phase where a lot of stuff is happening). So I guess with a bit of programming work this could be integrated.
kabirgoel 1y ago
This is great. Poking into the source, I find it interesting that the author implemented a custom turn detection strategy, instead of using Silero VAD (which is standard in the voice agents space). I’m very curious why they did it this way and what benefits they observed. For folks that are curious about the state of the voice agents space, Daily (the WebRTC company) has a great guide [1], as well as an open-source framework that allows you to build AI voice chat similar to OP's with lots of utilities [2]. Disclaimer: I work at Cartesia, which services a lot of these voice agents use cases, and Daily is a friend. [1]: https://voiceaiandvoiceagents.com https://voiceaiandvoiceagents.com [2]: https://docs.pipecat.ai/getting-started/overview https://docs.pipecat.ai/getting-started/overview
- koljab 1y ago
  It's in fact using Silero via RealtimeSTT. RealtimeSTT tells when silence starts. Then a binary sentence classification model is used on the realtime transcription text which infers blazingly fast (10ms) and returns a probability between 0 and 1 indicating if the current spoken sentence is considered "complete". The turn detection component takes this information to calculate the silence waiting time until "turn is over".
  - thekaranchawla 1y ago
    This is the exact strategy I'm using for the real-time voice agent I'm building. Livekit also published a custom turn detection model that works really well based on the video they released, which was cool to see. Code: https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-turn-detector https://github.com/livekit/agents/tree/main/livekit-plugins/... Blog: https://blog.livekit.io/using-a-transformer-to-improve-end-of-turn-detection/ https://blog.livekit.io/using-a-transformer-to-improve-end-o...
bufferoverflow 1y ago
It's fast, but it doesn't sound good. Many voice chat AIs are way ahead and sound natural.
znpy 1y ago
This kind of thing immediately made me think about the 512gb mac studio. If this works as good on that hardware as it does on the recommended nvidia cards, then the $15k is not much the price of the hardware but rather the price of having a full conversational at home, private.
- foobahhhhh 1y ago
  Exciting... because that'll be $1500 at some point. Then $150. Then $15 I.e. on a cheap android old gen phone.
- karolist 1y ago
  You don't need a 512GB mac studio for this, TTS latency would be worse than 16GB 5080.
nitrogen99 1y ago
Will this work on a Raspberry Pi?
- regularfry 1y ago
  Not reliably. It can only drive Whisper quickly enough to appear real-time because of the GPU, and without that you're limited to the tiny/small/base models to get latency into single-digit seconds. Edit to add: this might not be true since whisper-large-v3-turbo got released. I've not tried that on a pi 5 yet.
2809 1y ago
Looks neat. Be good to get AMD/Intel support of course.
orliesaurus 1y ago
added a star because the revolution will come from these repos - thank you Author for working on this in the open!
hegemon8 1y ago
This is an impressive project—great work! I’m curious anyone has came across similar work, but for multi-lingual voice agents, especially those that handle non-English languages and English + X well. Does a Translation step right after the ASR step make sense at all? Any pointers—papers, repos —would be appreciated!
krick 1y ago
Cool for a weekend project, but honestly ChatGPT is still kinda shit at dialogues. I wonder if that's the issue with technology or OpenAI's fine-tuning (and suspect the latter), but it cannot talk like normal people do: shut up if it has nothing to add of value, ask reasonable follow-up questions if user doesn't understand something or there's ambiguity in the question. Also, on topic of follow-up questions: I don't remember which update introduced that attempt to increase engagement by finishing every post with stupid irrelevant follow-up question, but it's really annoying. It also works on me, despite hating ChatGPT it's kinda an instinct to treat humanly something that speaks vaguely like a human.
- nolroz 1y ago
  Do you hate any of the other models less?
- sebra 1y ago
  I added this to personal instructions to make it less annoying: • No compliments, flattery, or emotional rapport. • Focus on clear reasoning and evidence. • Be critical of users assumptions when needed. • Ask follow-up questions only when essential for accuracy. However, I'm kinda concerned with crippling it by adding custom prompts. It's kinda hard to know how to use AI efficiently. But the glazing and random follow-up questions feel more like a result of some A/B testing UX-research rather than improving the results of the model.
  - genewitch 1y ago
    I often ask copilot about phrases I hear that I don't know or understand, like "what is a key party" - where I just want it to define it, and it will output three paragraphs that end with some suggestion that I am interested in it. It is something that local models I have tried do not do, unless you are being conversational with it. I imagine openai gets a bit more pennies if they add the open ended questions to the end of every reply, and that's why it's done. I get annoyed if people patronize me, so too I get annoyed at a computer.
stevage 1y ago
Hell yeah, exactly.
lhl 1y ago
Maybe of interest, I built and open-sourced a similar (web-based) end-to-end voice project last year for an AMD Hackathon: https://github.com/lhl/voicechat2 https://github.com/lhl/voicechat2 As a submission for an AMD Hackathon, one big thing is that I tested all the components to work with RDNA3 cards. It's built to allow for swappable components for the SRT, LLM, TTS (the tricky stuff was making websockets work and doing some sentence-based interleaving to lower latency). Here's a full write up on the project: https://www.hackster.io/lhl/voicechat2-local-ai-voice-chat-4c48f2 https://www.hackster.io/lhl/voicechat2-local-ai-voice-chat-4... (I've don't really have time to maintain that project, but it can be a good starting point for anyone that's looking to hack their own thing together.)
SillyUsername 1y ago
Have you considered using Dia for the TTS? I believe this is currently "best in class" https://github.com/nari-labs/dia https://github.com/nari-labs/dia
shengfengli 1y ago
[dead]
dhvjcshdvn 1y ago
[flagged]
pinoy420 1y ago
[dead]
grav 1y ago
In the demo, is there any specific reason that the voice doesn't go "up" in pitch when asking questions? Even the (many) rethorical questions would in my view improve by having a bit of a pitch change before the question mark.
- dade_ 1y ago
  There’s no SSML. The model that came up with the text knows what it’s saying in theory and therefore would know that it’s a question, if the mood should be sombre or excited and then can pass this information as SSML tags to the text to speech synthesizer. The problem I’ve been seeing is that pretty much all of these models are just outputting text and the text is being shoved into the TTS. It’s on my list to look into projects that have embedded these tags so that on the one hand you have like open web UI that’s showing a user text, but there’s actually an embedded set of tags that are handled by the TTS so that it sounds more natural. This project looks hackable for that purpose.
regularfry 1y ago
We really, really need something to take Whisper's crown for streaming. Faster-whisper is great, but Whisper itself was never built for real-time use. For this demo to be real-time, it relies on having a beefy enough GPU that it can push 30 seconds of audio through one of the more capable (therefore bigger) models in a couple of hundred milliseconds. It's basically throwing hardware at the problem to paper over the fact that Whisper is just the wrong architecture. Don't get me wrong, it's great where it's great, but that's just not streaming.
blueblisters 1y ago
Does this work for simultaneous multiple clients at the same endpoint?
smallflykkoo 1y ago
[dead]
sabellito 1y ago
Every time I see these things, they look cool as hell, I get excited, then I try to get them working on my gaming PC (that has the GPU), I spend 1-2h fighting with python and give up. Today's issue is that my python version is 3.12 instead of <3.12,>=3.9. Installing python 3.11 from the official website does nothing, I give up. It's a shame that the amazing work done by people like the OP gets underused because of this mess outside of their control. "Just use docker". Have you tried using docker on windows? There's a reason I never do dev work on windows. I spent most of my career in the JVM and Node, and despite the issues, never had to deal with this level of lack of compatibility.
- jhoho 1y ago
  Let me introduce you to the beautiful world of virtual environments. They save you the headache of getting a full installation to run, especially when using Windows. I prefer miniconda, but venv also does the job.
  - regularfry 1y ago
    Virtual environments with venv don't answer the python version problem unless you throw another tool into the mix.
    - underdeserver 1y ago
      Hmm? My venvs do include the Python version (via symlink to /bin). Don't yours?
      - diggan 1y ago
        If you use something like uv (expanded here: https://news.ycombinator.com/item?id=43904078 https://news.ycombinator.com/item?id=43904078), I think it does. But if you just do `python -m venv .venv`, you get the specific version you used to create the virtual environment with. Some OSes seem to distribute binaries like `python3.8`, `python3.9` and so on so you could do `python3.8 -m venv .venv` to look one env to a specific version, but a bit of a hassle.
ConfusedDog 1y ago
That's a big improvement over Siri tbh (interruption and latency), but Siri's answer generally kind of shorter than this. My general experience with Siri hasn't been great lately. For complex question, it just redirect to ChatGPT with an extra step for me to confirm. Often stops listening when I'm not even finished with my sentence, and gives "I don't know anything about that" way too often.
XCSme 1y ago
The demo reminded me of this amazing post: https://sambleckley.com/writing/church-of-interruption.html https://sambleckley.com/writing/church-of-interruption.html
singularity2001 1y ago
Voice in Text out is the way to go except for very simple Use Cases / questions.
rane 1y ago
What are currently the best options for low latency TTS and STT as external services? If you want to host an app with these capabilities on a VPS, anything that requires a GPU doesn't seem feasible.
jak0 1y ago
Can this be tweaked somehow to try to reproduce the experience of Aqua Voice? https://withaqua.com/ https://withaqua.com/
glossardi 1y ago
I need this for my next app! Great job, thank you for sharing!

524 points by koljab 1y ago

I had been working on something like it when I came across this. Excellent work. Love the demo.

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8 https://www.youtube.com/watch?v=HM_IQuuuPX8 The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and RealtimeTTS (supporting engines like Coqui XTTSv2/Kokoro) to achieve around 500ms response latency, even when running larger local models like a 24B Mistral fine-tune via Ollama. Key aspects: Designed for local LLMs (Ollama primarily, OpenAI connector included). Interruptible conversation. Smart turn detection to avoid cutting the user off mid-thought. Dockerized setup available for easier dependency management. It requires a decent CUDA-enabled GPU for good performance due to the STT/TTS models. Would love to hear your feedback on the approach, performance, potential optimizations, or any features you think are essential for a good local voice AI experience. The code is here: https://github.com/KoljaB/RealtimeVoiceChat https://github.com/KoljaB/RealtimeVoiceChat

pzo 1y ago
This looks great will definitely have a look. I'm just wondering if you tested fastRTC from hugging face? I haven't done that curious about speed between this vs fastrtc vs pipecat.
- koljab 1y ago
  Yes, I tested it. I'm not that sure what they created there. It adds some noticable latency compared towards using raw websockets. Imho it's not supposed to, but it did it nevertheless in my tests.
ivape 1y ago
Would you say you are using the best-in-class speech to text libs at the moment? I feel like this space is moving fast because the last time I was headed down this track, I was sure whisper-cpp was the best.
- koljab 1y ago
  I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to evaluate these latest models, they look strong.
  - ivape 1y ago
    Yeah, I figured you would know. Thanks for that, bookmarking that asr leaderboard.
  - kristopolous 1y ago
    https://yummy-fir-7a4.notion.site/dia https://yummy-fir-7a4.notion.site/dia is the new hotness.
    - koljab 1y ago
      Tried that one. Quality is great but sometimes generations fail and it's rather slow. Also needs ~13 GB of VRAM, it's not my first choice for voice agents tbh.
      - kristopolous 1y ago
        alright, dumb question. (1) I assume these things can do multiple languages (2) Given (1), can you strip all the languages you aren't using and speed things up?
        koljab 1y ago
        Actually good question. I'd say probably not. You can't easily "unlearn" things from the model weights (and even if this alone doesn't help). You could retrain/finetune the model heavily on a single language but again that alone does not speed up inference. To gain speed you'd have to bring the parameter count down and train the model from scratch with a single language only. That might work but it's also quite probable that it introduces other issues in the synthesis. In a perfect world the model would only use all that "free parameters" not used now for other languages for a better synthesis of that single trained language. Might be true to a certain degree, but it's not exactly how ai parameter scaling works.
dotancohen 1y ago
This looks great. What hardware do you use, or have you tested it on?
- koljab 1y ago
  I only tested it on my 4090 so far
  - echelon 1y ago
    Are you using all local models, or does it also use cloud inference? Proprietary models? Which models are running in which places? Cool utility!
    - koljab 1y ago
      All local models: - VAD: Webrtcvad (first fast check) followed by SileroVAD (high compute verification) - Transcription: base.en whisper (CTranslate2) - Turn Detection: KoljaB/SentenceFinishedClassification (selftrained BERT-model) - LLM: hf.co/bartowski/huihui-ai_Mistral-Small-24B-Instruct-2501-abliterated-GGUF:Q4_K_M (easily switchable) - TTS: Coqui XTTSv2, switchable to Kokoro or Orpheus (this one is slower)
      - echelon 1y ago
        That's excellent. Really amazing bringing all of these together like this. Hopefully we get an open weights version of Sesame [1] soon. Keep watching for it, because that'd make a killer addition to your app. [1] https://www.sesame.com/ https://www.sesame.com/
        koljab 1y ago
        That would be absolutely awesome. But I doubt it, since they released a shitty version of that amazing thing they put online. I feel they aren't planning to give us their top model soon.
zaggynl 1y ago
Neat! I'm already using openwebui/ollama with a 7900 xtx but the STT and TTS parts don't seem to work with it yet: 2025-05-05 20:53:15,808] [WARNING] [real_accelerator.py:194:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it. Error loading model for checkpoint ./models/Lasinya: This op had not been implemented on CPU backend.
- dankwizard 1y ago
  I've given up trying to locally use LLMs on AMD
  - lhl 1y ago
    Basically anything llama.cpp (Vulkan backend) should work out of the box w/o much fuss (LM Studio, Ollama, etc). The HIP backend can have a big prefill speed boost on some architectures (high-end RDNA3 for example). For everything else, I keep notes here: https://llm-tracker.info/howto/AMD-GPUs https://llm-tracker.info/howto/AMD-GPUs
dummydummy1234 1y ago
Have you looked at pipecat, seems to be similar trying to do standardized backend/webrtc turn detection pipelines.
- koljab 1y ago
  Did not look into that one. Looks quite good, I will try that soon.
peterldowns 1y ago
Can you explain more about the "Coqui XTTS Lasinya" models that the code is using? What are these, and how were they trained/finetuned? I'm assuming you're the one who uploaded them to huggingface, but there's no model card or README https://huggingface.co/KoljaB/XTTS_Models https://huggingface.co/KoljaB/XTTS_Models In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/audio_module.py#L106 https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...
- Buckaroo9 1y ago
  https://huggingface.co/coqui/XTTS-v2 https://huggingface.co/coqui/XTTS-v2
- optimog 1y ago
  Seems like they are out of business. Their homepage mentions "Coqui is schutting down"* That is probably the reason you can't find that much. *https://coqui.ai/ https://coqui.ai/
- wkat4242 1y ago
  Yeah I really dislike the whisperiness of this voice "Lasinya". It sounds too much like an erotic phone service. I wonder if there's any alternative voice? I don't see Lasinya even mentioned in the public coqui models: https://github.com/coqui-ai/STT-models/releases https://github.com/coqui-ai/STT-models/releases . But I don't see a list of other model names I could use either. I tried to select kokoro in the python module but it says in the logs that only coqui is available. I do have to say the coqui models sound really good, it's just the type of voice that puts me off. The default prompt is also way too "girlfriendy" but that was easily fixed. But for the voice, I simply don't know what the other options are for this engine. PS: Forgive my criticism of the default voice but I'm really impressed with the responsiveness of this. It really responds so fast. Thanks for making this!
  - koljab 1y ago
    Yeah I know the voice polarizes, I trained it for myself, so it's not an official release. You can change the voice here: https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/audio_module.py#L108 https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a... Create a subfolder in the app container: ./models/some_folder_name Copy the files from your desired voice into that folder: config.json, model.pth, vocab.json and speakers_xtts.pth (you can copy the speakers_xtts.pth from Lasinya, it's the same for every voice) Then change the specific_model="Lasinya" line in audio_module.py into specific_model="some_folder_name". If you change TTS_START_ENGINE to "kokoro" in server.py it's supposed to work, what does happen then? Can you post the log message?
karimf 1y ago
Do you have any information on how long each step take? Like how many ms on each step of the pipeline? I'm curious how fast it will run if we can get this running on a Mac. Any ballpark guess?
- koljab 1y ago
  LLM and TTS latency get's determined and logged at the start. It's around 220ms for the LLM returning the first synthesizable sentence fragment (depending on the length of the fragment, which is usually something between 3 and 10 words). Then around 80ms of TTS until the first audio chunk is delivered. STT with base.en you can neglect, it's under 5 ms, VAD same. Turn detection model also adds around 20 ms. I have zero clue if and how fast this runs on a Mac.
riquito 1y ago
Very cool, thanks for sharing. A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?
- justlikereddit 1y ago
  Modify it with an ultra light LLM agent that always listens that uses a wake word to agentically call the paid API?
  - Dr4kn 1y ago
    You could use open wake word. Which Home Assistant developed for its own Voice Assistant
    - supermatt 1y ago
      It was developed by David Scripka: https://github.com/dscripka/openWakeWord https://github.com/dscripka/openWakeWord
- TeMPOraL 1y ago
  FWIW, wake words are a stopgap; if we want to have a Star Trek level voice interfaces, where the computer responds only when you actually meant to call it, as opposed to using the wake word as a normal word in the conversation, the computer needs to be constantly listening. A good analogy here is to think of the computer (assistant) as another person in the room, busy with their own stuff but paying attention to the conversations happening around them, in case someone suddenly requests their assistance. This, of course, could be handled by a more lightweight LLM running locally and listening for explicit mentions/addressing the computer/assistant, as opposed to some context-free wake words.
  - Dr4kn 1y ago
    Home Assistant is much nearer to this than other solutions. You have a wake word, but it can also speak to you based on automations. You come home and it could tell you that the milk is empty, but with a holiday coming up you probably should go shopping.
jokethrowaway 1y ago
Neat! I build something almost identical last week (closed source, not my IP) and I recommend: NeMo Parakeet (even faster than insanely_fast_whisper), F5-TTS (fast + very good quality voice cloning), Qwen3-4B for LLM (amazing quality).
tmaly 1y ago
What is the min VRAM needed on the GPU to run this? I did not see that on the github
- koljab 1y ago
  With the current 24b LLM model it's 24 GB. I have no clue how far down you can go with the GPU is using smaller models, you can set the model in server.py. Quite sure 16 GB will work but at some point it will probably fail.

smusamashah 1y ago

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what he was saying. Length of pauses wasn't a problem. I don't how complex that problem is though. Probably another AI needs to analyse the input so far a decide if it's a pause or not.

qwertox 1y ago
Maybe we should settle on some special sound or word which officially signals that we're making a pause for whatever reason, but that we intend to continue with dictating in a couple of seconds. Like "Hmm, wait".
- twodave 1y ago
  Alternatively we could pretend it’s a radio and follow those conventions.
- ivape 1y ago
  Two input streams sounds like a good hacky solution. One input stream captures everything, the second is on the look out for your filler words like "um, aahh, waaiit, no nevermind, scratch that". The second stream can act as the veto-command and cut off the LLM. A third input stream can simply be on the lookout for long pauses. All this gets very resource intensive quickly. I been meaning to make this but since I haven't, I'm going to punish myself and just give the idea away. Hopefully I'll learn my lesson.
- flippy_flops 1y ago
  Need some vocal version of “heredoc”
  - accrual 1y ago
    "Hello AI, over", "Hello human, over". :) Oh, wait: "How do I iterate over a list-", "Iteration is a process where..." :p

IshKebab 1y ago

Impressive! I guess the speech synthesis quality is the best available open source at the moment? The endgame of this is surely a continuously running wave to wave model with no text tokens at all? Or at least none in the main path.

koljab 1y ago
This is coqui xttsv2 because it can be tuned to deliver the first token in under 100 ms. Gives the best balance between quality and speed currently imho. If it's only about quality I'd say there are better models out there.
- deleted 1y ago
  [deleted]

oldgregg 1y ago

Nice work, I like the lightweight web front end and your implementation of VAD.

breaker-kind 1y ago

why is your AI chatbot talking in a bizarre attempt at AAVE?

PhunkyPhil 1y ago
This is the system prompt https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/system_prompt.txt https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/s... My favorite line: "You ARE this charming, witty, wise girlfriend. Don't explain how you're talking or thinking; just be that person."
- diggernet 1y ago
  I was hoping she'd let him have it for the way he kept interrupting her. But unfortunately it looks like he was just interrupting the TTS, so the LLM probably had no indication of the interuptions.
- kevinsync 1y ago
  I still crack up at the idea of 'personality prompting', mostly because the most engaging and delightful IRL persons who knock us off our guard in a non-threatening way are super natural and possess that "It Factor" that's impossible to articulate lol -- probably because it's multimodal with humans and voice/cadence/vocab/timing/delivery isn't 100% of the attraction. That said, it's not like we have any better alternatives at the moment, but just something I think about when I try to digest a meaty personality prompt.
  - koljab 1y ago
    This character prompt has undergone so many iterations with LLMs it's not funny anymore. "Make her act more bold." - "She again talked about her character description, prevent that!"
  - varispeed 1y ago
    Aren't humans doing it as well? It's called affirmations. Many people do this as their morning "boot" time.

joshstrange 1y ago

This is very, very cool! The interrupting was a "wow" moment for me (I know it's not "new new" but to see it so well done in open source was awesome). Question about the Interrupt feature, how does it handle "Mmk", "Yes", "Of course", "cough", etc? Aside from the sycophancy from OpenAI's voice chat (no, not every question I ask is a "great question!") I dislike that a noise sometimes stops the AI from responding and there isn't a great way to get back on track, to pick up where you left off. It's a hard problem, how do you stop replying quickly AND make sure you are stopping for a good reason?

koljab 1y ago
That's a great question! My first implementation was interruption on voice activity after echo cancellation. It still had way too many false positives. I changed it to incoming realtime transcription as a trigger. That adds a bit of latency but that gets compensated by way better accuracy. Edit: just realized the irony but it's really a good question lol
- joshstrange 1y ago
  That answer is even more than I could have hoped for. I worried doing that might be too slow. I wonder if it could be improved (without breaking something else) to "know" when to continue based on what it heard (active listening), maybe after a small pause. I'd put up with a chance of it continuing when I don't want it to as long as "Stop" would always work as a final fallback. Also, it took me longer than I care to admit to get your irony reference. Well done. Edit: Just to expand on that in case it was not clear, this would be the ideal case I think: LLM: You're going to want to start by installing XYZ, then you Human: Ahh, right LLM: Slight pause, makes sure that there is nothing more and checks if the reply is a follow up question/response or just active listening LLM: ...Then you will want to...
- snet0 1y ago
  > That's a great question! Never forget what AI stole from us. This used to be a compliment, a genuine appreciation of a good question well-asked. Now it's tainted with the slimy, servile, sycophantic stink of AI chat models.

jedberg 1y ago

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable and a 500ms delay as exceptional. - Every voice assistant up to that point (and probably still today) has a minimum delay of about 300ms, because they all use silence detection to decide when to start responding, and you need about 300ms of silence to reliably differentiate that from a speaker's normal pause - Alexa actually has a setting to increase this wait time for slower speakers. You'll notice in this demo video that the AI never interrupts him, which is what makes it feel like a not quite human interaction (plus the stilted intonations of the voice). Humans appear to process speech in a much more steaming why, constantly updating their parsing of the sentence until they have a high enough confidence level to respond, but using context clues and prior knowledge. For a voice assistant to get the "human" levels, it will have to work more like this, where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning.

koljab 1y ago
Thanks a lot, great insights. Exactly the kind of feedback that I need to improve things further.
- jedberg 1y ago
  Love what you're doing, glad I could help!
joshstrange 1y ago
> where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning. I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multiple generations internally and starts generating after every word, or at least starts asking sub-LLM models "Do I have enough to reply?" and once it does it generates a reply and interrupts. I'm not sure how most apps handle the user interrupting, in regards to the conversation context. Do they stop generation but use what they have generated already in the context? Do they cut off where the LLM got interrupted? Something like "LLM: ..and then the horse walked... -USER INTERRUPTED-. User: ....". It's not a purely-voice-LLM issue but it comes up way more for that since rarely are you stopping generation (in the demo, that's been done for a while when he interrupts), just the TTS.

fintechie 1y ago

Quite good, it would sound much better with SOTA voices though: https://github.com/nari-labs/dia https://github.com/nari-labs/dia

koljab 1y ago
Dia is too slow, I need a time to first audio chunk of ~100 milliseconds. Also generations fail too often (artifacts etc)
thamer 1y ago
Does Dia support configuring voices now? I looked at it when it was first released, and you could only specify [S1] [S2] for the speakers, but not how they would sound. There was also a very prominent issue where the voices would be sped up if the text was over a few sentences long; the longer the text, the faster it was spoken. One suggestion was to split the conversation into chunks with only one or two "turns" per speaker, but then you'd hear two voices then two more, then two more… with no way to configure any of it. Dia looked cool on the surface when it was released, but it was only a demo for now and not at all usable for any real use case, even for a personal app. I'm sure they'll get to these issues eventually, but most comments I've seen so far recommending it are from people who have not actually used it or they would know of these major limitations.

dcreater 1y ago

Does the docker container work on Mac?

koljab 1y ago
I doubt TTS will be fast enough for realtime without a Nvidia GPU

cannonpr 1y ago

Kind of surprised nobody has brought up https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice#demo https://www.sesame.com/research/crossing_the_uncanny_valley_... It interacts nearly like a human, can and does interrupt me once it has enough context in many situations, and has exceedingly low levels of latency, using for the first time was a fairly shocking experience for me.

varispeed 1y ago
Didn't expect it to be that good! Nice.
- ic4l 1y ago
  Yeah, thats one of the best ones I have seen, and it popped up a while ago.

briga 1y ago

I'm starting to feel like LLMs need to be tuned for shorter responses. For every short sentence you give them they outputs paragraphs of text. Sometimes it's even good text, but not every input sentence needs a mini-essay in response. Very cool project though. Maybe you can fine tune the prompt to change how chatty your AI is.

lacoolj 1y ago

Call me when the AI can interrupt YOU :)

esafak 1y ago
In Soviet Russia...
alluro2 1y ago
Apparently, based on other comments that mentioned it, this one can and will if it's confident enough it has sufficient context/information: https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice#demo https://www.sesame.com/research/crossing_the_uncanny_valley_...
foobahhhhh 1y ago
The next Turing test. Can you have a heated debate and not tell it was AI. Once it can emulate a 13 year old talking to their parent I will then worry about AGI

tintor 1y ago

After interrupt, unspoken words from LLM are still in the chat window. Is LLM even aware that it was interrupted and where exactly?

koljab 1y ago
It's not aware. The information that it had been interrupted would be something we can easily add to the next user chat request. Where exactly is harder, because at least for Coqui XTTSv2 we don't have TTS wordstamps (we do have them for Kokoro though). So adding the information where it had been interrupted would be easily possible when using Kokoro as TTS system. With Coqui we'd need to add another transcription on the tts output including word timestamps. That would cost more compute than a normal transcription and word timestamps aren't perfectly accurate. Yet directly after an interruption there's not that much concurrent need for compute (like in the end of turn detection phase where a lot of stuff is happening). So I guess with a bit of programming work this could be integrated.

kabirgoel 1y ago

This is great. Poking into the source, I find it interesting that the author implemented a custom turn detection strategy, instead of using Silero VAD (which is standard in the voice agents space). I’m very curious why they did it this way and what benefits they observed. For folks that are curious about the state of the voice agents space, Daily (the WebRTC company) has a great guide [1], as well as an open-source framework that allows you to build AI voice chat similar to OP's with lots of utilities [2]. Disclaimer: I work at Cartesia, which services a lot of these voice agents use cases, and Daily is a friend. [1]: https://voiceaiandvoiceagents.com https://voiceaiandvoiceagents.com [2]: https://docs.pipecat.ai/getting-started/overview https://docs.pipecat.ai/getting-started/overview

koljab 1y ago
It's in fact using Silero via RealtimeSTT. RealtimeSTT tells when silence starts. Then a binary sentence classification model is used on the realtime transcription text which infers blazingly fast (10ms) and returns a probability between 0 and 1 indicating if the current spoken sentence is considered "complete". The turn detection component takes this information to calculate the silence waiting time until "turn is over".
- thekaranchawla 1y ago
  This is the exact strategy I'm using for the real-time voice agent I'm building. Livekit also published a custom turn detection model that works really well based on the video they released, which was cool to see. Code: https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-turn-detector https://github.com/livekit/agents/tree/main/livekit-plugins/... Blog: https://blog.livekit.io/using-a-transformer-to-improve-end-of-turn-detection/ https://blog.livekit.io/using-a-transformer-to-improve-end-o...

bufferoverflow 1y ago

It's fast, but it doesn't sound good. Many voice chat AIs are way ahead and sound natural.

znpy 1y ago

This kind of thing immediately made me think about the 512gb mac studio. If this works as good on that hardware as it does on the recommended nvidia cards, then the $15k is not much the price of the hardware but rather the price of having a full conversational at home, private.

foobahhhhh 1y ago
Exciting... because that'll be $1500 at some point. Then $150. Then $15 I.e. on a cheap android old gen phone.
karolist 1y ago
You don't need a 512GB mac studio for this, TTS latency would be worse than 16GB 5080.

nitrogen99 1y ago

Will this work on a Raspberry Pi?

regularfry 1y ago
Not reliably. It can only drive Whisper quickly enough to appear real-time because of the GPU, and without that you're limited to the tiny/small/base models to get latency into single-digit seconds. Edit to add: this might not be true since whisper-large-v3-turbo got released. I've not tried that on a pi 5 yet.

2809 1y ago

Looks neat. Be good to get AMD/Intel support of course.

orliesaurus 1y ago

added a star because the revolution will come from these repos - thank you Author for working on this in the open!

hegemon8 1y ago

This is an impressive project—great work! I’m curious anyone has came across similar work, but for multi-lingual voice agents, especially those that handle non-English languages and English + X well. Does a Translation step right after the ASR step make sense at all? Any pointers—papers, repos —would be appreciated!

krick 1y ago

Cool for a weekend project, but honestly ChatGPT is still kinda shit at dialogues. I wonder if that's the issue with technology or OpenAI's fine-tuning (and suspect the latter), but it cannot talk like normal people do: shut up if it has nothing to add of value, ask reasonable follow-up questions if user doesn't understand something or there's ambiguity in the question. Also, on topic of follow-up questions: I don't remember which update introduced that attempt to increase engagement by finishing every post with stupid irrelevant follow-up question, but it's really annoying. It also works on me, despite hating ChatGPT it's kinda an instinct to treat humanly something that speaks vaguely like a human.

nolroz 1y ago
Do you hate any of the other models less?
sebra 1y ago
I added this to personal instructions to make it less annoying: • No compliments, flattery, or emotional rapport. • Focus on clear reasoning and evidence. • Be critical of users assumptions when needed. • Ask follow-up questions only when essential for accuracy. However, I'm kinda concerned with crippling it by adding custom prompts. It's kinda hard to know how to use AI efficiently. But the glazing and random follow-up questions feel more like a result of some A/B testing UX-research rather than improving the results of the model.
- genewitch 1y ago
  I often ask copilot about phrases I hear that I don't know or understand, like "what is a key party" - where I just want it to define it, and it will output three paragraphs that end with some suggestion that I am interested in it. It is something that local models I have tried do not do, unless you are being conversational with it. I imagine openai gets a bit more pennies if they add the open ended questions to the end of every reply, and that's why it's done. I get annoyed if people patronize me, so too I get annoyed at a computer.

stevage 1y ago

Hell yeah, exactly.

lhl 1y ago

Maybe of interest, I built and open-sourced a similar (web-based) end-to-end voice project last year for an AMD Hackathon: https://github.com/lhl/voicechat2 https://github.com/lhl/voicechat2 As a submission for an AMD Hackathon, one big thing is that I tested all the components to work with RDNA3 cards. It's built to allow for swappable components for the SRT, LLM, TTS (the tricky stuff was making websockets work and doing some sentence-based interleaving to lower latency). Here's a full write up on the project: https://www.hackster.io/lhl/voicechat2-local-ai-voice-chat-4c48f2 https://www.hackster.io/lhl/voicechat2-local-ai-voice-chat-4... (I've don't really have time to maintain that project, but it can be a good starting point for anyone that's looking to hack their own thing together.)

SillyUsername 1y ago

Have you considered using Dia for the TTS? I believe this is currently "best in class" https://github.com/nari-labs/dia https://github.com/nari-labs/dia

shengfengli 1y ago

[dead]

dhvjcshdvn 1y ago

[flagged]

pinoy420 1y ago

[dead]

grav 1y ago

In the demo, is there any specific reason that the voice doesn't go "up" in pitch when asking questions? Even the (many) rethorical questions would in my view improve by having a bit of a pitch change before the question mark.

dade_ 1y ago
There’s no SSML. The model that came up with the text knows what it’s saying in theory and therefore would know that it’s a question, if the mood should be sombre or excited and then can pass this information as SSML tags to the text to speech synthesizer. The problem I’ve been seeing is that pretty much all of these models are just outputting text and the text is being shoved into the TTS. It’s on my list to look into projects that have embedded these tags so that on the one hand you have like open web UI that’s showing a user text, but there’s actually an embedded set of tags that are handled by the TTS so that it sounds more natural. This project looks hackable for that purpose.

regularfry 1y ago

We really, really need something to take Whisper's crown for streaming. Faster-whisper is great, but Whisper itself was never built for real-time use. For this demo to be real-time, it relies on having a beefy enough GPU that it can push 30 seconds of audio through one of the more capable (therefore bigger) models in a couple of hundred milliseconds. It's basically throwing hardware at the problem to paper over the fact that Whisper is just the wrong architecture. Don't get me wrong, it's great where it's great, but that's just not streaming.

blueblisters 1y ago

Does this work for simultaneous multiple clients at the same endpoint?

smallflykkoo 1y ago

[dead]

sabellito 1y ago

Every time I see these things, they look cool as hell, I get excited, then I try to get them working on my gaming PC (that has the GPU), I spend 1-2h fighting with python and give up. Today's issue is that my python version is 3.12 instead of <3.12,>=3.9. Installing python 3.11 from the official website does nothing, I give up. It's a shame that the amazing work done by people like the OP gets underused because of this mess outside of their control. "Just use docker". Have you tried using docker on windows? There's a reason I never do dev work on windows. I spent most of my career in the JVM and Node, and despite the issues, never had to deal with this level of lack of compatibility.

jhoho 1y ago
Let me introduce you to the beautiful world of virtual environments. They save you the headache of getting a full installation to run, especially when using Windows. I prefer miniconda, but venv also does the job.
- regularfry 1y ago
  Virtual environments with venv don't answer the python version problem unless you throw another tool into the mix.
  - underdeserver 1y ago
    Hmm? My venvs do include the Python version (via symlink to /bin). Don't yours?
    - diggan 1y ago
      If you use something like uv (expanded here: https://news.ycombinator.com/item?id=43904078 https://news.ycombinator.com/item?id=43904078), I think it does. But if you just do `python -m venv .venv`, you get the specific version you used to create the virtual environment with. Some OSes seem to distribute binaries like `python3.8`, `python3.9` and so on so you could do `python3.8 -m venv .venv` to look one env to a specific version, but a bit of a hassle.

ConfusedDog 1y ago

That's a big improvement over Siri tbh (interruption and latency), but Siri's answer generally kind of shorter than this. My general experience with Siri hasn't been great lately. For complex question, it just redirect to ChatGPT with an extra step for me to confirm. Often stops listening when I'm not even finished with my sentence, and gives "I don't know anything about that" way too often.

XCSme 1y ago

The demo reminded me of this amazing post: https://sambleckley.com/writing/church-of-interruption.html https://sambleckley.com/writing/church-of-interruption.html

singularity2001 1y ago

Voice in Text out is the way to go except for very simple Use Cases / questions.

rane 1y ago

What are currently the best options for low latency TTS and STT as external services? If you want to host an app with these capabilities on a VPS, anything that requires a GPU doesn't seem feasible.

jak0 1y ago

Can this be tweaked somehow to try to reproduce the experience of Aqua Voice? https://withaqua.com/ https://withaqua.com/

glossardi 1y ago

I need this for my next app! Great job, thank you for sharing!