koljab

6 ms·

1.
by koljab 1y ago
Didn't find many coqui finetunes too so far. I have David Attenborough and Snoop Dogg finetunes on my huggingface, quality is medium. Coqui can to 17 languages. The problem with RealtimeVoiceChat repo is turn detection, the model I use
2.
by koljab 1y ago
Yes, you're absolutely right. I'll provide UV and conda support soon, especially for Windows. I'm using python 3.10 still, maybe that's the issue. You can always mail me your current problem or log an issue, I really car
3.
by koljab 1y ago
That would be absolutely awesome. But I doubt it, since they released a shitty version of that amazing thing they put online. I feel they aren't planning to give us their top model soon.
4.
by koljab 1y ago
Maybe possible, I did not look into that much for Coqui XTTS. What i know is that the quantized versions for Orpheus sound noticably worse. I feel audio models are quite sensitive to quantization.
5.
by koljab 1y ago
LLM and TTS latency get's determined and logged at the start. It's around 220ms for the LLM returning the first synthesizable sentence fragment (depending on the length of the fragment, which is usually something between 3 and 10
6.
by koljab 1y ago
Lasinya voice is a XTTS 2.0.2 finetune I made with a self-created, synthesized dataset. I used https://github.com/daswer123/xtts-finetune-webui for training.
7.
by koljab 1y ago
With the current 24b LLM model it's 24 GB. I have no clue how far down you can go with the GPU is using smaller models, you can set the model in server.py. Quite sure 16 GB will work but at some point it will probably fail.
8.
by koljab 1y ago
Yeah I know the voice polarizes, I trained it for myself, so it's not an official release. You can change the voice here: https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a... Create
9.
by koljab 1y ago
Did not look into that one. Looks quite good, I will try that soon.
10.
by koljab 1y ago
Yes, I tested it. I'm not that sure what they created there. It adds some noticable latency compared towards using raw websockets. Imho it's not supposed to, but it did it nevertheless in my tests.
11.
by koljab 1y ago
That would be quite easy to integrate. RealtimeSTT already has wakeword support for both pvporcupine and openwakewords.
12.
by koljab 1y ago
It's not aware. The information that it had been interrupted would be something we can easily add to the next user chat request. Where exactly is harder, because at least for Coqui XTTSv2 we don't have TTS wordstamps (we do have t
13.
by koljab 1y ago
It's in fact using Silero via RealtimeSTT. RealtimeSTT tells when silence starts. Then a binary sentence classification model is used on the realtime transcription text which infers blazingly fast (10ms) and returns a probability betwe
14.
by koljab 1y ago
This character prompt has undergone so many iterations with LLMs it's not funny anymore. "Make her act more bold." - "She again talked about her character description, prevent that!"
15.
by koljab 1y ago
I doubt TTS will be fast enough for realtime without a Nvidia GPU
16.
by koljab 1y ago
Dia is too slow, I need a time to first audio chunk of ~100 milliseconds. Also generations fail too often (artifacts etc)
17.
by koljab 1y ago
All local models: - VAD: Webrtcvad (first fast check) followed by SileroVAD (high compute verification) - Transcription: base.en whisper (CTranslate2) - Turn Detection: KoljaB/SentenceFinishedClassification (selftrained BERT-model) -
18.
by koljab 1y ago
Thanks a lot, great insights. Exactly the kind of feedback that I need to improve things further.
19.
by koljab 1y ago
Actually good question. I'd say probably not. You can't easily "unlearn" things from the model weights (and even if this alone doesn't help). You could retrain/finetune the model heavily on a single language bu
20.
by koljab 1y ago
That's a great question! My first implementation was interruption on voice activity after echo cancellation. It still had way too many false positives. I changed it to incoming realtime transcription as a trigger. That adds a bit of la
21.
by koljab 1y ago
Tried that one. Quality is great but sometimes generations fail and it's rather slow. Also needs ~13 GB of VRAM, it's not my first choice for voice agents tbh.
22.
by koljab 1y ago
This is coqui xttsv2 because it can be tuned to deliver the first token in under 100 ms. Gives the best balance between quality and speed currently imho. If it's only about quality I'd say there are better models out there.
23.
by koljab 1y ago
I only tested it on my 4090 so far
24.
by koljab 1y ago
I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to ev
25.
Show HN: Real-time AI Voice Chat at ~500ms Latency(github.com)
524 points by koljab 1y ago|227 comments
26.
by koljab 1y ago
I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs. Quick Demo Video (50s): https:&#x2F