renus

6 ms·

1.
Breaking Language Barriers - Moving closer towards production-ready Hindi ASR(collabora.com)
9 points by renus 1y ago|0 comments
2.
by renus 3y ago
Everything runs locally, we use: - WhisperLive for the transcription - https://github.com/collabora/WhisperLive - WhisperSpeech for the text-to-speech - https://github.com/collabora/WhisperSpeech
3.
by renus 3y ago
We are going to put the sample interface into the Docker, so it's more mainly: > docker run --gpus all --shm-size 64G -p 80:80 -it ghcr.io/collabora/whisperfusion:latest instead of: > docker run --gpus all --shm-size 64
4.
by renus 3y ago
A fast turnaround time is also super important; if the transcription is not correct, waiting multiple seconds for each turn would kill the application. E.g., ordering food using voice is only convenient if it gets me right all the time; if
5.
by renus 3y ago
To streamline the experience we don't send the transcription to the LLM after the pause, since we are using the time we wait for the end of sentence trigger (pause) to generate the LLM and text-to-speech output. So ideally once we dete
6.
by renus 3y ago
We tested https://huggingface.co/cognitivecomputations/dolphin-2_6-phi... as well, in some tasks it performs better. That said, you can use Mistral as well, we support a few models through TensorRT-LLM.
7.
by renus 3y ago
pyryt posted https://arxiv.org/abs/2010.10874 , which might be helpful here, but we probably end off with personalized models that learned from conversation styles. A magic stop/processing word would be the easiest
8.
by renus 3y ago
We will add the details, thanks for pointing it out.
9.
by renus 3y ago
https://github.com/collabora/WhisperLive
10.
by renus 3y ago
Good point; another area we are currently looking into is predicting intention; often, when talking to someone, we have a good idea of what that person might say next. That would not only help with latency but also, allow us to give better
11.
by renus 3y ago
For the transcription part, we are looking into W2v-BERT 2.0 as well and will make it available in a live-streaming context. That said, Whisper, especially small (<50ms), is not as compute-heavy; right now, most of the compute is consume
12.
by renus 3y ago
WhisperLive builds upon the Whisper model; for the demo, we used small.en, but you can also use large without introducing a bigger latency for the overall pipeline since the transcription process is decoupled from the LLM and text-to-speech
13.
by renus 3y ago
WhisperFusion is fully open-source - https://github.com/collabora/WhisperFusion
14.
by renus 3y ago
Curious to see how this works? Check out our demo page - https://col.la/transcription to generate your own transcription, summary, and translation, or use our browser extension - https://github.com/collabora
15.
by renus 3y ago
Using state-of-the-art natural language processing techniques, we developed an AI-driven automatic transcription, summarization, and translation pipeline.
16.
AI-Crafted Daily Digest: Exploring Latest ML Developments(podcast.mlcult.org)
1 points by renus 3y ago|0 comments
17.
Deep Learning Optimizer Visualization(vis.ensmallen.org)
77 points by renus 7y ago|4 comments
18.
Deep learning optimizer visualization(mlpack.org)
2 points by renus 9y ago|0 comments