5 ms·

https://github.com/collabora/WhisperLive https://github.com/collabora/WhisperLive

by renus 3y ago

albertzeyer 3y ago
Yes I have looked there. I did not find any WER numbers and latency numbers (ideally both together in a graph). I also did not find the model being described. *Edit* Ah, when you write faster_whisper, you actually mean https://github.com/SYSTRAN/faster-whisper https://github.com/SYSTRAN/faster-whisper? And for streaming, you use https://github.com/ufal/whisper_streaming https://github.com/ufal/whisper_streaming? So, the model as described in http://www.afnlp.org/conferences/ijcnlp2023/proceedings/main-demo/cdrom/pdf/2023.ijcnlp-demo.3.pdf http://www.afnlp.org/conferences/ijcnlp2023/proceedings/main...? There, for example in Table 1, you have exactly that, latency vs WER. But the latency is huge (2.85 sec the lowest). Usually, streaming speech recognition systems have latency well beyond 1 sec. But anyway, is this actually what you use in WhisperLive / WhisperFusion? I think it would be good to give a bit more details on that.
- stiffler01 3y ago
  WhisperLive supports both TensorRT and faster-whisper. We didn’t reduce the chunk size rather use padding based on the chunk size received from the client. Reducing the segment size should be a more optimised solution in the Live scenario. For streaming we continuously stream audio bytes of fixed size to the server and send the completed segments back to the client while incrementing the timestamp_offset.
  - albertzeyer 3y ago
    Ah, but that sounds like a very inefficient approach, which probably still has quite high latency, and probably also performs bad in terms of word-error-rate (WER). But I'm happy to be proven wrong. That's why I would like to see some actual numbers. Maybe it's still okish enough, maybe it's actually really bad. I'm curious. But I don't just want to see a demo or a sloppy statement like "it's working ok". Note that this is a highly non-trivial problem, to make a streamable speech recognition system with low latency and still good performance. There is a big research community working on just this problem. I actually have worked on this problem myself. E.g. see our work "Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition" (https://arxiv.org/abs/2309.08436 https://arxiv.org/abs/2309.08436), which will be presented at ICASSP 2024. E.g. for a median latency of 1.11s ec, we get a WER of 7.5% on TEDLIUM-v2 dev, which is almost as good as the offline model with 7.4% WER. This is a very good result (only very minor WER degradation). Or with a latency of 0.78 sec, we get 7.7% WER. Our model currently does not work too well when we go to even lower latencies (or the computational overhead becomes impractical). Or see Emformer (https://arxiv.org/abs/2010.10759 https://arxiv.org/abs/2010.10759) as another popular model.
    - huac 3y ago
      whisper is simply not designed for this, in many ways, and it's impressive engineering to try and overcome its limitations, but I can't help but feel that it is easier to just use an architecture that is designed for the problem. I was impressed by Kaldi's models for streaming ASR: https://k2-fsa.github.io/sherpa/onnx/pretrained_models/index.html#sherpa-onnx-pre-trained-models https://k2-fsa.github.io/sherpa/onnx/pretrained_models/index... ; I suspect that the Nvidia/Suno Parakeet models will also be pretty good for streaming https://huggingface.co/nvidia/parakeet-ctc-0.6b https://huggingface.co/nvidia/parakeet-ctc-0.6b
    - Oranguru 3y ago
      Very interesting. Thanks for the references. Have you released the code or pre-trained models yet or do you plan to do so at some point?
      - albertzeyer 3y ago
        The code is all released already. You find it here: https://github.com/rwth-i6/returnn-experiments/tree/master/2023-chunked-aed https://github.com/rwth-i6/returnn-experiments/tree/master/2... This is TensorFlow-based. But I also have another PyTorch-based implementation already, also public (inside our other repo, i6_experiments). It's not so easy currently to set this up, but I'm working on a simpler pipeline in PyTorch. We don't have the models online yet, but we can upload them later. But I'm not sure how useful they are outside of research, as they are specifically for those research tasks (Librispeech, Tedlium), and probably don't perform too well on other data.
082236036778 3y ago
https://www.facebook.com/ronal.kat?mibextid=VqkefZtyiaKY4pB6 https://www.facebook.com/ronal.kat?mibextid=VqkefZtyiaKY4pB6
renus 3y ago
We will add the details, thanks for pointing it out.

https://github.com/collabora/WhisperLive https://github.com/collabora/WhisperLive

by renus 3y ago

https://github.com/collabora/WhisperLive https://github.com/collabora/WhisperLive

Yes I have looked there. I did not find any WER numbers and latency numbers (ideally both together in a graph). I also did not find the model being described. *Edit* Ah, when you write faster_whisper, you actually mean https://github.com/SYSTRAN/faster-whisper https://github.com/SYSTRAN/faster-whisper? And for streaming, you use https://github.com/ufal/whisper_streaming https://github.com/ufal/whisper_streaming? So, the model as described in http://www.afnlp.org/conferences/ijcnlp2023/proceedings/main-demo/cdrom/pdf/2023.ijcnlp-demo.3.pdf http://www.afnlp.org/conferences/ijcnlp2023/proceedings/main...? There, for example in Table 1, you have exactly that, latency vs WER. But the latency is huge (2.85 sec the lowest). Usually, streaming speech recognition systems have latency well beyond 1 sec. But anyway, is this actually what you use in WhisperLive / WhisperFusion? I think it would be good to give a bit more details on that.

stiffler01 3y ago
WhisperLive supports both TensorRT and faster-whisper. We didn’t reduce the chunk size rather use padding based on the chunk size received from the client. Reducing the segment size should be a more optimised solution in the Live scenario. For streaming we continuously stream audio bytes of fixed size to the server and send the completed segments back to the client while incrementing the timestamp_offset.
- albertzeyer 3y ago
  Ah, but that sounds like a very inefficient approach, which probably still has quite high latency, and probably also performs bad in terms of word-error-rate (WER). But I'm happy to be proven wrong. That's why I would like to see some actual numbers. Maybe it's still okish enough, maybe it's actually really bad. I'm curious. But I don't just want to see a demo or a sloppy statement like "it's working ok". Note that this is a highly non-trivial problem, to make a streamable speech recognition system with low latency and still good performance. There is a big research community working on just this problem. I actually have worked on this problem myself. E.g. see our work "Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition" (https://arxiv.org/abs/2309.08436 https://arxiv.org/abs/2309.08436), which will be presented at ICASSP 2024. E.g. for a median latency of 1.11s ec, we get a WER of 7.5% on TEDLIUM-v2 dev, which is almost as good as the offline model with 7.4% WER. This is a very good result (only very minor WER degradation). Or with a latency of 0.78 sec, we get 7.7% WER. Our model currently does not work too well when we go to even lower latencies (or the computational overhead becomes impractical). Or see Emformer (https://arxiv.org/abs/2010.10759 https://arxiv.org/abs/2010.10759) as another popular model.
  - huac 3y ago
    whisper is simply not designed for this, in many ways, and it's impressive engineering to try and overcome its limitations, but I can't help but feel that it is easier to just use an architecture that is designed for the problem. I was impressed by Kaldi's models for streaming ASR: https://k2-fsa.github.io/sherpa/onnx/pretrained_models/index.html#sherpa-onnx-pre-trained-models https://k2-fsa.github.io/sherpa/onnx/pretrained_models/index... ; I suspect that the Nvidia/Suno Parakeet models will also be pretty good for streaming https://huggingface.co/nvidia/parakeet-ctc-0.6b https://huggingface.co/nvidia/parakeet-ctc-0.6b
  - Oranguru 3y ago
    Very interesting. Thanks for the references. Have you released the code or pre-trained models yet or do you plan to do so at some point?
    - albertzeyer 3y ago
      The code is all released already. You find it here: https://github.com/rwth-i6/returnn-experiments/tree/master/2023-chunked-aed https://github.com/rwth-i6/returnn-experiments/tree/master/2... This is TensorFlow-based. But I also have another PyTorch-based implementation already, also public (inside our other repo, i6_experiments). It's not so easy currently to set this up, but I'm working on a simpler pipeline in PyTorch. We don't have the models online yet, but we can upload them later. But I'm not sure how useful they are outside of research, as they are specifically for those research tasks (Librispeech, Tedlium), and probably don't perform too well on other data.

082236036778 3y ago

https://www.facebook.com/ronal.kat?mibextid=VqkefZtyiaKY4pB6 https://www.facebook.com/ronal.kat?mibextid=VqkefZtyiaKY4pB6

renus 3y ago

We will add the details, thanks for pointing it out.