6 ms·

I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TD

by koljab 1y ago

I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to evaluate these latest models, they look strong.

ivape 1y ago
Yeah, I figured you would know. Thanks for that, bookmarking that asr leaderboard.
kristopolous 1y ago
https://yummy-fir-7a4.notion.site/dia https://yummy-fir-7a4.notion.site/dia is the new hotness.
- koljab 1y ago
  Tried that one. Quality is great but sometimes generations fail and it's rather slow. Also needs ~13 GB of VRAM, it's not my first choice for voice agents tbh.
  - kristopolous 1y ago
    alright, dumb question. (1) I assume these things can do multiple languages (2) Given (1), can you strip all the languages you aren't using and speed things up?
    - koljab 1y ago
      Actually good question. I'd say probably not. You can't easily "unlearn" things from the model weights (and even if this alone doesn't help). You could retrain/finetune the model heavily on a single language but again that alone does not speed up inference. To gain speed you'd have to bring the parameter count down and train the model from scratch with a single language only. That might work but it's also quite probable that it introduces other issues in the synthesis. In a perfect world the model would only use all that "free parameters" not used now for other languages for a better synthesis of that single trained language. Might be true to a certain degree, but it's not exactly how ai parameter scaling works.
      - nardi 1y ago
        I don't know what I'm talking about, but could you use distillation techniques?
        koljab 1y ago
        Maybe possible, I did not look into that much for Coqui XTTS. What i know is that the quantized versions for Orpheus sound noticably worse. I feel audio models are quite sensitive to quantization.
oezi 1y ago
Paraket is english only. Stick with Whisper. The core innovation is happening in TTS at the moment.

by koljab 1y ago

ivape 1y ago

Yeah, I figured you would know. Thanks for that, bookmarking that asr leaderboard.

kristopolous 1y ago

https://yummy-fir-7a4.notion.site/dia https://yummy-fir-7a4.notion.site/dia is the new hotness.

koljab 1y ago
Tried that one. Quality is great but sometimes generations fail and it's rather slow. Also needs ~13 GB of VRAM, it's not my first choice for voice agents tbh.
- kristopolous 1y ago
  alright, dumb question. (1) I assume these things can do multiple languages (2) Given (1), can you strip all the languages you aren't using and speed things up?
  - koljab 1y ago
    Actually good question. I'd say probably not. You can't easily "unlearn" things from the model weights (and even if this alone doesn't help). You could retrain/finetune the model heavily on a single language but again that alone does not speed up inference. To gain speed you'd have to bring the parameter count down and train the model from scratch with a single language only. That might work but it's also quite probable that it introduces other issues in the synthesis. In a perfect world the model would only use all that "free parameters" not used now for other languages for a better synthesis of that single trained language. Might be true to a certain degree, but it's not exactly how ai parameter scaling works.
    - nardi 1y ago
      I don't know what I'm talking about, but could you use distillation techniques?
      - koljab 1y ago
        Maybe possible, I did not look into that much for Coqui XTTS. What i know is that the quantized versions for Orpheus sound noticably worse. I feel audio models are quite sensitive to quantization.

oezi 1y ago

Paraket is english only. Stick with Whisper. The core innovation is happening in TTS at the moment.