5 ms·
Large text-to-speech and speech-to-text models have been greatly improving recently. But I wish there were an offline, on-device, multilingual text-to-speech s
by kibbi 1y ago
Large text-to-speech and speech-to-text models have been greatly improving recently.
But I wish there were an offline, on-device, multilingual text-to-speech solution with good voices for a standard PC — one that doesn't require a GPU, tons of RAM, or max out the CPU.
In my research, I didn't find anything that fits the bill. People often mention Tortoise TTS, but I think it garbles words too often. The only plug-in solution for desktop apps I know of is the commercial and rather pricey Acapela SDK.
I hope someone can shrink those new neural network–based models to run efficiently on a typical computer. Ideally, it should run at under 50% CPU load on an average Windows laptop that’s several years old, and start speaking almost immediately (less than 400ms delay).
The same goes for speech-to-text. Whisper.cpp is fine, but last time I looked, it wasn't able to transcribe audio at real-time speed on a standard laptop.
I'd pay for something like this as long as it's less expensive than Acapela.
(My use case is an AAC app.)
- ZeroTalent 1y agoLook into https://superwhisper.com https://superwhisper.com and their local models. Pretty decent.
- kibbi 1y agoThank you, but they say "Offline models only run really well on Apple Silicon macs."
- ZeroTalent 1y agoMany SOTA apps are, unfortunately, only for Apple M Macs.
- 5kg 1y agoMay I introduce to you https://huggingface.co/canopylabs/orpheus-3b-0.1-ft https://huggingface.co/canopylabs/orpheus-3b-0.1-ft (no affiliation) it's English only afaics.
- kibbi 1y agoThe sample sounds impressive, but based on their claim -- 'Streaming inference is faster than playback even on an A100 40GB for the 3 billion parameter model' -- I don't think this could run on a standard laptop.
- dharmab 1y agoI use Piper for one of my apps. It runs on CPU and doesn't require a GPU. It will run well on a raspberry pi. I found a couple of permissively licensed voices that could handle technical terms without garbling them. However, it is unmaintained and the Apple Silicon build is broken. My app also uses whisper.cpp. It runs in real time on Apple Sillicon or on modern fast CPUs like AMD's gaming CPUs.
- kibbi 1y agoI had already suspected that I hadn't found all the possibilities regarding Tortoise TTS, Coqui, Piper, etc. It is sometimes difficult to determine how good a TTS framework really is. Do you possibly have links to the voices you found?
- dharmab 1y agoHere's my code! https://github.com/dharmab/skyeye/tree/main/pkg/synthesizer https://github.com/dharmab/skyeye/tree/main/pkg/synthesizer
- wingworks 1y agoDid you try Kokoro? You can self host that. https://huggingface.co/spaces/hexgrad/Kokoro-TTS https://huggingface.co/spaces/hexgrad/Kokoro-TTS
- kibbi 1y agoThanks! But I get the impression that with Kokoro, a strong CPU still requires about two seconds to generate one sentence, which is too much of a delay for a TTS voice in an AAC app. I'd rather accept a little compromise regarding the voice and intonation quality, as long as the TTS system doesn't frequently garble words. The AAC app is used on tablet PCs running from battery, so the lower the CPU usage and energy draw, the better.
- SamPatt 1y agoDefinitely give it a try yourself. It's very small and shouldn't be hard to test.
- Ey7NFZ3P0nzAe 1y agoI heard good things about fish audio