7 ms·

StreamVC: Real-Time Low-Latency Voice Conversion

99 points by trevett 2y ago

LisaST 2y ago
[flagged]
gnat 2y ago
From the poster: In this work, we propose a light-weight (~20M param.) causal voice conversion solution that can run in real-time with low latency on a commercially available mobile device. The key design elements are: (1) using a causal encoder to learn soft speech units; (2) injecting whitened f0 to improve pitch stability without leaking source speaker info. In our later V2 version, we found that f0 rescaling followed by a NSF-style harmonic-plus-noise conditioning (as is done in RVC) results in better quality.
- deleted 2y ago
  [deleted]
huac 2y ago
The samples were released a while back: https://google-research.github.io/seanet/stream_vc/ https://google-research.github.io/seanet/stream_vc/
- modeless 2y ago
  Not a very good demo page. It's difficult to judge real world quality with such unenthusiastic reading, unrealistic sentences, and unfamiliar voices. Typical of speech papers. It would be much better if celebrities were used as target voices, as we all know what they sound like and can therefore judge quality better. But I suppose that would be too controversial for Google. In general I think it is silly that voice cloning research has focused so much (exclusively?) on cloning voices from just a few seconds of audio. It puts a pretty low ceiling on quality. Many nuances of a person's communication style will not be contained in such a small amount of data. Sure you can match their pitch and timbre, but voice cloning should be more than that.
  - refulgentis 2y ago
    > But I suppose that would be too controversial for Google. You don't have to suppose anything: it is actually settled law that its bad to just willy-nilly use people's voices if you feel like it, even if its just a sound-alike!
    - mik1998 2y ago
      So, what do we do with actual people who have a very similar voice to some "more famous" person? It's quite silly when voices are far away from being unique to a person.
    - kaliqt 2y ago
      Sound-alike is not actionable, and anyone who claims it infringes should be dragged through the courts. Not your voice, not your IP, not your right.
      - huac 2y ago
        except to the extent that your voice may be part of your image, which is actionable: https://en.wikipedia.org/wiki/Midler_v._Ford_Motor_Co https://en.wikipedia.org/wiki/Midler_v._Ford_Motor_Co.
    - modeless 2y ago
      So it's illegal for standup comics to do impressions? Or maybe there is some nuance here?
      - refulgentis 2y ago
        Thanks modeless, I appreciate the comment -- yes, I agree, there is a substantive difference between "us[ing] people's voices" and "[doing an] impression [of someone's voice]". It's good you pointed this out, law can be hard for swes, it can tickle the "unambiguous rule" part of the brain we employ every day.
    - nine_k 2y ago
- ascorbic 2y ago
  For those confused as I was - it's not trying to match the accent of the target speech in those samples, just the timbre. To quote the paper: > Voice conversion refers to altering the style of a speech signal while preserving its linguistic content. While style encompasses many aspects of speech, such as emotion, prosody, accent, and whispering, in this work we focus on the conversion of speaker timbre only while keeping the linguistic and para-linguistic information unchanged.
udev4096 2y ago
Actual paper: https://arxiv.org/pdf/2401.03078 https://arxiv.org/pdf/2401.03078
DrMiaow 2y ago
[flagged]
- deleted 2y ago
  [deleted]
coldblues 2y ago
https://github.com/hrnoh24/stream-vc https://github.com/hrnoh24/stream-vc https://github.com/yuval-reshef/StreamVC https://github.com/yuval-reshef/StreamVC Unofficial implementations of StreamVC
- android521 2y ago
  what is the hardware requirements
judiisis 2y ago
What is the current best Foss(or otherwise) implementation for voice changer/anonymiser?
- coldblues 2y ago
  Last time I checked, it was https://github.com/w-okada/voice-changer https://github.com/w-okada/voice-changer Requires a decent amount of VRAM and runs poorly with pretty bad quality (IMO)
  - ipnon 2y ago
    Once again we see evidence that AI-for-all is not bottlenecked by research but by the physical limitations of compute infrastructure.
    - coldblues 2y ago
      I wouldn't say that when the application is a strung-up Python Frankenstein monster (not to be too demeaning to the author).
    - WithinReason 2y ago
      More efficient architectures are possible. It's bottlenecked by research.
deleted 2y ago
[deleted]
neilk 2y ago
What are the anticipated use cases? I know of one: transgender people often would like to alter the timbre of their voice and spend a lot of time training their voice. At least for online scenarios, this can just do it. But other than that AI voice altering research seems like it benefits mostly scammers? I’m just wondering what they tell themselves they’re doing. I didn’t see this in the paper.
- pgt 2y ago
  From the abstract: "making it applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios."
- thinkski 2y ago
  I suspect one is masking that a call center is in a low wage country, e.g. make customer in U.S. believe they’re talking to someone in U.S. while paying a fraction of the U.S. wage.
  - neilk 2y ago
    Right. I thought of that too, but it doesn’t mask accents, at least not yet I suppose if you could make agents all sound the same they would be interchangeable, and companies always love that. It’s Anjali or Ligaya or Dolores but now they all sound like “Becky”?
- webappguy 2y ago
  RFK Jr.
  - neilk 2y ago
    You’re getting downvoted perhaps because people think you’re saying something political, but I think you mean “a stronger voice for people with physical issues producing speech”. I have a friend who has a faint, scratchy voice because his throat is riddled with benign growths that a surgeon has to dig out of him every few years. Eventually he will probably lose his voice. Maybe?
manishsharan 2y ago
Are there any use cases that is driving this ? Is there a huge burning need for technology ? Are kidnappers and con-men a huge under-served market that Google is hoping to serve ? Deep Fake videos not convincing enough to serve the need of fraudsters ? I am totally against regulating AI but shit like this gives fodder to the other side.
- leobg 2y ago
  > applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios. It’s not a desire I ever had. But maybe people are different? Alternatively, building the solution was so much fun that the question of whether this is a problem that should be solved was never asked.
- Ukv 2y ago
  Voice anonymization is the use-case mentioned by the paper. If you're recording a video or communicating online, having this over your voice would benefit privacy by avoiding revealing your real voice that can be matched back to your face/name/job/etc. I think a lot of people are currently reluctant to use their voice at all online for privacy reasons, resorting to only text. Also allows people uncomfortable with their natural voice, in particular transgender people, to communicate closer to how they wish to be perceived. Or even for someone to use their own natural voice from previous recordings if some temporary or chronic disease/disorder has impaired it. There are probably a bunch of creative applications - like doing character voices for a D&D session or reading an audiobook. Obviously depends on the preferences of those involved, and many will currently dislike it on the basis of it being AI, but I think over time we'll see the tech integrated in interesting ways. I imagine the majority of the use will be in entertainment/memes/satire - joining a call with an amusing voice on, or the equivalent of Snapchat's face filters. Not something critical that we couldn't do without, but still a fun application. I don't see much benefit to kidnappers in this; if you just need to send an anonymous message without much concern about flow and latency, text or traditional TTS is fine.
- TylerE 2y ago
  Since the quality is pretty listenable, one use case I can see is youtubers who want to do voiceovers on their videos, but not be linked to their real world identity. Heck, I can even see broadcasting uses. Imagine if every on-air personality had good target files made ahead of time, so then when they catch a cold, production runs their lapel mic feed through this, using the "good" target sample, and remove all the congestion and raspiness.
froglus 2y ago
is it like discord or just voice chat, because i like to have things twice!!

99 points by trevett 2y ago

[flagged]

gnat 2y ago

From the poster: In this work, we propose a light-weight (~20M param.) causal voice conversion solution that can run in real-time with low latency on a commercially available mobile device. The key design elements are: (1) using a causal encoder to learn soft speech units; (2) injecting whitened f0 to improve pitch stability without leaking source speaker info. In our later V2 version, we found that f0 rescaling followed by a NSF-style harmonic-plus-noise conditioning (as is done in RVC) results in better quality.

deleted 2y ago
[deleted]

huac 2y ago

The samples were released a while back: https://google-research.github.io/seanet/stream_vc/ https://google-research.github.io/seanet/stream_vc/

modeless 2y ago
Not a very good demo page. It's difficult to judge real world quality with such unenthusiastic reading, unrealistic sentences, and unfamiliar voices. Typical of speech papers. It would be much better if celebrities were used as target voices, as we all know what they sound like and can therefore judge quality better. But I suppose that would be too controversial for Google. In general I think it is silly that voice cloning research has focused so much (exclusively?) on cloning voices from just a few seconds of audio. It puts a pretty low ceiling on quality. Many nuances of a person's communication style will not be contained in such a small amount of data. Sure you can match their pitch and timbre, but voice cloning should be more than that.
- refulgentis 2y ago
  > But I suppose that would be too controversial for Google. You don't have to suppose anything: it is actually settled law that its bad to just willy-nilly use people's voices if you feel like it, even if its just a sound-alike!
  - mik1998 2y ago
    So, what do we do with actual people who have a very similar voice to some "more famous" person? It's quite silly when voices are far away from being unique to a person.
  - kaliqt 2y ago
    Sound-alike is not actionable, and anyone who claims it infringes should be dragged through the courts. Not your voice, not your IP, not your right.
    - huac 2y ago
      except to the extent that your voice may be part of your image, which is actionable: https://en.wikipedia.org/wiki/Midler_v._Ford_Motor_Co https://en.wikipedia.org/wiki/Midler_v._Ford_Motor_Co.
  - modeless 2y ago
    So it's illegal for standup comics to do impressions? Or maybe there is some nuance here?
    - refulgentis 2y ago
      Thanks modeless, I appreciate the comment -- yes, I agree, there is a substantive difference between "us[ing] people's voices" and "[doing an] impression [of someone's voice]". It's good you pointed this out, law can be hard for swes, it can tickle the "unambiguous rule" part of the brain we employ every day.
  - nine_k 2y ago
ascorbic 2y ago
For those confused as I was - it's not trying to match the accent of the target speech in those samples, just the timbre. To quote the paper: > Voice conversion refers to altering the style of a speech signal while preserving its linguistic content. While style encompasses many aspects of speech, such as emotion, prosody, accent, and whispering, in this work we focus on the conversion of speaker timbre only while keeping the linguistic and para-linguistic information unchanged.

udev4096 2y ago

Actual paper: https://arxiv.org/pdf/2401.03078 https://arxiv.org/pdf/2401.03078

DrMiaow 2y ago

[flagged]

deleted 2y ago
[deleted]

coldblues 2y ago

https://github.com/hrnoh24/stream-vc https://github.com/hrnoh24/stream-vc https://github.com/yuval-reshef/StreamVC https://github.com/yuval-reshef/StreamVC Unofficial implementations of StreamVC

android521 2y ago
what is the hardware requirements

judiisis 2y ago

What is the current best Foss(or otherwise) implementation for voice changer/anonymiser?

coldblues 2y ago
Last time I checked, it was https://github.com/w-okada/voice-changer https://github.com/w-okada/voice-changer Requires a decent amount of VRAM and runs poorly with pretty bad quality (IMO)
- ipnon 2y ago
  Once again we see evidence that AI-for-all is not bottlenecked by research but by the physical limitations of compute infrastructure.
  - coldblues 2y ago
    I wouldn't say that when the application is a strung-up Python Frankenstein monster (not to be too demeaning to the author).
  - WithinReason 2y ago
    More efficient architectures are possible. It's bottlenecked by research.

deleted 2y ago

[deleted]

neilk 2y ago

What are the anticipated use cases? I know of one: transgender people often would like to alter the timbre of their voice and spend a lot of time training their voice. At least for online scenarios, this can just do it. But other than that AI voice altering research seems like it benefits mostly scammers? I’m just wondering what they tell themselves they’re doing. I didn’t see this in the paper.

pgt 2y ago
From the abstract: "making it applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios."
thinkski 2y ago
I suspect one is masking that a call center is in a low wage country, e.g. make customer in U.S. believe they’re talking to someone in U.S. while paying a fraction of the U.S. wage.
- neilk 2y ago
  Right. I thought of that too, but it doesn’t mask accents, at least not yet I suppose if you could make agents all sound the same they would be interchangeable, and companies always love that. It’s Anjali or Ligaya or Dolores but now they all sound like “Becky”?
webappguy 2y ago
RFK Jr.
- neilk 2y ago
  You’re getting downvoted perhaps because people think you’re saying something political, but I think you mean “a stronger voice for people with physical issues producing speech”. I have a friend who has a faint, scratchy voice because his throat is riddled with benign growths that a surgeon has to dig out of him every few years. Eventually he will probably lose his voice. Maybe?

manishsharan 2y ago

Are there any use cases that is driving this ? Is there a huge burning need for technology ? Are kidnappers and con-men a huge under-served market that Google is hoping to serve ? Deep Fake videos not convincing enough to serve the need of fraudsters ? I am totally against regulating AI but shit like this gives fodder to the other side.

leobg 2y ago
> applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios. It’s not a desire I ever had. But maybe people are different? Alternatively, building the solution was so much fun that the question of whether this is a problem that should be solved was never asked.
Ukv 2y ago
Voice anonymization is the use-case mentioned by the paper. If you're recording a video or communicating online, having this over your voice would benefit privacy by avoiding revealing your real voice that can be matched back to your face/name/job/etc. I think a lot of people are currently reluctant to use their voice at all online for privacy reasons, resorting to only text. Also allows people uncomfortable with their natural voice, in particular transgender people, to communicate closer to how they wish to be perceived. Or even for someone to use their own natural voice from previous recordings if some temporary or chronic disease/disorder has impaired it. There are probably a bunch of creative applications - like doing character voices for a D&D session or reading an audiobook. Obviously depends on the preferences of those involved, and many will currently dislike it on the basis of it being AI, but I think over time we'll see the tech integrated in interesting ways. I imagine the majority of the use will be in entertainment/memes/satire - joining a call with an amusing voice on, or the equivalent of Snapchat's face filters. Not something critical that we couldn't do without, but still a fun application. I don't see much benefit to kidnappers in this; if you just need to send an anonymous message without much concern about flow and latency, text or traditional TTS is fine.
TylerE 2y ago
Since the quality is pretty listenable, one use case I can see is youtubers who want to do voiceovers on their videos, but not be linked to their real world identity. Heck, I can even see broadcasting uses. Imagine if every on-air personality had good target files made ahead of time, so then when they catch a cold, production runs their lapel mic feed through this, using the "good" target sample, and remove all the congestion and raspiness.

froglus 2y ago

is it like discord or just voice chat, because i like to have things twice!!