5 ms·

Accents in latent spaces: How AI hears accent strength in English

245 points by ilyausorov 1y ago

rayrah 1y ago
Cool stuff
dgroshev 1y ago
I'm torn. On the one hand, the tech is impressive, and the demo is nicely done. On the other, I think the demo completely misses the point. There's a disconnect between what learners need to learn and what this model optimises for, and it's probably largely explainable by how difficult (maybe even impossible) getting training datasets is. That, and marketing. I believe most learners optimise for two things: being understood [1] and not being grating to the ear [2]. Both goals hinge on acquiring the right set of phonemes and phonetic "tools", because the sets of meaningfully distinct sounds (phonemes) and "tools" rarely match between languages. For example, most (all?) Slavic languages have way fewer meaningfully distinct vowels than English. Meaningfully distinct is the crucial part. Russian word "молоко" as it's most often pronounced has three different vowels, at least two of which would be distinct to an English speaker, but Russian speakers hear that as one-ish vowel. And I mean "hear it": it's not a conscious phenomenon! Phoneme recognition is completely subconscious, so unless specifically trained, people often don't hear the difference between sounds that are obviously different to people who speak that language natively [3]. Same goes for phonetic "tools". English speakers shorten vowels when followed by non-voiced consonants, which makes "heart" and "hard" distinguishable even when t/d are transformed into the same sound (glottal stop or a tap). This "tool" is not available in many languages, so people use it incorrectly and it sounds confusing. So, how would ML models learn this mapping between sounds and phonemes, especially when it's non-local (like with the preceding vowel length)? It's relatively straightforward to find large sets of speech samples labelled with their speakers' backgrounds, but that's just sounds, not phonemes! There is very little signal showing which sound structures matter for humans listening to the sound and which don't. [4] There's also a set of moral issues connected to the "target accent" approach. Teaching learners to acquire an accent that superficially sounds like whatever they chose as a "target" devalues all other accents, that are just as valid and are just as English, because they have the same phonetic system (phonemes + "tools"). It can also make people sound a bit cringe, which I saw first hand. Ideally learners should learn phonetic systems, not superficial accents. That's what makes speech intelligible and natural, even if it's has an exotic flavour [5][6]. Systems like the one the company is building do the opposite. I guess they are easier to build and easier to sell. [1]: On that path lies a nice surprise: being understood and understanding are two sides of the same medal, so learning how to be understood a language learner inevitably starts to understand better. Being able to hear the full set of phonemes is the key to both. [2]: There's a vast, VAST difference between people not paying attention to how someone speaks and them not being able to tell that something's off when prompted. [3]: Nice demonstration for non-Hindi speakers: https://www.youtube.com/watch?v=-I7iUUp-cX8 https://www.youtube.com/watch?v=-I7iUUp-cX8 When isolated and spoken slowly, the consonants might sound different, but in normal speech they sound practically indistinguishable to English speakers with no prior exposure. Native speakers would hear the difference as clear as you would in cap/cup! [4]: Take their viral accent recognition demo. Anecdotally, among three non-native speakers with different backgrounds I talked to, the demo was guessing the mother tongue much better than native speakers, and it errors were different. This is a sign of the model learning to recognise the wrong things. [5]: Ever noticed how films almost always cast native English speakers imitating non-English accents rather than people for whom that's their first language? That's why, English phonetic system with sprinkles of phonetic flavour is much more understandable. [6]: By the way, Arnold Schwarzenegger understands this very well.
- Nopoint2 1y ago
  I don't believe that phonemes are real. They are a product of centuries long literacy, and the stereotypically "foreign" accent results from people being taught to construct the words from phonemes. Not all languages can be neatly split into a nice set of phonemes - Danish phonology in particular seems mostly imaginary, and the "insane grammar" of Old Irish appears to result from the fact that word/morpheme boundaries can occur within the "phonemes".
  - stonelazy 1y ago
    > I don't believe that phonemes are real I used to think the same until I came across the VQ-VAE paper [1], where the goal was to let the model learn the smallest units of speech without any supervision. Interestingly, the model ended up learning units that closely aligned with the phonemes we recognize today. [1] - https://arxiv.org/pdf/1711.00937 https://arxiv.org/pdf/1711.00937
    - Nopoint2 1y ago
      The gibberish it produces shows that something quite different is being encoded. It completely fails with Spanish. There is no reason why it would, if it used phonemes. They think that the footnote explains the poor accuracy, but that is in fact how phonemes supposedly work.
treetalker 1y ago
This is cool and one of the applications of LLMs that I'm actually looking forward to: accent training when acquiring a new language, particularly hearing what you would sound like without an accent! That said, I found the recording of Victor's speech after practicing with the recording of his own unaccented voice to be far less intelligible than his original recording. Looking forward to seeing the developments in this particular application.
- ilyausorov 1y ago
  Fair point! When Victor tried to speed up to speak as fast as Coach Eliza, while it sounded somewhat less accented, a few parts of the phrase did get less intelligible. 10 minutes of practice is only a start after all. Interesting to note that we're also developing a separate measure of intelligibility that will give a separate sense of how intelligible versus accented something is.
georgewsinger 1y ago
This is so cool. Real-time accent feedback is something language learners have never had throughout all of human history, until now. Along similar lines, it would be useful to map a speaker's vowels in vowel-space (and likewise for consonants?) to compare native to non-native speakers. I can't wait until something like this is available for Japanese.
- pjc50 1y ago
  > something language learners have never had throughout all of human history .. unless they had access to a native speaker and/or vocal coach? While an automated Henry Higgins is nifty, it's not something humans haven't been able to do themselves.
  - anadalakra 1y ago
    Native speakers are less helpful at this than you might think. Speech coaches are absolutely the way to go, but they're outside the price range for most people ($200+/hr for a good coach). BoldVoice gives coach-level feedback and instruction at a price point that everyone can access, on demand.
    - astrange 1y ago
      You can take a language class rather than have a personal instructor. Although accents are a sensitive topic so I don't remember mine going into it much.
      - anadalakra 1y ago
        As someone who took English classes for years growing up, I wish that were the case. In fact, most teachers don't really know how to teach pronunciation. Also, in a typical group class setting, it's challenging to give each student one-on-one feedback. On BoldVoice, we solve that with 1) unlimited instant feedback from sound-level AI - your most patient coach. 2) in-depth video lessons from the best coaches in the world (Hollywood accent coaches). I'm a cofounder of BoldVoice, by the way. :)
mckirk 1y ago
Is it just me, or did the sound files get hugged-to-death?
pjc50 1y ago
What the vector-space data gets right, and what the human commentary tends not to, is the idea that accents are a complex statistical distribution. You should be careful about the concept of a "default" or "neutral" accent. Telecommunications has spent the 20th century flattening accents together, as has accent discrimination. There's always the tendency for people to say "my accent is the neutral standard against which all others should be measured".
- lurk2 1y ago
  > There's always the tendency for people to say "my accent is the neutral standard against which all others should be measured". You can measure this by mutual intelligibility with other accent groupings.
  - dqv 1y ago
    Well, no. That would also measure what accents someone has the most exposure to, which doesn't necessarily reflect its "absolute intelligibility", but rather its popularity. Popularity and optimality are not the same thing. You would first need to measure if an accent's popularity is a result of its optimality to make the claim that your measure is accurate.
    - lurk2 1y ago
      > Popularity and optimality are not the same thing. Yes they are. If we lived in a world where Australia was a world superpower we might be having this conversation upside down and with r’s where there shouldn’t be any, but we don’t. Every student wants to learn to speak with an American accent because it has the highest level of intelligibility owing to exposure via cinema, music, expatriate communities, etc.
      - dqv 1y ago
        Then you're not describing mutual intelligibility: > In linguistics, mutual intelligibility is a relationship between different but related language varieties in which speakers of the different varieties can readily understand each other without prior familiarity or special effort. (emphasis mine) https://en.wikipedia.org/wiki/Mutual_intelligibility https://en.wikipedia.org/wiki/Mutual_intelligibility
vessenes 1y ago
This is super cool. A suggestion and some surprise: I’m surprised by your assertion that there’s no clustering. I see the representation shows no clustering, and believe you that there is therefore no broad high-dimensional clustering. I also agree that the demo where Victor’s voice moves closer to Eliza’s sounds more native. But, how can it be that you can show directionality toward “native” without clustering? I would read this as a problem with my embedding, not a feature. Perhaps there are some smaller-dimensional sub-axes that do encode what sort of accent someone has? Suggestion for the BoldVoice team: if you’d like to go viral, I suggest you dig into American idiolects — two that are hard not to talk about / opine on / retweet are AAVE and Gay male speech (not sure if there’s a more formal name for this, it’s what Wikipedia uses). I’m in a mixed race family, and we spent a lot of time playing with ChatGPT’s AAVE abilities which have, I think sadly, been completely nerfed over the releases. Chat seems to have no sense of shame when it says speaking like one of my kids is harmful; I imagine the well intentioned OpenAI folks were sort of thinking the opposite when they cut it out. It seems to have a list of “okay” and “bad” idiolects baked in - for instance, it will give you a thick Irish accent, a Boston accent, a NY/Bronx accent, but no Asian/SE Asian accents. I like the idea of an idiolect-manager, something that could help me move my speech more or less toward a given idiolect. Similarly England is a rich minefield of idiolects, from scouse to highly posh. I’m guessing you guys are aimed at the call center market based on your demo, but there could be a lot more applications! Voice coaches in Hollywood (the good ones) charge hundreds of dollar per hour, so there’s a valuable if small market out there for much of this. Thanks for the demo and write up. Very cool.
- BalinKing 1y ago
  (Minor nitpick, but I think "dialect" is a more appropriate word than "idiolect" here—at least according to Wikipedia, "idiolect" refers to a single person's way of speaking, whereas AAVE et al. are shared and are therefore considered dialects.)
  - vessenes 1y ago
    OK, good read for me here. Based on your feedback and some research, I think I should have use ‘sociolect’ for both in that I was less complaining about ChatGPT’s unwillingness to use, say, finna, in a sentence, and more complaining about the vocalized accents. Anyway good catch, thanks!
fxtentacle 1y ago
What a great AI use-case! At first, I felt excited ... But then I read their privacy policy. They want permission to save all of my audio interactions for all eternity. It's so sad that I will never try out their (admittedly super cool) AI tech.
- anadalakra 1y ago
  You can reach out and request your data to be deleted at any time.
  - fxtentacle 1y ago
    "if you wish to opt out of future collection of voice samples, you may do so by disabling voice-related features in the BoldVoice app. Please note that this may limit the functionality of certain services." Yeah, I can opt out. By not using any voice-related feature in their voice training app.
    - anadalakra 1y ago
      If you're still actively using the app, the voice will be retained and processed so that you can receive instant feedback, and also so that you receive additional personalized practice items and video lessons based on your speech needs. If you don't want the samples saved "in perpetuity", you can request them to be deleted once you decide that you're done with the application. Hope this helps!
- 3abiton 1y ago
  I wonder if voice will be treated like facial data in terms of privacy. At least the EU has strong provisions against PII.
joshjhargreaves 1y ago
Damn, this is really cool.
- oscar120 1y ago
  thanks!
adhsu01 1y ago
Super cool work, congrats BoldVoice team! I've always thought that one of the non-obvious applications of voice cloning/matching is the ability to show a language learner what they would sound like with a more native accent.
- oscar120 1y ago
  this^
- ilyausorov 1y ago
  This and more exciting features are coming to the BoldVoice app soon!
asveikau 1y ago
Victor's problem isn't really the vowels or pacing. The final consonants are soft or not really audible. I am not hearing the /ŋ/ of "long" as the most marked example. It sounds closer to "law". In his "improved" recording he hasn't fixed this. I sometimes see content on social media encouraging people to sound more native or improve their accent. But IMO it's perfectly ok to have an accent, as long as the speech meets some baseline of intelligibility. (So Victor needs to work on "long" but not "days".) I've even come across people who are trying to mimick a native accent but lose intelligibility, where they'd sound better with their foreign accent. (An example I've seen is a native Spanish speaker trying to imitate the American accent's intervocalic T and D, and I don't understand them. A Spanish /t/ or /d/ would be different from most English language accents, but be way more understandable.)
- anadalakra 1y ago
  "If Victor wanted to move beyond this point, the sound-by-sound phonetic analysis available in the BoldVoice app would allow him to understand the patterns in pronunciation and stress that contribute to Eliza’s accent and teach him how to apply them in his own speech." Indeed Victor would likely receive a personalized lesson and practice on the NG sound on the app.
- JoshTko 1y ago
  Thank you for pinpoints my confusion/disconnect on what lack of improvement that I was sensing. There was an improvement on pacing, and cadence, yes, but that was not the main challenge with Victors accent. Visually I'd say victor improved by at most 5% and not 50% as indicated by the visualization. In some regards it was even harder to understand than the original due to speed and cadence without improvement in core pronunciation.
- gxs 1y ago
  Yeah, as long as it’s intelligible an accent is perfectly fine It’s also perfectly fine to want to sound like a native speaker - whether it be because they are self conscious, think it will benefit them in some way, or simply want to feel like they are speaking “correctly” Sorry to pick on you, it’s just amazing to me how sensitive we are to “inclusivity” to the point where we almost discourage people wanting to fit in
wbroo 1y ago
Very interestng! Have you tested for other factors like speaking speed, emotional tone, or microphone quality to see what else is (or isn’t) influencing model perception?
- ilyausorov 1y ago
  For sure we did! The training data we used for this was purposely highly varied to account for these various factors so they don't cause too much bias in the model. But there's also an error rate regardless of how good you make it. We keep improving!
ccppurcell 1y ago
Oh pssh. There's no such thing as accent strength. There's only accent distance. Accent strength is just an artefact of distance from the accent of a socially dominant group.
- semiquaver 1y ago
  What a silly nitpick. You’re just using different words to say the same thing.
- ilyausorov 1y ago
  Sure, that's fair. We apply labels that have a connotation of strength based on the distance, but the underlying calculation is indeed based on distance.
- dmurray 1y ago
  The article defines accent strength in precisely this way, as the difference "relative to native speakers of English". That group has a vast range of accents, but it's believable that that range occupies an identifiable part of the multi-dimensional accent space, and has very little overlap with, for example, beginner ESL students from China. Even between native speakers, I bet you could come up with some measure of centrality and measure accent strength as a distance from that. And if language families exist upon a continuum - there must be some point on that continuum where you are no longer speaking English, but say Scots or Friesian or Nigerian Creole instead. Accents close to those points are objectively stronger. But there is a lot of freedom in how you measure centrality - if you weight by number of speakers, you might expect to get some mid-American or mid-Atlantic accent, but wind up with the dialect of semi-literate Hyderabad call centre workers.
  - ilyausorov 1y ago
    Indeed, although the inference output of the model is based on the ratings input that we trained it on. And that rating input was done by American English native speakers, so this iteration of the model is centered towards those accents more than e.g. UK or Australian or other accents of English from outside the US.
Goofy_Coyote 1y ago
Glad to see BoldVoice here. I’ve been using it for a few months, and I can confirm it’s working.
- ilyausorov 1y ago
  Happy to see a happy BoldVoice user. Please don't hesitate to reach out to our team with feedback or thoughts on how we can continue to improve your learning journey. Helping you succeed is our #1 priority!
sonny3690 1y ago
This is some insanely cool work. It's going to help so many people.
- ilyausorov 1y ago
  Thanks, we're doing our best!
childintime 1y ago
I didn't find international english, would have been interesting. Also, the USA writing convention falls short, like "who put the dot inside the string." crazy. Rationals "put the dot after the string". No spelling corrector should change that.
Unearned5161 1y ago
I'm always very entertained when I'm talking with someone and pick up on some very slight deviation from the "norm" in their accent. I think it shows two things: that its near impossible to totally wipe that fingerprint of a past tongue, and that our ears are incredibly adept pieces of tooling
SamBam 1y ago
Like others recently, I've been extremely impressed by LLM's ability to play GeoGuessr, or, more generally, to geo-locate random snapshots that you give them, with what seem (to me) to be almost no context clues. (I gave ChatGPT loads of holiday snapshots, screenshotted to remove metadata, and it did amazingly.) I assume that, with enough training, we could get similarly accurate guesses of a person's linguistic history from their voice data. Obviously it would be extremely tricky for lots of people. For instance, many people think I sound English or Irish. I grew up in France to American parents who both went to Oxford and spent 15 years in England. I wouldn't be surprised, though, if a well-trained model could do much better on my accent than "you sound kinda Irish."
- chris_va 1y ago
  I bet you are right. I had a forensic linguistics TA during college who was able to identify the island in southeast Asia one of the students grew up on, and where they moved to in the UK as a teenager before coming to the US (if I am remembering this story right). From what I gather, there are a lot of clues in how we speak that most brains edit out when parsing language.
  - dhosek 1y ago
    Or the classic scene in Mrs Doubtfire where Pierce Brosnan attempts to locate the origin of Robin Williams’s fake English accent.
- ilyausorov 1y ago
  We actually did something like this for non-native English speakers a few months back. Check out https://accentoracle.com https://accentoracle.com (most mind-blowing if you're a non native English speaker)
  - reactormonk 1y ago
    Swiss-German accent doesn't seem to be on the list, so it guessed mostly Swedish.
  - 1y ago
dgan 1y ago
wow always wanted to know an objective measure of my Russian accent in French. I ve been living here for a long, long time and some people tell me it's impossible to recognise where i come from. i d like to put that to test
oezi 1y ago
Did you publish that accent dataset somewhere?
- ilyausorov 1y ago
  No, the dataset isn't published beyond what you see on the 2D visualization. Sorry.
- AJoxo 1y ago
  you may be interested in Mozillas CommonVoice dataset
  - oezi 1y ago
    Does CommonVoice have any kind of tagging related to the accent/skill of the speaker?
ccheever 1y ago
This is really cool. Just had an employee at our company start expensing BoldVoice. Being able to be understood more easily is a big deal for global remote employees. (Note - I am a small investor in BoldVoice)
runelohrhauge 1y ago
This is fascinating work. Love seeing how you’re combining machine learning with practical coaching to support real accent improvement. The concept of an “accent fingerprint” is especially clever, and the visualization of progress in latent space really brings it to life. Excited to see where you take this next!
WhitneyLand 1y ago
The hear my own voice without an accent thing is a really cool party trick. I’d consider making this feature available free with super low friction, maybe no signup required, to get some viral traction.
- ilyausorov 1y ago
  What if it was already available? Try it out at https://accentfilter.com https://accentfilter.com!
  - PaulDavisThe1st 1y ago
    Hmmm. Initially impressive but upon retries and reflection ... not that great. It doesn't even maintain timing ... unless that's part of the transform.
    - ilyausorov 1y ago
      Indeed yeah that’s one of the key weaknesses of the approach that we’re using. It overrides the speakers cadence and accent while keeping their voice profile / timbre in place. Different techniques may not do this but also may not copy over the accent to the resulting clip as effectively. So far we’re using this to support pedagogical (and lead-gen) use cases where we think it works sufficiently enough.
      - PaulDavisThe1st 1y ago
        Let's put it a different way. I grew up in the UK till 24. I've lived in the USA for 36 years. The UK/US accent conversions dramatically altered my voice/accent; the AU one left it mostly unchanged. This is offensive :))
  - lukeinator42 1y ago
    Is the approach being used to do accented TTS (or just reference recordings), and then a tone color conversion model that just changes the timbre? Because if I say a completely different sentence it still says the original words, haha.
deleted 1y ago
[deleted]
sardines 1y ago
How's the "accent conversion model" work? Is it all embedding based? If so—and if you want to transfer-learn new downstream models from embeddings—then seems to me you are onto a very effective way of doing data augmentation. It's expensive to do data augmentation on raw waveforms since you always need to run the STFT again; but if you've pre-computed & cached embeddings and can do data augmentation there, it would be super fast.
avalys 1y ago
I (an American from suburban Connecticut) was recently in London for an event and someone misheard me. Another Londoner said “It’s because of your accent!”, which of course was nonsense to me. What accent? I’d be really interested to play with this tool and see what it thinks of my accent. Can it tell where I grew up? Can it tell what my parents’ native languages are (not English!) A free tool like this would be great marketing for this company.
- ilyausorov 1y ago
  We did built two free tools, which are geared towards non-native English speakers. You can find them at https://accentoracle.com https://accentoracle.com and https://accentfilter.com https://accentfilter.com. They're less effective for English native speakers, but could still be fun.
- dhosek 1y ago
  What I find interesting is that it seems that folks from the UK tend to focus on consonants in distinguishing accents while in the US we distinguish more on vowels.
Muromec 1y ago
Scammers around here calling me saying my tax number is blocked will be happy to mask their accent with the one they are supposed to have. They will probably buy a premium subscription for this too. Best kind of AI in the best kind of a free market.
cmcconomy 1y ago
This could be cool to apply as an acting aid, for actors trying to get in the role of a particular dialect-speaker.
fernandorojo 1y ago
This is cool. I hadn’t come across an objective measurement of accents before.
az09mugen 1y ago
Unrelated with th article itself, but I strongly disagree with the usage of the verb "hears" used in the title. This verb is used for living organisms, and AFAIK, AI is not living, nor an organism. This kind of subtle yet nasty personification is what encourages people to believe more in the fact that an AI "thinks" by itself or by extension that "AGI" is really close, or even worse, tricks more easily people into believe AI outputs. "AI detects" would have been way more suitable.

245 points by ilyausorov 1y ago

Cool stuff

I'm torn. On the one hand, the tech is impressive, and the demo is nicely done. On the other, I think the demo completely misses the point. There's a disconnect between what learners need to learn and what this model optimises for, and it's probably largely explainable by how difficult (maybe even impossible) getting training datasets is. That, and marketing. I believe most learners optimise for two things: being understood [1] and not being grating to the ear [2]. Both goals hinge on acquiring the right set of phonemes and phonetic "tools", because the sets of meaningfully distinct sounds (phonemes) and "tools" rarely match between languages. For example, most (all?) Slavic languages have way fewer meaningfully distinct vowels than English. Meaningfully distinct is the crucial part. Russian word "молоко" as it's most often pronounced has three different vowels, at least two of which would be distinct to an English speaker, but Russian speakers hear that as one-ish vowel. And I mean "hear it": it's not a conscious phenomenon! Phoneme recognition is completely subconscious, so unless specifically trained, people often don't hear the difference between sounds that are obviously different to people who speak that language natively [3]. Same goes for phonetic "tools". English speakers shorten vowels when followed by non-voiced consonants, which makes "heart" and "hard" distinguishable even when t/d are transformed into the same sound (glottal stop or a tap). This "tool" is not available in many languages, so people use it incorrectly and it sounds confusing. So, how would ML models learn this mapping between sounds and phonemes, especially when it's non-local (like with the preceding vowel length)? It's relatively straightforward to find large sets of speech samples labelled with their speakers' backgrounds, but that's just sounds, not phonemes! There is very little signal showing which sound structures matter for humans listening to the sound and which don't. [4] There's also a set of moral issues connected to the "target accent" approach. Teaching learners to acquire an accent that superficially sounds like whatever they chose as a "target" devalues all other accents, that are just as valid and are just as English, because they have the same phonetic system (phonemes + "tools"). It can also make people sound a bit cringe, which I saw first hand. Ideally learners should learn phonetic systems, not superficial accents. That's what makes speech intelligible and natural, even if it's has an exotic flavour [5][6]. Systems like the one the company is building do the opposite. I guess they are easier to build and easier to sell. [1]: On that path lies a nice surprise: being understood and understanding are two sides of the same medal, so learning how to be understood a language learner inevitably starts to understand better. Being able to hear the full set of phonemes is the key to both. [2]: There's a vast, VAST difference between people not paying attention to how someone speaks and them not being able to tell that something's off when prompted. [3]: Nice demonstration for non-Hindi speakers: https://www.youtube.com/watch?v=-I7iUUp-cX8 https://www.youtube.com/watch?v=-I7iUUp-cX8 When isolated and spoken slowly, the consonants might sound different, but in normal speech they sound practically indistinguishable to English speakers with no prior exposure. Native speakers would hear the difference as clear as you would in cap/cup! [4]: Take their viral accent recognition demo. Anecdotally, among three non-native speakers with different backgrounds I talked to, the demo was guessing the mother tongue much better than native speakers, and it errors were different. This is a sign of the model learning to recognise the wrong things. [5]: Ever noticed how films almost always cast native English speakers imitating non-English accents rather than people for whom that's their first language? That's why, English phonetic system with sprinkles of phonetic flavour is much more understandable. [6]: By the way, Arnold Schwarzenegger understands this very well.

Nopoint2 1y ago
I don't believe that phonemes are real. They are a product of centuries long literacy, and the stereotypically "foreign" accent results from people being taught to construct the words from phonemes. Not all languages can be neatly split into a nice set of phonemes - Danish phonology in particular seems mostly imaginary, and the "insane grammar" of Old Irish appears to result from the fact that word/morpheme boundaries can occur within the "phonemes".
- stonelazy 1y ago
  > I don't believe that phonemes are real I used to think the same until I came across the VQ-VAE paper [1], where the goal was to let the model learn the smallest units of speech without any supervision. Interestingly, the model ended up learning units that closely aligned with the phonemes we recognize today. [1] - https://arxiv.org/pdf/1711.00937 https://arxiv.org/pdf/1711.00937
  - Nopoint2 1y ago
    The gibberish it produces shows that something quite different is being encoded. It completely fails with Spanish. There is no reason why it would, if it used phonemes. They think that the footnote explains the poor accuracy, but that is in fact how phonemes supposedly work.

treetalker 1y ago

This is cool and one of the applications of LLMs that I'm actually looking forward to: accent training when acquiring a new language, particularly hearing what you would sound like without an accent! That said, I found the recording of Victor's speech after practicing with the recording of his own unaccented voice to be far less intelligible than his original recording. Looking forward to seeing the developments in this particular application.

ilyausorov 1y ago
Fair point! When Victor tried to speed up to speak as fast as Coach Eliza, while it sounded somewhat less accented, a few parts of the phrase did get less intelligible. 10 minutes of practice is only a start after all. Interesting to note that we're also developing a separate measure of intelligibility that will give a separate sense of how intelligible versus accented something is.

georgewsinger 1y ago

This is so cool. Real-time accent feedback is something language learners have never had throughout all of human history, until now. Along similar lines, it would be useful to map a speaker's vowels in vowel-space (and likewise for consonants?) to compare native to non-native speakers. I can't wait until something like this is available for Japanese.

pjc50 1y ago
> something language learners have never had throughout all of human history .. unless they had access to a native speaker and/or vocal coach? While an automated Henry Higgins is nifty, it's not something humans haven't been able to do themselves.
- anadalakra 1y ago
  Native speakers are less helpful at this than you might think. Speech coaches are absolutely the way to go, but they're outside the price range for most people ($200+/hr for a good coach). BoldVoice gives coach-level feedback and instruction at a price point that everyone can access, on demand.
  - astrange 1y ago
    You can take a language class rather than have a personal instructor. Although accents are a sensitive topic so I don't remember mine going into it much.
    - anadalakra 1y ago
      As someone who took English classes for years growing up, I wish that were the case. In fact, most teachers don't really know how to teach pronunciation. Also, in a typical group class setting, it's challenging to give each student one-on-one feedback. On BoldVoice, we solve that with 1) unlimited instant feedback from sound-level AI - your most patient coach. 2) in-depth video lessons from the best coaches in the world (Hollywood accent coaches). I'm a cofounder of BoldVoice, by the way. :)

mckirk 1y ago

Is it just me, or did the sound files get hugged-to-death?

pjc50 1y ago

What the vector-space data gets right, and what the human commentary tends not to, is the idea that accents are a complex statistical distribution. You should be careful about the concept of a "default" or "neutral" accent. Telecommunications has spent the 20th century flattening accents together, as has accent discrimination. There's always the tendency for people to say "my accent is the neutral standard against which all others should be measured".

lurk2 1y ago
> There's always the tendency for people to say "my accent is the neutral standard against which all others should be measured". You can measure this by mutual intelligibility with other accent groupings.
- dqv 1y ago
  Well, no. That would also measure what accents someone has the most exposure to, which doesn't necessarily reflect its "absolute intelligibility", but rather its popularity. Popularity and optimality are not the same thing. You would first need to measure if an accent's popularity is a result of its optimality to make the claim that your measure is accurate.
  - lurk2 1y ago
    > Popularity and optimality are not the same thing. Yes they are. If we lived in a world where Australia was a world superpower we might be having this conversation upside down and with r’s where there shouldn’t be any, but we don’t. Every student wants to learn to speak with an American accent because it has the highest level of intelligibility owing to exposure via cinema, music, expatriate communities, etc.
    - dqv 1y ago
      Then you're not describing mutual intelligibility: > In linguistics, mutual intelligibility is a relationship between different but related language varieties in which speakers of the different varieties can readily understand each other without prior familiarity or special effort. (emphasis mine) https://en.wikipedia.org/wiki/Mutual_intelligibility https://en.wikipedia.org/wiki/Mutual_intelligibility

vessenes 1y ago

This is super cool. A suggestion and some surprise: I’m surprised by your assertion that there’s no clustering. I see the representation shows no clustering, and believe you that there is therefore no broad high-dimensional clustering. I also agree that the demo where Victor’s voice moves closer to Eliza’s sounds more native. But, how can it be that you can show directionality toward “native” without clustering? I would read this as a problem with my embedding, not a feature. Perhaps there are some smaller-dimensional sub-axes that do encode what sort of accent someone has? Suggestion for the BoldVoice team: if you’d like to go viral, I suggest you dig into American idiolects — two that are hard not to talk about / opine on / retweet are AAVE and Gay male speech (not sure if there’s a more formal name for this, it’s what Wikipedia uses). I’m in a mixed race family, and we spent a lot of time playing with ChatGPT’s AAVE abilities which have, I think sadly, been completely nerfed over the releases. Chat seems to have no sense of shame when it says speaking like one of my kids is harmful; I imagine the well intentioned OpenAI folks were sort of thinking the opposite when they cut it out. It seems to have a list of “okay” and “bad” idiolects baked in - for instance, it will give you a thick Irish accent, a Boston accent, a NY/Bronx accent, but no Asian/SE Asian accents. I like the idea of an idiolect-manager, something that could help me move my speech more or less toward a given idiolect. Similarly England is a rich minefield of idiolects, from scouse to highly posh. I’m guessing you guys are aimed at the call center market based on your demo, but there could be a lot more applications! Voice coaches in Hollywood (the good ones) charge hundreds of dollar per hour, so there’s a valuable if small market out there for much of this. Thanks for the demo and write up. Very cool.

BalinKing 1y ago
(Minor nitpick, but I think "dialect" is a more appropriate word than "idiolect" here—at least according to Wikipedia, "idiolect" refers to a single person's way of speaking, whereas AAVE et al. are shared and are therefore considered dialects.)
- vessenes 1y ago
  OK, good read for me here. Based on your feedback and some research, I think I should have use ‘sociolect’ for both in that I was less complaining about ChatGPT’s unwillingness to use, say, finna, in a sentence, and more complaining about the vocalized accents. Anyway good catch, thanks!

fxtentacle 1y ago

What a great AI use-case! At first, I felt excited ... But then I read their privacy policy. They want permission to save all of my audio interactions for all eternity. It's so sad that I will never try out their (admittedly super cool) AI tech.

anadalakra 1y ago
You can reach out and request your data to be deleted at any time.
- fxtentacle 1y ago
  "if you wish to opt out of future collection of voice samples, you may do so by disabling voice-related features in the BoldVoice app. Please note that this may limit the functionality of certain services." Yeah, I can opt out. By not using any voice-related feature in their voice training app.
  - anadalakra 1y ago
    If you're still actively using the app, the voice will be retained and processed so that you can receive instant feedback, and also so that you receive additional personalized practice items and video lessons based on your speech needs. If you don't want the samples saved "in perpetuity", you can request them to be deleted once you decide that you're done with the application. Hope this helps!
3abiton 1y ago
I wonder if voice will be treated like facial data in terms of privacy. At least the EU has strong provisions against PII.

joshjhargreaves 1y ago

Damn, this is really cool.

oscar120 1y ago
thanks!

adhsu01 1y ago

Super cool work, congrats BoldVoice team! I've always thought that one of the non-obvious applications of voice cloning/matching is the ability to show a language learner what they would sound like with a more native accent.

oscar120 1y ago
this^
ilyausorov 1y ago
This and more exciting features are coming to the BoldVoice app soon!

asveikau 1y ago

Victor's problem isn't really the vowels or pacing. The final consonants are soft or not really audible. I am not hearing the /ŋ/ of "long" as the most marked example. It sounds closer to "law". In his "improved" recording he hasn't fixed this. I sometimes see content on social media encouraging people to sound more native or improve their accent. But IMO it's perfectly ok to have an accent, as long as the speech meets some baseline of intelligibility. (So Victor needs to work on "long" but not "days".) I've even come across people who are trying to mimick a native accent but lose intelligibility, where they'd sound better with their foreign accent. (An example I've seen is a native Spanish speaker trying to imitate the American accent's intervocalic T and D, and I don't understand them. A Spanish /t/ or /d/ would be different from most English language accents, but be way more understandable.)

anadalakra 1y ago
"If Victor wanted to move beyond this point, the sound-by-sound phonetic analysis available in the BoldVoice app would allow him to understand the patterns in pronunciation and stress that contribute to Eliza’s accent and teach him how to apply them in his own speech." Indeed Victor would likely receive a personalized lesson and practice on the NG sound on the app.
JoshTko 1y ago
Thank you for pinpoints my confusion/disconnect on what lack of improvement that I was sensing. There was an improvement on pacing, and cadence, yes, but that was not the main challenge with Victors accent. Visually I'd say victor improved by at most 5% and not 50% as indicated by the visualization. In some regards it was even harder to understand than the original due to speed and cadence without improvement in core pronunciation.
gxs 1y ago
Yeah, as long as it’s intelligible an accent is perfectly fine It’s also perfectly fine to want to sound like a native speaker - whether it be because they are self conscious, think it will benefit them in some way, or simply want to feel like they are speaking “correctly” Sorry to pick on you, it’s just amazing to me how sensitive we are to “inclusivity” to the point where we almost discourage people wanting to fit in

wbroo 1y ago

Very interestng! Have you tested for other factors like speaking speed, emotional tone, or microphone quality to see what else is (or isn’t) influencing model perception?

ilyausorov 1y ago
For sure we did! The training data we used for this was purposely highly varied to account for these various factors so they don't cause too much bias in the model. But there's also an error rate regardless of how good you make it. We keep improving!

ccppurcell 1y ago

Oh pssh. There's no such thing as accent strength. There's only accent distance. Accent strength is just an artefact of distance from the accent of a socially dominant group.

semiquaver 1y ago
What a silly nitpick. You’re just using different words to say the same thing.
ilyausorov 1y ago
Sure, that's fair. We apply labels that have a connotation of strength based on the distance, but the underlying calculation is indeed based on distance.
dmurray 1y ago
The article defines accent strength in precisely this way, as the difference "relative to native speakers of English". That group has a vast range of accents, but it's believable that that range occupies an identifiable part of the multi-dimensional accent space, and has very little overlap with, for example, beginner ESL students from China. Even between native speakers, I bet you could come up with some measure of centrality and measure accent strength as a distance from that. And if language families exist upon a continuum - there must be some point on that continuum where you are no longer speaking English, but say Scots or Friesian or Nigerian Creole instead. Accents close to those points are objectively stronger. But there is a lot of freedom in how you measure centrality - if you weight by number of speakers, you might expect to get some mid-American or mid-Atlantic accent, but wind up with the dialect of semi-literate Hyderabad call centre workers.
- ilyausorov 1y ago
  Indeed, although the inference output of the model is based on the ratings input that we trained it on. And that rating input was done by American English native speakers, so this iteration of the model is centered towards those accents more than e.g. UK or Australian or other accents of English from outside the US.

Goofy_Coyote 1y ago

Glad to see BoldVoice here. I’ve been using it for a few months, and I can confirm it’s working.

ilyausorov 1y ago
Happy to see a happy BoldVoice user. Please don't hesitate to reach out to our team with feedback or thoughts on how we can continue to improve your learning journey. Helping you succeed is our #1 priority!

sonny3690 1y ago

This is some insanely cool work. It's going to help so many people.

ilyausorov 1y ago
Thanks, we're doing our best!

childintime 1y ago

I didn't find international english, would have been interesting. Also, the USA writing convention falls short, like "who put the dot inside the string." crazy. Rationals "put the dot after the string". No spelling corrector should change that.

Unearned5161 1y ago

I'm always very entertained when I'm talking with someone and pick up on some very slight deviation from the "norm" in their accent. I think it shows two things: that its near impossible to totally wipe that fingerprint of a past tongue, and that our ears are incredibly adept pieces of tooling

SamBam 1y ago

Like others recently, I've been extremely impressed by LLM's ability to play GeoGuessr, or, more generally, to geo-locate random snapshots that you give them, with what seem (to me) to be almost no context clues. (I gave ChatGPT loads of holiday snapshots, screenshotted to remove metadata, and it did amazingly.) I assume that, with enough training, we could get similarly accurate guesses of a person's linguistic history from their voice data. Obviously it would be extremely tricky for lots of people. For instance, many people think I sound English or Irish. I grew up in France to American parents who both went to Oxford and spent 15 years in England. I wouldn't be surprised, though, if a well-trained model could do much better on my accent than "you sound kinda Irish."

chris_va 1y ago
I bet you are right. I had a forensic linguistics TA during college who was able to identify the island in southeast Asia one of the students grew up on, and where they moved to in the UK as a teenager before coming to the US (if I am remembering this story right). From what I gather, there are a lot of clues in how we speak that most brains edit out when parsing language.
- dhosek 1y ago
  Or the classic scene in Mrs Doubtfire where Pierce Brosnan attempts to locate the origin of Robin Williams’s fake English accent.
ilyausorov 1y ago
We actually did something like this for non-native English speakers a few months back. Check out https://accentoracle.com https://accentoracle.com (most mind-blowing if you're a non native English speaker)
- reactormonk 1y ago
  Swiss-German accent doesn't seem to be on the list, so it guessed mostly Swedish.
- 1y ago

dgan 1y ago

wow always wanted to know an objective measure of my Russian accent in French. I ve been living here for a long, long time and some people tell me it's impossible to recognise where i come from. i d like to put that to test

oezi 1y ago

Did you publish that accent dataset somewhere?

ilyausorov 1y ago
No, the dataset isn't published beyond what you see on the 2D visualization. Sorry.
AJoxo 1y ago
you may be interested in Mozillas CommonVoice dataset
- oezi 1y ago
  Does CommonVoice have any kind of tagging related to the accent/skill of the speaker?

ccheever 1y ago

This is really cool. Just had an employee at our company start expensing BoldVoice. Being able to be understood more easily is a big deal for global remote employees. (Note - I am a small investor in BoldVoice)

runelohrhauge 1y ago

This is fascinating work. Love seeing how you’re combining machine learning with practical coaching to support real accent improvement. The concept of an “accent fingerprint” is especially clever, and the visualization of progress in latent space really brings it to life. Excited to see where you take this next!

WhitneyLand 1y ago

The hear my own voice without an accent thing is a really cool party trick. I’d consider making this feature available free with super low friction, maybe no signup required, to get some viral traction.

ilyausorov 1y ago
What if it was already available? Try it out at https://accentfilter.com https://accentfilter.com!
- PaulDavisThe1st 1y ago
  Hmmm. Initially impressive but upon retries and reflection ... not that great. It doesn't even maintain timing ... unless that's part of the transform.
  - ilyausorov 1y ago
    Indeed yeah that’s one of the key weaknesses of the approach that we’re using. It overrides the speakers cadence and accent while keeping their voice profile / timbre in place. Different techniques may not do this but also may not copy over the accent to the resulting clip as effectively. So far we’re using this to support pedagogical (and lead-gen) use cases where we think it works sufficiently enough.
    - PaulDavisThe1st 1y ago
      Let's put it a different way. I grew up in the UK till 24. I've lived in the USA for 36 years. The UK/US accent conversions dramatically altered my voice/accent; the AU one left it mostly unchanged. This is offensive :))
- lukeinator42 1y ago
  Is the approach being used to do accented TTS (or just reference recordings), and then a tone color conversion model that just changes the timbre? Because if I say a completely different sentence it still says the original words, haha.

deleted 1y ago

[deleted]

sardines 1y ago

How's the "accent conversion model" work? Is it all embedding based? If so—and if you want to transfer-learn new downstream models from embeddings—then seems to me you are onto a very effective way of doing data augmentation. It's expensive to do data augmentation on raw waveforms since you always need to run the STFT again; but if you've pre-computed & cached embeddings and can do data augmentation there, it would be super fast.

avalys 1y ago

I (an American from suburban Connecticut) was recently in London for an event and someone misheard me. Another Londoner said “It’s because of your accent!”, which of course was nonsense to me. What accent? I’d be really interested to play with this tool and see what it thinks of my accent. Can it tell where I grew up? Can it tell what my parents’ native languages are (not English!) A free tool like this would be great marketing for this company.

ilyausorov 1y ago
We did built two free tools, which are geared towards non-native English speakers. You can find them at https://accentoracle.com https://accentoracle.com and https://accentfilter.com https://accentfilter.com. They're less effective for English native speakers, but could still be fun.
dhosek 1y ago
What I find interesting is that it seems that folks from the UK tend to focus on consonants in distinguishing accents while in the US we distinguish more on vowels.

Muromec 1y ago

Scammers around here calling me saying my tax number is blocked will be happy to mask their accent with the one they are supposed to have. They will probably buy a premium subscription for this too. Best kind of AI in the best kind of a free market.

cmcconomy 1y ago

This could be cool to apply as an acting aid, for actors trying to get in the role of a particular dialect-speaker.

fernandorojo 1y ago

This is cool. I hadn’t come across an objective measurement of accents before.

az09mugen 1y ago

Unrelated with th article itself, but I strongly disagree with the usage of the verb "hears" used in the title. This verb is used for living organisms, and AFAIK, AI is not living, nor an organism. This kind of subtle yet nasty personification is what encourages people to believe more in the fact that an AI "thinks" by itself or by extension that "AGI" is really close, or even worse, tricks more easily people into believe AI outputs. "AI detects" would have been way more suitable.