6 ms·

Parakeet is incredibly fast and accurate even on CPU, and it supports streaming now also in TDT3.

parl_match 2mo ago
Apple likely needed a model that ran on their NPU natively. - parakeet usually runs on Bfloat16. NPU doesn't support that - CPU is not as fast as the NPU for these ops on A-series, and even on modern CPUs, there's a latency delay - Parakeet latency is fine but "fine" may not be good enough for Apple's UX team. - CPU increases power consumption over dedicated float blocks So I would say that Parakeet was a non-option for Apple to ship, although it should be in the benchmarks anyways!
- foobarqux 2mo ago
  Fluidaudio implements Parakeet on ANE. I'd like to know how SpeechAnalyzer compares in speed. https://github.com/FluidInference/FluidAudio https://github.com/FluidInference/FluidAudio
  - foobarqux 2mo ago
    Just tried test using yap on a single ~1hr mp3: yap/Speechanalyzer is about 50% slower than fluidaudio on M1. yap interface is nicer though. https://github.com/finnvoor/yap https://github.com/finnvoor/yap
  - hamza_q_ 2mo ago
    Recently contributed a patch to FluidAudio that sped up Parakeet V2 and V3 to 320x and 282x faster than real time, respectively: https://github.com/FluidInference/FluidAudio/pull/507 https://github.com/FluidInference/FluidAudio/pull/507 That means one hour of audio transcribed in 11.25 and 12.75 seconds. The Inscribe post doesn't give a speed factor for SpeechAnalyzer. However, this Argmax blog post reports 70: https://www.argmaxinc.com/blog/apple-and-argmax https://www.argmaxinc.com/blog/apple-and-argmax Based on that, FluidAudio is ~4.6x and ~4.0x faster.
    - foobarqux 2mo ago
      The difference using an mp3 seems to be smaller: yap seems to use about the same time but fluidaudio seems to take twice as long. Do you happen to know why?
      - hamza_q_ 2mo ago
        Investigated this and it turned out to be an amusing bug: audio decoding was happening three times instead of just once lol. I've put up a PR to remove the wasteful redundant decoding: https://github.com/FluidInference/FluidAudio/pull/799 https://github.com/FluidInference/FluidAudio/pull/799 With the updated PR code, ran a test comparing transcribing (using Parakeet V3) a 1 hr stereo 44.1 kHz mp3 vs the same audio in 16 kHz mono wav format. The result was about 21.3% slower with the mp3 vs the wav, i.e. that's the overhead of decoding + resampling. Currently the decoding + resampling is done up front. If it was done in a pipelined fashion with the inference, that overhead can be eliminated. This is what I did in a recent app I made: https://apps.apple.com/us/app/drea-podcast-ad-blocker/id6759070798 https://apps.apple.com/us/app/drea-podcast-ad-blocker/id6759... It uses FluidAudio as well, but I forked it and replaced the audio decoding code to (a) use mpg123 instead of the native Apple API and (b) do audio decoding and inference in a pipelined fashion. These two changes effectively eliminated the overhead. mpg123 is quite a bit faster than the native Apple API at mp3 decoding (has some very optimized arm64 assembly routines), and the pipelining ensures that the inference is never starved by the mp3 decoding. Contributing this pipelined setup to FluidAudio would be good.
        satvikpendem 2mo ago
        That app was exactly what I was looking for, something like SponsorBlock but for podcasts but I suppose using AI for finding the ads works too. Any chance it'll release on Android?
        hamza_q_ 2mo ago
        Yep it's something I wanted for a while too; there were existing apps that did this, but had two issues: they were paid, and the UI was subpar. So for mine, I made sure it's fully free and that the UI is on par with Apple Podcasts, Spotify, etc. Making the ad-finding cheap enough such that I could make it free turned out to be harder than expected. The main issue you run into is dynamic, location-targeted ads. So I came up with a novel technique that uses Shazam-style audio fingerprints for accurate matching, instead of their normal use case, which is identification. This technique is what allows the ad finding to be very cheap, allowing me to make it free. The SponsorBlock model would actually not work for podcasts, due to dynamic ads. I.e. the location and content of the ads in episodes these days varies by download location. You need the media to be static, like YouTube, for SponsorBlock model to work. Therefore, using an LLM to find the ads + the fingerprints matching in combination is an efficient technique. Android has def been the most requested thing thus far haha. It'll be a decent undertaking due to me having written the app fully in Swift, i.e. it'll be a complete rewrite. I'll also need to replace FluidAudio with some good, fast Android equivalent. The goal of making this app was to create something impressive so that I could get a job. Haven't gotten a job yet, but if and when I do, then I'll have time & resources to think about doing an Android version. Currently a bit stressed and occupied from the job search lol.

Apple likely needed a model that ran on their NPU natively. - parakeet usually runs on Bfloat16. NPU doesn't support that - CPU is not as fast as the NPU for these ops on A-series, and even on modern CPUs, there's a latency delay - Parakeet latency is fine but "fine" may not be good enough for Apple's UX team. - CPU increases power consumption over dedicated float blocks So I would say that Parakeet was a non-option for Apple to ship, although it should be in the benchmarks anyways!

foobarqux 2mo ago
Fluidaudio implements Parakeet on ANE. I'd like to know how SpeechAnalyzer compares in speed. https://github.com/FluidInference/FluidAudio https://github.com/FluidInference/FluidAudio
- foobarqux 2mo ago
  Just tried test using yap on a single ~1hr mp3: yap/Speechanalyzer is about 50% slower than fluidaudio on M1. yap interface is nicer though. https://github.com/finnvoor/yap https://github.com/finnvoor/yap
- hamza_q_ 2mo ago
  Recently contributed a patch to FluidAudio that sped up Parakeet V2 and V3 to 320x and 282x faster than real time, respectively: https://github.com/FluidInference/FluidAudio/pull/507 https://github.com/FluidInference/FluidAudio/pull/507 That means one hour of audio transcribed in 11.25 and 12.75 seconds. The Inscribe post doesn't give a speed factor for SpeechAnalyzer. However, this Argmax blog post reports 70: https://www.argmaxinc.com/blog/apple-and-argmax https://www.argmaxinc.com/blog/apple-and-argmax Based on that, FluidAudio is ~4.6x and ~4.0x faster.
  - foobarqux 2mo ago
    The difference using an mp3 seems to be smaller: yap seems to use about the same time but fluidaudio seems to take twice as long. Do you happen to know why?
    - hamza_q_ 2mo ago
      Investigated this and it turned out to be an amusing bug: audio decoding was happening three times instead of just once lol. I've put up a PR to remove the wasteful redundant decoding: https://github.com/FluidInference/FluidAudio/pull/799 https://github.com/FluidInference/FluidAudio/pull/799 With the updated PR code, ran a test comparing transcribing (using Parakeet V3) a 1 hr stereo 44.1 kHz mp3 vs the same audio in 16 kHz mono wav format. The result was about 21.3% slower with the mp3 vs the wav, i.e. that's the overhead of decoding + resampling. Currently the decoding + resampling is done up front. If it was done in a pipelined fashion with the inference, that overhead can be eliminated. This is what I did in a recent app I made: https://apps.apple.com/us/app/drea-podcast-ad-blocker/id6759070798 https://apps.apple.com/us/app/drea-podcast-ad-blocker/id6759... It uses FluidAudio as well, but I forked it and replaced the audio decoding code to (a) use mpg123 instead of the native Apple API and (b) do audio decoding and inference in a pipelined fashion. These two changes effectively eliminated the overhead. mpg123 is quite a bit faster than the native Apple API at mp3 decoding (has some very optimized arm64 assembly routines), and the pipelining ensures that the inference is never starved by the mp3 decoding. Contributing this pipelined setup to FluidAudio would be good.
      - satvikpendem 2mo ago
        That app was exactly what I was looking for, something like SponsorBlock but for podcasts but I suppose using AI for finding the ads works too. Any chance it'll release on Android?
        hamza_q_ 2mo ago
        Yep it's something I wanted for a while too; there were existing apps that did this, but had two issues: they were paid, and the UI was subpar. So for mine, I made sure it's fully free and that the UI is on par with Apple Podcasts, Spotify, etc. Making the ad-finding cheap enough such that I could make it free turned out to be harder than expected. The main issue you run into is dynamic, location-targeted ads. So I came up with a novel technique that uses Shazam-style audio fingerprints for accurate matching, instead of their normal use case, which is identification. This technique is what allows the ad finding to be very cheap, allowing me to make it free. The SponsorBlock model would actually not work for podcasts, due to dynamic ads. I.e. the location and content of the ads in episodes these days varies by download location. You need the media to be static, like YouTube, for SponsorBlock model to work. Therefore, using an LLM to find the ads + the fingerprints matching in combination is an efficient technique. Android has def been the most requested thing thus far haha. It'll be a decent undertaking due to me having written the app fully in Swift, i.e. it'll be a complete rewrite. I'll also need to replace FluidAudio with some good, fast Android equivalent. The goal of making this app was to create something impressive so that I could get a job. Haven't gotten a job yet, but if and when I do, then I'll have time & resources to think about doing an Android version. Currently a bit stressed and occupied from the job search lol.