10 ms·

OpenAI charges by the minute, so speed up your audio

740 points by georgemandis 1y ago

georgemandis 1y ago
I was trying to summarize a 40-minute talk with OpenAI’s transcription API, but it was too long. So I sped it up with ffmpeg to fit within the 25-minute cap. It worked quite well (Up to 3x speeds) and was cheaper and faster, so I wrote about it. Felt like a fun trick worth sharing. There’s a full script and cost breakdown.
- bravesoul2 1y ago
  You could have kept quiet and started a cheaper than openai transcription business :)
  - behnamoh 1y ago
    Sure, but now the world is a better place because he shared something useful!
  - 4b11b4 1y ago
    Pre-processing of the audio still a valid biz, multiple types of pre-processing might be valid
  - hn8726 1y ago
    Or openai will do it themselves for transcription tasks
  - ilyakaminsky 1y ago
    I've already done that [1]. A fraction of the price, 24-hour limit per file, and speedup tricks like the OP's are welcome. :) [1] https://speechischeap.com https://speechischeap.com
    - bravesoul2 1y ago
      Nice. Don't expect you to spill the beans but is it doing OK (some customers?) Just wondering if I cam build a retirement out of APIs :)
      - ilyakaminsky 1y ago
        It's sustainable, but not enough to retire on at this point. > Just wondering if I cam build a retirement out of APIs :) I think it's possible, but you need to find a way to add value beyond the commodity itself (e.g., audio classification and speaker diarization in my case).
    - satvikpendem 1y ago
      Can it do real-time transcription with diarization? I'm looking for that for a product feature I'm working on. Currently I've seen Speechmatics do this well, haven't heard of others.
      - ilyakaminsky 1y ago
        Not yet. The gains in efficiency come from optimizing the speedup factor. Real-time audio cannot be processed any faster than 1× by definition.
ada1981 1y ago
We discovered this last month. There is also prob a way to send a smaller sampler of audio at diff speeds and compare them to get a speed optimization with no quality loss unique for each clip.
- moralestapia 1y ago
  >We discovered this last month. Nice. Any blog post, twitter comment or anything pointing to that?
  - ada1981 1y ago
    We didn’t think to publish it; it actually seemed so obvious I assumed it was a widely known thing. We were developing an AI that processes someone’s Instagram profile (reels, comments, etc.) and provided insights and realized we could 2x the audio to cut time and costs.
- appleaday1 1y ago
  source?
  - ada1981 1y ago
    Oh I wasn’t trying to take credit for it, we just discovered we could do this last month and assumed it was widely known and implemented it. I could find a screen shot of our internal texts I suppose, but we didn’t publish anything on it.
brendanfinan 1y ago
would this also work for my video consisting of 10,000 PDFs? https://news.ycombinator.com/item?id=44125598 https://news.ycombinator.com/item?id=44125598
- raincole 1y ago
  Geez, that repo[0] has 8k stars on Github? Are people just staring it for meme value or something? Is this a scam? [0]: https://github.com/Olow304/memvid https://github.com/Olow304/memvid
- jasonjmcghee 1y ago
  I can't tell if this is a meme or not. And if someone had this idea and pitched it to Claude (the model this project was vibe coded with) it would be like "what a great idea!"
mcc1ane 1y ago
Longer*
- canyp 1y ago
  Came here just for this.
simonw 1y ago
There was a similar trick which worked with Gemini versions prior to Gemini 2.0: they charged a flat rate of 258 tokens for an image, and it turns out you could fit more than 258 tokens of text in an image of text and use that for a discount!
- Graziano_M 1y ago
  Well a picture is worth a thousand tokens.
heeton 1y ago
A point on skimming vs taking the time to read something properly. I read a transcript + summary of that exact talk. I thought it was fine, but uninteresting, I moved on. Later I saw it had been put on youtube and I was on the train, so I watched the whole thing at normal speed. I had a huge number of different ideas, thoughts and decisions, sparked by watching the whole thing. This happens to me in other areas too. Watching a conference talk in person is far more useful to me than watching it online with other distractions. Watching it online is more useful again than reading a summary. Going for a walk to think about something deeply beats a 10 minute session to "solve" the problem and forget it. Slower is usually better for thinking.
- pluc 1y ago
  Seriously this is bonkers to me. I, like many hackers, hated school because they just threw one-size-fits-all knowledge at you and here we are, paying for the privilege to have that in every facet of our lives. Reading is a pleasure. Watching a lecture or a talk and feeling the pieces fall into place is great. Having your brain work out the meaning of things is surely something that defines us as a species. We're willingly heading for such stupidity, I don't get it. I don't get how we can all be so blind at what this is going to create.
  - itake 1y ago
    > I don't get how we can all be so blind at what this is going to create. There is too much information. people are trying to optimize breadth over depth, but obviously there are costs to this.
  - hooverd 1y ago
    If you're not listening to summaries of different audiobooks at 2x speed in each ear you're not contentmaxing.
    - lovestory 1y ago
      Or just use notebookLM to convert your books into an hour long podcasts /s
b0a04gl 1y ago
[dead]
timerol 1y ago
> Is It Accurate? > I don’t know—I didn’t watch it, lol. That was the whole point. And if that answer makes you uncomfortable, buckle-up for this future we're hurtling toward. Boy, howdy. This is a great bit of work, and the author accurately summarizes my discomfort
- BHSPitMonkey 1y ago
  As if human-generated transcriptions of audio ever came with guarantees of accuracy? This kind of transformation has always come with flaws, and I think that will continue to be expected implicitly. Far more worrying is the public's trust in _interpretations_ and claims of _fact_ produced by gen AI services, or at least the popular idea that "AI" is more trustworthy/unbiased than humans, journalists, experts, etc.
  - angst 1y ago
    at least human-generated transcriptions have entities that we can hold responsible for...
    - _kb 1y ago
      That still holds true for gen-AI. Organisations that provide transcription services can’t offload responsibility to a language model any more than they can to steno keyboard manufacturers. If you are the one feeding content to a model then you are that responsible entity.
- raincole 1y ago
  A lot of people read newspaper. Newspaper is essentially just an inaccurate summary of what really happened. So I don't find this realization that uncomfortable.
  - dmix 1y ago
    That's why I find the idea of training breaking news on Reddit or Twitter funny, wild exaggerations and targeted spin is the sort of stuff that does best on those sites and generates the most comments, 50% of the output would be lies.
jasonjmcghee 1y ago
Heads up, the token cost breakdown tables look white on white to me. I'm in dark mode on iOS using Brave.
- georgemandis 1y ago
  Should be fixed now. Thank you!
w-m 1y ago
With transcribing a talk by Andrej, you already picked the most challenging case possible, speed-wise. His natural talking speed is already >=1.5x that of a normal human. One of the people you absolutely have to set your YouTube speed back down to 1x when listening to follow what's going on. In the idea of making more of an OpenAI minute, don't send it any silence. E.g. ffmpeg -i video-audio.m4a \ -af "silenceremove=start_periods=1:start_duration=0:start_threshold=-50dB:\ stop_periods=-1:stop_duration=0.02:stop_threshold=-50dB,\ apad=pad_dur=0.02" \ -c:a aac -b:a 128k output_minpause.m4a -y will cut the talk down from 39m31s to 31m34s, by replacing any silence (with a -50dB threshold) longer than 20ms by a 20ms pause. And to keep with the spirit of your post, I measured only that the input file got shorter, I didn't look at all at the quality of the transcription by feeding it the shorter version.
- georgemandis 1y ago
  Oooh fun! I had a feeling there was more ffmpeg wizardry I could be leaning into here. I'll have to try this later—thanks for the idea!
  - w-m 1y ago
    In the meantime I realized that the apad part is nonsensical - it pads the end of the stream, not at each silence-removed cut. I wanted to get angry at o3 for proposing this, but then I had a look at the silenceremove= documentation myself: https://ffmpeg.org/ffmpeg-filters.html#silenceremove https://ffmpeg.org/ffmpeg-filters.html#silenceremove Good god. You couldn't make that any more convoluted and hard-to-grasp if you wanted to. You gotta love ffmpeg! I now think this might be a good solution: ffmpeg -i video-audio.m4a \ -af "silenceremove=start_periods=1:stop_periods=-1:stop_duration=0.15:stop_threshold=-40dB:detection=rms" \ -c:a aac -b:a 128k output.m4a -y
    - snickerdoodle12 1y ago
      I love ffmpeg but the documentation is often close to incomprehensible.
appleaday1 1y ago
I use the youtube trick, will share it here, but upload to youtube and use their built in transcription service to translate to text for you, and than use gemini pro 2.5 to rebuild the transcript. ffmpeg \ -f lavfi \ -i color=c=black:s=1920x1080:r=5 \ -i file_you_want_transcripted.wav \ -c:v libx264 \ -preset medium \ -tune stillimage \ -crf 28 \ -c:a aac \ -b:a 192k \ -pix_fmt yuv420p \ -shortest \ file_you_upload_to_youtube_for_free_transcripts.mp4 This works VERY well for my needs.
KTibow 1y ago
This is really interesting, although the cheapest route is still to use an alternative audio-compatible LLM (Gemini 2.0 Flash Lite, Phi 4 Multimodal) or an alternative host for Whisper (Deepinfra, Fal).
fallinditch 1y ago
When extracting transcripts from YouTube videos, can anyone give advice on the best (cost effective, quick, accurate) way to do this? I'm confused because I read in various places that the YouTube API doesn't provide access to transcripts ... so how do all these YouTube transcript extractor services do it? I want to build my own YouTube summarizer app. Any advice and info on this topic greatly appreciated!
- vjerancrnjak 1y ago
  If YouTube placed autogenerated captions you can download them free of charge with yt-dlp.
- rob 1y ago
  There's a tool that uses YouTube's unofficial APIs to get them if they're available: https://github.com/jdepoix/youtube-transcript-api https://github.com/jdepoix/youtube-transcript-api For our internal tool that transcribes local city council meetings on YouTube (often 1-3 hours long), we found that these automatic ones were never available though. (Our tool usually 'processes' the videos within ~5-30 mins of being uploaded, so that's also why none are probably available 'officially' yet.) So we use yt-dlp to download the highest quality audio and then process them with whisper via Groq, which is way cheaper (~$0.02-0.04/hr with Groq compared to $0.36/hr via OpenAI's API.) Sometimes groq errors out so there's built-in support for Replicate and Deepgram as well. We run yt-dlp on our remote Linode server and I have a Python script I created that will automatically login to YouTube with a "clean" account and extract the proper cookies.txt file, and we also generate a 'po token' using another tool: https://github.com/iv-org/youtube-trusted-session-generator https://github.com/iv-org/youtube-trusted-session-generator Both cookies.txt and the "po token" get passed to yt-dlp when running on the Linode server and I haven't had to re-generate anything in over a month. Runs smoothly every day. (Note that I don't use cookies/po_token when running locally at home, it usually works fine there.)
  - fallinditch 1y ago
    Very useful, thanks. So does this mean that every month or so you have to create a new 'clean' YouTube account and use that to create new po_token/cookies? It's frustrating to have to jump through all these hoops just to extract transcripts when the YouTube Data API already gives reasonable limits to free API calls ... would be nice if they allowed transcripts too. Do you think the various YouTube transcript extractor services all follow a similar method as yours?
topaz0 1y ago
I have a way that is (all but) free -- just watch the video if you care about it, or decide not to if you don't, and move on with your life.
Tepix 1y ago
Why would you give up your privacy by sending what interests you to OpenAI when whisper doesn't need that much computer in the first place? With faster-whisper (int8, batch=8) you can transcripe 13 minutes of audio in 51 seconds on CPU.
- anigbrowl 1y ago
  I came here to ask the same question. This is a well-solved problem, red queen racing it seems utterly pointless, a symptom of reflexive adversarialism.
  - poly2it 1y ago
    > symptom of reflexive adversarialism Is there a definition for this expression? I don't catch you. > ... using corporate technology for the solved problem is a symptom of self-directed skepticism by the user against the corporate institutions ... Eh?
    - anigbrowl 1y ago
      Reflexively adopting an adversarial approach (trying to outwit Open AI by using sped-up audio) instead of being content with the perfectly reliable, free, and quite efficient solution that already exists. >> ... using corporate technology for the solved problem is a symptom of self-directed skepticism by the user against the corporate institutions ... > Eh? I don't know who wrote that or why you pasted in response to me.
- ProllyInfamous 1y ago
  I am a blue collar electrician. Not a coder (but definitely geeky). Whisper works quite well on Apple Silicon with simple drag/drop install (i.e. no terminal commands). Program is free; you can get an M4 mini for ~$550; don't see how an online platform can even compete with this, except for one-off customers (i.e. not great repeat customers). We used it to transcribe ddaayyss of audio microcassettes which my mother had made during her lifetime. Whisper.app even transcribed a few hours that are difficult to comprehend as a human listener. It is VERY fast. I've used the text to search for timestamps worth listening to, skipping most dead-space (e.g. she made most while driving, in a stream of not-always-focused consciousness).
pimlottc 1y ago
Appreciated the concise summary + code snippet upfront, followed by more detail and background for those interested. More articles should be written this way!
rob 1y ago
For anybody trying to do this in bulk, instead of using OpenAI's whisper via their API, you can also use Groq [0] which is much cheaper: [0] https://groq.com/pricing/ https://groq.com/pricing/ Groq is ~$0.02/hr with distil-large-v3, or ~$0.04/hr with whisper-large-v3-turbo. I believe OpenAI comes out to like ~$0.36/hr. We do this internally with our tool that automatically transcribes local government council meetings right when they get uploaded to YouTube. It uses Groq by default, but I also added support for Replicate and Deepgram as backups because sometimes Groq errors out.
- georgemandis 1y ago
  Interesting! At $0.02 to $0.04 an hour I don't suspect you've been hunting for optimizations, but I wonder if this "speed up the audio" trick would save you even more. > We do this internally with our tool that automatically transcribes local government council meetings right when they get uploaded to YouTube Doesn't YouTube do this for you automatically these days within a day or so?
  - rob 1y ago
    > Doesn't YouTube do this for you automatically these days within a day or so? Oh yeah, we do a check first and use youtube-transcript-api if there's an automatic one available: https://github.com/jdepoix/youtube-transcript-api https://github.com/jdepoix/youtube-transcript-api The tool usually detects them within like ~5 mins of being uploaded though, so usually none are available yet. Then it'll send the summaries to our internal Slack channel for our editors, in case there's anything interesting to 'follow up on' from the meeting. Probably would be a good idea to add a delay to it and wait for the automatic ones though :)
  - jerjerjer 1y ago
    > I wonder if this "speed up the audio" trick would save you even more. At this point you'll need to at least check how much running ffmpeg costs. Probably less than $0.01 per hour of audio (approximate savings) but still.
  - ks2048 1y ago
stogot 1y ago
Love this idea but the accuracy section is lacking. Couldnt you do a simple diff of the outputs and see how many differences there are? .5% or 5%?
- georgemandis 1y ago
  Yeah, I'd like to do a more formal analysis of the outputs if I can carve out the time. I don't think a simple diff is the way to go, at least for what I'm interested in. What I care about more is the overall accuracy of the summary—not the word-for-word transcription. The test I want to setup is using LLMs to evaluate the summarized output and see if the primary themes/topics persist. That's more interesting and useful to me for this exercise.
tmaly 1y ago
The whisper model weights are free. You could save even more by just using them locally.
- pzo 1y ago
  but this is still great trick if you want to reduce latency or inference speed even with local models e.g. in realtime chatbot
55555 1y ago
This seems like a good place for me to complain about the fact that the automatically generated subtitle files Youtube creates are horribly malformed. Every sentence is repeated twice. In many subtitle files, the subtitle timestamp ranges overlap one another while also repeating every sentence twice in two different ranges. It's absolutely bizarre and has been like this for years or possibly forever. Here's an example - I apologize that it's not in English. I don't know if this issue affects English. https://pastebin.com/raw/LTBps80F https://pastebin.com/raw/LTBps80F
- xenator 1y ago
  Seems like Thai. Thai translation and recognition is like 10 years ago comparing to other languages I'm dealing with in my everyday life. Good news tho is the same level was for Russian years ago, and now it is near perfect.
  - 55555 1y ago
    Well the weird thing is honestly their speech to text recognizes 97% of words correctly. The subtitle content is pretty perfect. It’s just the formatting that’s awful.
amelius 1y ago
Solution: charge by number of characters generated.
dataviz1000 1y ago
I built a Chrome extension with one feature that transcribes audio to text in the browser using huggingface/transformers.js running the OpenAI Whisper model with WebGPU. It works perfect! Here is a list of examples of all the things you can do in the browser with webgpu for free. [0] The last thing in the world I want to do is listen or watch presidential social media posts, but, on the other hand, sometimes enormously stupid things are said which move the SP500 up or down $60 in a session. So this feature queries for new posts every minute, does ORC image to text and transcribe video audio to text locally, sends the post with text for analysis, all in the background inside a Chrome extension before notify me of anything economically significant. [0] https://github.com/huggingface/transformers.js/tree/main/examples https://github.com/huggingface/transformers.js/tree/main/exa... [1] https://github.com/adam-s/doomberg-terminal https://github.com/adam-s/doomberg-terminal
- kgc 1y ago
  Impressive
karpathy 1y ago
Omg long post. TLDR from an LLM for anyone interested Speed your audio up 2–3× with ffmpeg before sending it to OpenAI’s gpt-4o-transcribe: the shorter file uses fewer input-tokens, cuts costs by roughly a third, and processes faster with little quality loss (4× is too fast). A sample yt-dlp → ffmpeg → curl script shows the workflow. ;)
- lordspace 1y ago
  that's a really good summary :)
- bravesoul2 1y ago
  This is the sort of content I want to see in Tweets and LinkedIn posts. I have been thinking for a while how do you make good use of the short space in those places. LLM did well here.
- georgemandis 1y ago
  Hahaha. Okay, okay... I will watch it now ;) (Thanks for your good sense of humor)
  - karpathy 1y ago
    I like that your post deliberately gets to the point first and then (optionally) expands later, I think it's a good and generally underutilized format. I often advise people to structure their emails in the same way, e.g. first just cutting to the chase with the specific ask, then giving more context optionally below. It's not my intention to bloat information or delivery but I also don't super know how to follow this format especially in this kind of talk. Because it's not so much about relaying specific information (like your final script here), but more as a collection of prompts back to the audience as things to think about. My companion tweet to this video on X had a brief TLDR/Summary included where I tried, but I didn't super think it was very reflective of the talk, it was more about topics covered. Anyway, I am overall a big fan of doing more compute at the "creation time" to compress other people's time during "consumption time" and I think it's the respectful and kind thing to do.
xg15 1y ago
That's really cool! Also, isn't this effectively the same as supplying audio with a sampling rate of 8kHz instead of the 16kHz that the model is supposed to work with?
anshumankmr 1y ago
Someone should try transcribing Eminem's Rap god with this trick.
alok-g 1y ago
>> by jumping straight to the point ... Love this! I wish more authors follow this approach. So many articles keep going all over the place before 'the point' appears. If trying, perhaps some 50% of the authors may realize that they don't _have_ a point.
pknerd 1y ago
I guess it'd work even if you make it 2.5 or evebn 3x.
donkey_brains 1y ago
Hmm…doesn’t this technique effectively make the minute longer, not shorter? Because you can pack more speech into a minute of recording? Seems like making a minute shorter would be counterproductive.
- StochasticLi 1y ago
  No. You're paying for a minute of audio, which will be more packed with speech, not for how long it's being computed.
impossiblefork 1y ago
Make the minutes longer, you mean.
pbbakkum 1y ago
This is great, thank you for sharing. I work on these APIs at OpenAI, it's a surprise to me that it still works reasonably well at 2/3x speed, but on the other hand for phone channels we get 8khz audio that is upsampled to 24khz for the model and it still works well. Note there's probably a measurable decrease in transcription accuracy that worsens as you deviate from 1x speed. Also we really need to support bigger/longer file uploads :)
- nerder92 1y ago
  Quick Feedback: Would it be cool to research this internally and maybe find a sweet spot in speed multiplier where the loss is minimal. This pre-processing is quite cheap and could bring down the API price eventually.
- georgemandis 1y ago
  I kind of want to take a more proper poke at this but focus more one summarization accuracy over word-for-word accuracy, though I see the value in both. I'm actually curious, if I run transcriptions back-to-back-to-back on the exact same audio, how much variance should I expect? Maybe I'll try three approaches: - A straight diff comparison (I know a lot of people are calling for this, but I really think this is less useful than it sounds) - A "variance within the modal" test running it multiple times against the same audio, tracking how much it varies between runs - An LLM analysis assessing if the primary points from a talk were captured and summarized at 1x, 2x, 3x, 4x runs (I think this is far more useful and interesting)
celltalk 1y ago
With this logic, you should also be able to trim the parts that doesn’t have words. Just add a cut-off for db, and trim the video before transcription. Possibly another 10-20% gain?
isubkhankulov 1y ago
Transcripts get much more valuable when one diarizes the audio beforehand to determine which speaker said what. I use this free tool to extract those and dump the transcripts into a LLM with basic prompts: https://contentflow.megalabs.co https://contentflow.megalabs.co
mt_ 1y ago
You can just dump the youtube link video in Google AI studio and ask it to transcribe the video with speaker labels and even ask it it to add useful visual clues, because the model is multimodal for video too.
- MaxDPS 1y ago
  Can I ask what you mean by “useful visual clues”?
  - mt_ 1y ago
    What is the speaker showcasing in its slides, what is it's body language and so on.
cprayingmantis 1y ago
I noticed something similar with images as inputs to Claude, you can scale down the images and still get good outputs. There is an accuracy drop off at a certain point but the token savings are worth doing a little tuning there.
- georgemandis 1y ago
  Definitely in the same spirit! Clearly the next thing we need to test is removing all the vowels from words, or something like that :)
meerab 1y ago
Interesting approach to transcript generation! I'm implementing a similar workflow for VideoToBe.com My Current Pipeline: Media Extraction - yt-dlp for reliable video/audio downloads Local Transcription - OpenAI Whisper running on my own hardware (no API costs) Storage & UI - Transcripts stored in S3 with a custom web interface for viewing Y Combinator playlist https://videotobe.com/play/playlist/ycombinator https://videotobe.com/play/playlist/ycombinator and Andrej's talk is https://videotobe.com/play/youtube/LCEmiRjPEtQ https://videotobe.com/play/youtube/LCEmiRjPEtQ After reading your blog post, I will be testing effect on speeding audio for locally-hosted Whisper models. Running Whisper locally eliminates the ongoing cost concerns since my infrastructure is already a sunk cost. Speeding audio could be an interesting performance enhancement to explore!
fuzztester 1y ago
Stop being slaves of extorters of any kind, and just leave. there is tons of this happening everywhere, and we need to fight this, and boycott it.
pottertheotter 1y ago
You can just ask Gemini to summarize it for you. It's free. I do it all the time with YouTube videos. Or you can just copy the transcript that YouTube provides below the video.
BrunoJo 1y ago
If you look for a cheaper transcription API you could als use https://Lemonfox.ai https://Lemonfox.ai. We've optimized the API for long audio files and are much faster and cheaper than OpenAI.
spapinwar 1y ago
[dead]
Raphell 1y ago
[flagged]
- weird-eye-issue 1y ago
  Can we ban this "person" for AI replies?
  - Raphell 1y ago
    I get it. But I'm just someone who likes to think things through and say them simply.
    - weird-eye-issue 1y ago
      You aren't saying anything simply. You are verbosely saying nothing.
conjecTech 1y ago
If you are hosting whisper yourself, you can do something slightly more elegant, but with the same effect. You can downsample/pool the context 2:1 (or potentially more) a few layers into the encoder. That allows you to do the equivalent of speeding up audio without worry about potential spectral losses. For whisper large v3, that gets you nearly double throughput in exchange for a relative ~4% WER increase.
- nomercy400 1y ago
  Do you have more details or examples on how to downsample the context in the encoder? I treat the encoder as an opaque block, so I have no idea where to start.
  - conjecTech 1y ago
    It's a very simple change in a vanilla python implementation. The encoder is a set of attention blocks, and the length of the attention can be changed without changing the calculation at all. Here(https://github.com/openai/whisper/blob/main/whisper/model.py#L201 https://github.com/openai/whisper/blob/main/whisper/model.py...) is the relevant code in the whisper repo. You'd just need to change the for loop to an enumerate and subsample the context along its length at the point you want. I believe it would be: for i, block in enumerate(self.blocks): x = block(x) if i==4: x = x[,,::2]
PeterStuer 1y ago
I wonder how much time and battery transcoding/uploading/downloading over coffeeshop wifi would realy save vs just running it locally through optimized Whisper.
- georgemandis 1y ago
  I had this same thought and won't pretend my fear was rational, haha. One thing that I thought was fairly clear in my write-up but feels a little lost in the comments: I didn't just try this with whisper. I tried it with their newer gpt-4o-transcription model, which seems considerably faster. There's no way to run that one locally.
KPennig86852 1y ago
But you know that you can run OpenAI's Whisper audio recognition model locally for free, right? It has very little GPU requirements, and the new "turbo" model works quite fast (there are also several Python libraries which make it significantly faster still).
dajonker 1y ago
Gemini 2.5 pro is, in my usage, quite superior for high quality transcriptions of phone calls, in Dutch in my case. As long as you upload the audio to GCS there you can easily process conversations of over an hour. It correctly identified and labeled speakers. The cheaper 2.5 flash made noticeably more mistakes, for example it didn't correctly output numbers while the Pro model did. As for OpenAI, their gpt-4o-transcribe model did worse than 2.5 flash, completely messing up names of places and/or people. Plus it doesn't label the conversation in turns, it just outputs a single continuous piece of text.
yashasolutions 1y ago
the question would be how to do that but also still get proper time code when using whisper to get the subtitles
ryanar 1y ago
In my experience, transcription software has no problem with transcribing sped up audio, or audio that is inaudible to humans or extremely loud (as long as not clipped), I wonder if LLM transcription works the same.
mushishi 1y ago
Do the APIs support simultaneous voice transcription in a way that different voices are tagged? (either in text or as metadata) If so: could you split the audiofile and process the latter half by pitch shifting, say an octave, and then merging them together to get shorter audiofile — then transcribe and join them back to a linear form, tagging removed. (You could insert some prerecorded voice to know at which point the second voice starts.). If pitch change is not enough, maybe manipulate it further by formants.
godot 1y ago
If you're already doing local ffmpeg stuff (i.e. pretty involved with code and scripting already) you're only a couple of steps more away from just downloading the openai-whisper models (or even the faster-whisper models which runs about two times faster). Since this looks like personal usage and not building production quality code, you can use AI (e.g. Cursor) to write a script to run the whisper model inference in seconds. Then there is no cost at all to run any length of audio. (since cost seems to be the primary factor of this article) On my m1 mac laptop it takes me about 30 seconds to run it on a 3-minute audio file. I'm guessing for a 40 minute talk it takes about 5-10 minutes to run.
- Tepix 1y ago
  Have you tried faster-whisper and whisper.cpp?
  - godot 1y ago
    Yeah, my mentioned times are with faster-whisper, but I have not tried whisper.cpp. I just use a python script to run the model.
ta8903 1y ago
This "hack" also works in real life, youtubers low to talk slowly to increase the video runtime so I watch everything other than songs at 2x speed (and that's only because their player doesn't let you go faster).
7speter 1y ago
So wait… is whisper transcription really all that slow locally on a M3 Macbook? It’s been a while since I used whispercpp, but I seem to remember it taking maybe 20 minutes on a comparatively slowpoke (and powerhungry) i5 12600k for maybe 40 minutes of audio; it might take less time on a faster m chip (maybe I’m imagining mobile apple silicon to be more performant than even desktop intel cpus), even less if there support built in for the built in gpu cores and other ai optimized silicon? Did I miss that the task was time sensitive?
raluk 1y ago
Our team is working with soniox.com They are the most acurate model that works real time.
Nevermark 1y ago
It's also rude to talk slow to them. Unless its Siri.
another_twist 1y ago
You'd need a WER comparison to check if it really is no drop in quality. With this trick, there might be trouble if the audio is noisy, and it may. ot always be obvious whether or not to speed up.
addaidirectory 1y ago
That's a clever idea. Their are alternatives to OpenAI for audio transcription. Check them out https://www.addaidirectory.com/categories/audio https://www.addaidirectory.com/categories/audio or scroll the home page https://www.addaidirectory.com https://www.addaidirectory.com for updates

740 points by georgemandis 1y ago

I was trying to summarize a 40-minute talk with OpenAI’s transcription API, but it was too long. So I sped it up with ffmpeg to fit within the 25-minute cap. It worked quite well (Up to 3x speeds) and was cheaper and faster, so I wrote about it. Felt like a fun trick worth sharing. There’s a full script and cost breakdown.

bravesoul2 1y ago
You could have kept quiet and started a cheaper than openai transcription business :)
- behnamoh 1y ago
  Sure, but now the world is a better place because he shared something useful!
- 4b11b4 1y ago
  Pre-processing of the audio still a valid biz, multiple types of pre-processing might be valid
- hn8726 1y ago
  Or openai will do it themselves for transcription tasks
- ilyakaminsky 1y ago
  I've already done that [1]. A fraction of the price, 24-hour limit per file, and speedup tricks like the OP's are welcome. :) [1] https://speechischeap.com https://speechischeap.com
  - bravesoul2 1y ago
    Nice. Don't expect you to spill the beans but is it doing OK (some customers?) Just wondering if I cam build a retirement out of APIs :)
    - ilyakaminsky 1y ago
      It's sustainable, but not enough to retire on at this point. > Just wondering if I cam build a retirement out of APIs :) I think it's possible, but you need to find a way to add value beyond the commodity itself (e.g., audio classification and speaker diarization in my case).
  - satvikpendem 1y ago
    Can it do real-time transcription with diarization? I'm looking for that for a product feature I'm working on. Currently I've seen Speechmatics do this well, haven't heard of others.
    - ilyakaminsky 1y ago
      Not yet. The gains in efficiency come from optimizing the speedup factor. Real-time audio cannot be processed any faster than 1× by definition.

ada1981 1y ago

We discovered this last month. There is also prob a way to send a smaller sampler of audio at diff speeds and compare them to get a speed optimization with no quality loss unique for each clip.

moralestapia 1y ago
>We discovered this last month. Nice. Any blog post, twitter comment or anything pointing to that?
- ada1981 1y ago
  We didn’t think to publish it; it actually seemed so obvious I assumed it was a widely known thing. We were developing an AI that processes someone’s Instagram profile (reels, comments, etc.) and provided insights and realized we could 2x the audio to cut time and costs.
appleaday1 1y ago
source?
- ada1981 1y ago
  Oh I wasn’t trying to take credit for it, we just discovered we could do this last month and assumed it was widely known and implemented it. I could find a screen shot of our internal texts I suppose, but we didn’t publish anything on it.

brendanfinan 1y ago

would this also work for my video consisting of 10,000 PDFs? https://news.ycombinator.com/item?id=44125598 https://news.ycombinator.com/item?id=44125598

raincole 1y ago
Geez, that repo[0] has 8k stars on Github? Are people just staring it for meme value or something? Is this a scam? [0]: https://github.com/Olow304/memvid https://github.com/Olow304/memvid
jasonjmcghee 1y ago
I can't tell if this is a meme or not. And if someone had this idea and pitched it to Claude (the model this project was vibe coded with) it would be like "what a great idea!"

mcc1ane 1y ago

Longer*

canyp 1y ago
Came here just for this.

simonw 1y ago

There was a similar trick which worked with Gemini versions prior to Gemini 2.0: they charged a flat rate of 258 tokens for an image, and it turns out you could fit more than 258 tokens of text in an image of text and use that for a discount!

Graziano_M 1y ago
Well a picture is worth a thousand tokens.

heeton 1y ago

A point on skimming vs taking the time to read something properly. I read a transcript + summary of that exact talk. I thought it was fine, but uninteresting, I moved on. Later I saw it had been put on youtube and I was on the train, so I watched the whole thing at normal speed. I had a huge number of different ideas, thoughts and decisions, sparked by watching the whole thing. This happens to me in other areas too. Watching a conference talk in person is far more useful to me than watching it online with other distractions. Watching it online is more useful again than reading a summary. Going for a walk to think about something deeply beats a 10 minute session to "solve" the problem and forget it. Slower is usually better for thinking.

pluc 1y ago
Seriously this is bonkers to me. I, like many hackers, hated school because they just threw one-size-fits-all knowledge at you and here we are, paying for the privilege to have that in every facet of our lives. Reading is a pleasure. Watching a lecture or a talk and feeling the pieces fall into place is great. Having your brain work out the meaning of things is surely something that defines us as a species. We're willingly heading for such stupidity, I don't get it. I don't get how we can all be so blind at what this is going to create.
- itake 1y ago
  > I don't get how we can all be so blind at what this is going to create. There is too much information. people are trying to optimize breadth over depth, but obviously there are costs to this.
- hooverd 1y ago
  If you're not listening to summaries of different audiobooks at 2x speed in each ear you're not contentmaxing.
  - lovestory 1y ago
    Or just use notebookLM to convert your books into an hour long podcasts /s

b0a04gl 1y ago

[dead]

timerol 1y ago

> Is It Accurate? > I don’t know—I didn’t watch it, lol. That was the whole point. And if that answer makes you uncomfortable, buckle-up for this future we're hurtling toward. Boy, howdy. This is a great bit of work, and the author accurately summarizes my discomfort

BHSPitMonkey 1y ago
As if human-generated transcriptions of audio ever came with guarantees of accuracy? This kind of transformation has always come with flaws, and I think that will continue to be expected implicitly. Far more worrying is the public's trust in _interpretations_ and claims of _fact_ produced by gen AI services, or at least the popular idea that "AI" is more trustworthy/unbiased than humans, journalists, experts, etc.
- angst 1y ago
  at least human-generated transcriptions have entities that we can hold responsible for...
  - _kb 1y ago
    That still holds true for gen-AI. Organisations that provide transcription services can’t offload responsibility to a language model any more than they can to steno keyboard manufacturers. If you are the one feeding content to a model then you are that responsible entity.
raincole 1y ago
A lot of people read newspaper. Newspaper is essentially just an inaccurate summary of what really happened. So I don't find this realization that uncomfortable.
- dmix 1y ago
  That's why I find the idea of training breaking news on Reddit or Twitter funny, wild exaggerations and targeted spin is the sort of stuff that does best on those sites and generates the most comments, 50% of the output would be lies.

jasonjmcghee 1y ago

Heads up, the token cost breakdown tables look white on white to me. I'm in dark mode on iOS using Brave.

georgemandis 1y ago
Should be fixed now. Thank you!

w-m 1y ago

With transcribing a talk by Andrej, you already picked the most challenging case possible, speed-wise. His natural talking speed is already >=1.5x that of a normal human. One of the people you absolutely have to set your YouTube speed back down to 1x when listening to follow what's going on. In the idea of making more of an OpenAI minute, don't send it any silence. E.g. ffmpeg -i video-audio.m4a \ -af "silenceremove=start_periods=1:start_duration=0:start_threshold=-50dB:\ stop_periods=-1:stop_duration=0.02:stop_threshold=-50dB,\ apad=pad_dur=0.02" \ -c:a aac -b:a 128k output_minpause.m4a -y will cut the talk down from 39m31s to 31m34s, by replacing any silence (with a -50dB threshold) longer than 20ms by a 20ms pause. And to keep with the spirit of your post, I measured only that the input file got shorter, I didn't look at all at the quality of the transcription by feeding it the shorter version.

georgemandis 1y ago
Oooh fun! I had a feeling there was more ffmpeg wizardry I could be leaning into here. I'll have to try this later—thanks for the idea!
- w-m 1y ago
  In the meantime I realized that the apad part is nonsensical - it pads the end of the stream, not at each silence-removed cut. I wanted to get angry at o3 for proposing this, but then I had a look at the silenceremove= documentation myself: https://ffmpeg.org/ffmpeg-filters.html#silenceremove https://ffmpeg.org/ffmpeg-filters.html#silenceremove Good god. You couldn't make that any more convoluted and hard-to-grasp if you wanted to. You gotta love ffmpeg! I now think this might be a good solution: ffmpeg -i video-audio.m4a \ -af "silenceremove=start_periods=1:stop_periods=-1:stop_duration=0.15:stop_threshold=-40dB:detection=rms" \ -c:a aac -b:a 128k output.m4a -y
  - snickerdoodle12 1y ago
    I love ffmpeg but the documentation is often close to incomprehensible.

appleaday1 1y ago

I use the youtube trick, will share it here, but upload to youtube and use their built in transcription service to translate to text for you, and than use gemini pro 2.5 to rebuild the transcript. ffmpeg \ -f lavfi \ -i color=c=black:s=1920x1080:r=5 \ -i file_you_want_transcripted.wav \ -c:v libx264 \ -preset medium \ -tune stillimage \ -crf 28 \ -c:a aac \ -b:a 192k \ -pix_fmt yuv420p \ -shortest \ file_you_upload_to_youtube_for_free_transcripts.mp4 This works VERY well for my needs.

KTibow 1y ago

This is really interesting, although the cheapest route is still to use an alternative audio-compatible LLM (Gemini 2.0 Flash Lite, Phi 4 Multimodal) or an alternative host for Whisper (Deepinfra, Fal).

fallinditch 1y ago

When extracting transcripts from YouTube videos, can anyone give advice on the best (cost effective, quick, accurate) way to do this? I'm confused because I read in various places that the YouTube API doesn't provide access to transcripts ... so how do all these YouTube transcript extractor services do it? I want to build my own YouTube summarizer app. Any advice and info on this topic greatly appreciated!

vjerancrnjak 1y ago
If YouTube placed autogenerated captions you can download them free of charge with yt-dlp.
rob 1y ago
There's a tool that uses YouTube's unofficial APIs to get them if they're available: https://github.com/jdepoix/youtube-transcript-api https://github.com/jdepoix/youtube-transcript-api For our internal tool that transcribes local city council meetings on YouTube (often 1-3 hours long), we found that these automatic ones were never available though. (Our tool usually 'processes' the videos within ~5-30 mins of being uploaded, so that's also why none are probably available 'officially' yet.) So we use yt-dlp to download the highest quality audio and then process them with whisper via Groq, which is way cheaper (~$0.02-0.04/hr with Groq compared to $0.36/hr via OpenAI's API.) Sometimes groq errors out so there's built-in support for Replicate and Deepgram as well. We run yt-dlp on our remote Linode server and I have a Python script I created that will automatically login to YouTube with a "clean" account and extract the proper cookies.txt file, and we also generate a 'po token' using another tool: https://github.com/iv-org/youtube-trusted-session-generator https://github.com/iv-org/youtube-trusted-session-generator Both cookies.txt and the "po token" get passed to yt-dlp when running on the Linode server and I haven't had to re-generate anything in over a month. Runs smoothly every day. (Note that I don't use cookies/po_token when running locally at home, it usually works fine there.)
- fallinditch 1y ago
  Very useful, thanks. So does this mean that every month or so you have to create a new 'clean' YouTube account and use that to create new po_token/cookies? It's frustrating to have to jump through all these hoops just to extract transcripts when the YouTube Data API already gives reasonable limits to free API calls ... would be nice if they allowed transcripts too. Do you think the various YouTube transcript extractor services all follow a similar method as yours?

topaz0 1y ago

I have a way that is (all but) free -- just watch the video if you care about it, or decide not to if you don't, and move on with your life.

Tepix 1y ago

Why would you give up your privacy by sending what interests you to OpenAI when whisper doesn't need that much computer in the first place? With faster-whisper (int8, batch=8) you can transcripe 13 minutes of audio in 51 seconds on CPU.

anigbrowl 1y ago
I came here to ask the same question. This is a well-solved problem, red queen racing it seems utterly pointless, a symptom of reflexive adversarialism.
- poly2it 1y ago
  > symptom of reflexive adversarialism Is there a definition for this expression? I don't catch you. > ... using corporate technology for the solved problem is a symptom of self-directed skepticism by the user against the corporate institutions ... Eh?
  - anigbrowl 1y ago
    Reflexively adopting an adversarial approach (trying to outwit Open AI by using sped-up audio) instead of being content with the perfectly reliable, free, and quite efficient solution that already exists. >> ... using corporate technology for the solved problem is a symptom of self-directed skepticism by the user against the corporate institutions ... > Eh? I don't know who wrote that or why you pasted in response to me.
ProllyInfamous 1y ago
I am a blue collar electrician. Not a coder (but definitely geeky). Whisper works quite well on Apple Silicon with simple drag/drop install (i.e. no terminal commands). Program is free; you can get an M4 mini for ~$550; don't see how an online platform can even compete with this, except for one-off customers (i.e. not great repeat customers). We used it to transcribe ddaayyss of audio microcassettes which my mother had made during her lifetime. Whisper.app even transcribed a few hours that are difficult to comprehend as a human listener. It is VERY fast. I've used the text to search for timestamps worth listening to, skipping most dead-space (e.g. she made most while driving, in a stream of not-always-focused consciousness).

pimlottc 1y ago

Appreciated the concise summary + code snippet upfront, followed by more detail and background for those interested. More articles should be written this way!

rob 1y ago

For anybody trying to do this in bulk, instead of using OpenAI's whisper via their API, you can also use Groq [0] which is much cheaper: [0] https://groq.com/pricing/ https://groq.com/pricing/ Groq is ~$0.02/hr with distil-large-v3, or ~$0.04/hr with whisper-large-v3-turbo. I believe OpenAI comes out to like ~$0.36/hr. We do this internally with our tool that automatically transcribes local government council meetings right when they get uploaded to YouTube. It uses Groq by default, but I also added support for Replicate and Deepgram as backups because sometimes Groq errors out.

georgemandis 1y ago
Interesting! At $0.02 to $0.04 an hour I don't suspect you've been hunting for optimizations, but I wonder if this "speed up the audio" trick would save you even more. > We do this internally with our tool that automatically transcribes local government council meetings right when they get uploaded to YouTube Doesn't YouTube do this for you automatically these days within a day or so?
- rob 1y ago
  > Doesn't YouTube do this for you automatically these days within a day or so? Oh yeah, we do a check first and use youtube-transcript-api if there's an automatic one available: https://github.com/jdepoix/youtube-transcript-api https://github.com/jdepoix/youtube-transcript-api The tool usually detects them within like ~5 mins of being uploaded though, so usually none are available yet. Then it'll send the summaries to our internal Slack channel for our editors, in case there's anything interesting to 'follow up on' from the meeting. Probably would be a good idea to add a delay to it and wait for the automatic ones though :)
- jerjerjer 1y ago
  > I wonder if this "speed up the audio" trick would save you even more. At this point you'll need to at least check how much running ffmpeg costs. Probably less than $0.01 per hour of audio (approximate savings) but still.
- ks2048 1y ago

stogot 1y ago

Love this idea but the accuracy section is lacking. Couldnt you do a simple diff of the outputs and see how many differences there are? .5% or 5%?

georgemandis 1y ago
Yeah, I'd like to do a more formal analysis of the outputs if I can carve out the time. I don't think a simple diff is the way to go, at least for what I'm interested in. What I care about more is the overall accuracy of the summary—not the word-for-word transcription. The test I want to setup is using LLMs to evaluate the summarized output and see if the primary themes/topics persist. That's more interesting and useful to me for this exercise.

tmaly 1y ago

The whisper model weights are free. You could save even more by just using them locally.

pzo 1y ago
but this is still great trick if you want to reduce latency or inference speed even with local models e.g. in realtime chatbot

55555 1y ago

This seems like a good place for me to complain about the fact that the automatically generated subtitle files Youtube creates are horribly malformed. Every sentence is repeated twice. In many subtitle files, the subtitle timestamp ranges overlap one another while also repeating every sentence twice in two different ranges. It's absolutely bizarre and has been like this for years or possibly forever. Here's an example - I apologize that it's not in English. I don't know if this issue affects English. https://pastebin.com/raw/LTBps80F https://pastebin.com/raw/LTBps80F

xenator 1y ago
Seems like Thai. Thai translation and recognition is like 10 years ago comparing to other languages I'm dealing with in my everyday life. Good news tho is the same level was for Russian years ago, and now it is near perfect.
- 55555 1y ago
  Well the weird thing is honestly their speech to text recognizes 97% of words correctly. The subtitle content is pretty perfect. It’s just the formatting that’s awful.

amelius 1y ago

Solution: charge by number of characters generated.

dataviz1000 1y ago

I built a Chrome extension with one feature that transcribes audio to text in the browser using huggingface/transformers.js running the OpenAI Whisper model with WebGPU. It works perfect! Here is a list of examples of all the things you can do in the browser with webgpu for free. [0] The last thing in the world I want to do is listen or watch presidential social media posts, but, on the other hand, sometimes enormously stupid things are said which move the SP500 up or down $60 in a session. So this feature queries for new posts every minute, does ORC image to text and transcribe video audio to text locally, sends the post with text for analysis, all in the background inside a Chrome extension before notify me of anything economically significant. [0] https://github.com/huggingface/transformers.js/tree/main/examples https://github.com/huggingface/transformers.js/tree/main/exa... [1] https://github.com/adam-s/doomberg-terminal https://github.com/adam-s/doomberg-terminal

kgc 1y ago
Impressive

karpathy 1y ago

Omg long post. TLDR from an LLM for anyone interested Speed your audio up 2–3× with ffmpeg before sending it to OpenAI’s gpt-4o-transcribe: the shorter file uses fewer input-tokens, cuts costs by roughly a third, and processes faster with little quality loss (4× is too fast). A sample yt-dlp → ffmpeg → curl script shows the workflow. ;)

lordspace 1y ago
that's a really good summary :)
bravesoul2 1y ago
This is the sort of content I want to see in Tweets and LinkedIn posts. I have been thinking for a while how do you make good use of the short space in those places. LLM did well here.
georgemandis 1y ago
Hahaha. Okay, okay... I will watch it now ;) (Thanks for your good sense of humor)
- karpathy 1y ago
  I like that your post deliberately gets to the point first and then (optionally) expands later, I think it's a good and generally underutilized format. I often advise people to structure their emails in the same way, e.g. first just cutting to the chase with the specific ask, then giving more context optionally below. It's not my intention to bloat information or delivery but I also don't super know how to follow this format especially in this kind of talk. Because it's not so much about relaying specific information (like your final script here), but more as a collection of prompts back to the audience as things to think about. My companion tweet to this video on X had a brief TLDR/Summary included where I tried, but I didn't super think it was very reflective of the talk, it was more about topics covered. Anyway, I am overall a big fan of doing more compute at the "creation time" to compress other people's time during "consumption time" and I think it's the respectful and kind thing to do.

xg15 1y ago

That's really cool! Also, isn't this effectively the same as supplying audio with a sampling rate of 8kHz instead of the 16kHz that the model is supposed to work with?

anshumankmr 1y ago

Someone should try transcribing Eminem's Rap god with this trick.

alok-g 1y ago

>> by jumping straight to the point ... Love this! I wish more authors follow this approach. So many articles keep going all over the place before 'the point' appears. If trying, perhaps some 50% of the authors may realize that they don't _have_ a point.

pknerd 1y ago

I guess it'd work even if you make it 2.5 or evebn 3x.

donkey_brains 1y ago

Hmm…doesn’t this technique effectively make the minute longer, not shorter? Because you can pack more speech into a minute of recording? Seems like making a minute shorter would be counterproductive.

StochasticLi 1y ago
No. You're paying for a minute of audio, which will be more packed with speech, not for how long it's being computed.

impossiblefork 1y ago

Make the minutes longer, you mean.

pbbakkum 1y ago

This is great, thank you for sharing. I work on these APIs at OpenAI, it's a surprise to me that it still works reasonably well at 2/3x speed, but on the other hand for phone channels we get 8khz audio that is upsampled to 24khz for the model and it still works well. Note there's probably a measurable decrease in transcription accuracy that worsens as you deviate from 1x speed. Also we really need to support bigger/longer file uploads :)

nerder92 1y ago
Quick Feedback: Would it be cool to research this internally and maybe find a sweet spot in speed multiplier where the loss is minimal. This pre-processing is quite cheap and could bring down the API price eventually.
georgemandis 1y ago
I kind of want to take a more proper poke at this but focus more one summarization accuracy over word-for-word accuracy, though I see the value in both. I'm actually curious, if I run transcriptions back-to-back-to-back on the exact same audio, how much variance should I expect? Maybe I'll try three approaches: - A straight diff comparison (I know a lot of people are calling for this, but I really think this is less useful than it sounds) - A "variance within the modal" test running it multiple times against the same audio, tracking how much it varies between runs - An LLM analysis assessing if the primary points from a talk were captured and summarized at 1x, 2x, 3x, 4x runs (I think this is far more useful and interesting)

celltalk 1y ago

With this logic, you should also be able to trim the parts that doesn’t have words. Just add a cut-off for db, and trim the video before transcription. Possibly another 10-20% gain?

isubkhankulov 1y ago

Transcripts get much more valuable when one diarizes the audio beforehand to determine which speaker said what. I use this free tool to extract those and dump the transcripts into a LLM with basic prompts: https://contentflow.megalabs.co https://contentflow.megalabs.co

mt_ 1y ago

You can just dump the youtube link video in Google AI studio and ask it to transcribe the video with speaker labels and even ask it it to add useful visual clues, because the model is multimodal for video too.

MaxDPS 1y ago
Can I ask what you mean by “useful visual clues”?
- mt_ 1y ago
  What is the speaker showcasing in its slides, what is it's body language and so on.

cprayingmantis 1y ago

I noticed something similar with images as inputs to Claude, you can scale down the images and still get good outputs. There is an accuracy drop off at a certain point but the token savings are worth doing a little tuning there.

georgemandis 1y ago
Definitely in the same spirit! Clearly the next thing we need to test is removing all the vowels from words, or something like that :)

meerab 1y ago

Interesting approach to transcript generation! I'm implementing a similar workflow for VideoToBe.com My Current Pipeline: Media Extraction - yt-dlp for reliable video/audio downloads Local Transcription - OpenAI Whisper running on my own hardware (no API costs) Storage & UI - Transcripts stored in S3 with a custom web interface for viewing Y Combinator playlist https://videotobe.com/play/playlist/ycombinator https://videotobe.com/play/playlist/ycombinator and Andrej's talk is https://videotobe.com/play/youtube/LCEmiRjPEtQ https://videotobe.com/play/youtube/LCEmiRjPEtQ After reading your blog post, I will be testing effect on speeding audio for locally-hosted Whisper models. Running Whisper locally eliminates the ongoing cost concerns since my infrastructure is already a sunk cost. Speeding audio could be an interesting performance enhancement to explore!

fuzztester 1y ago

Stop being slaves of extorters of any kind, and just leave. there is tons of this happening everywhere, and we need to fight this, and boycott it.

pottertheotter 1y ago

You can just ask Gemini to summarize it for you. It's free. I do it all the time with YouTube videos. Or you can just copy the transcript that YouTube provides below the video.

BrunoJo 1y ago

If you look for a cheaper transcription API you could als use https://Lemonfox.ai https://Lemonfox.ai. We've optimized the API for long audio files and are much faster and cheaper than OpenAI.

spapinwar 1y ago

[dead]

Raphell 1y ago

[flagged]

weird-eye-issue 1y ago
Can we ban this "person" for AI replies?
- Raphell 1y ago
  I get it. But I'm just someone who likes to think things through and say them simply.
  - weird-eye-issue 1y ago
    You aren't saying anything simply. You are verbosely saying nothing.

conjecTech 1y ago

If you are hosting whisper yourself, you can do something slightly more elegant, but with the same effect. You can downsample/pool the context 2:1 (or potentially more) a few layers into the encoder. That allows you to do the equivalent of speeding up audio without worry about potential spectral losses. For whisper large v3, that gets you nearly double throughput in exchange for a relative ~4% WER increase.

nomercy400 1y ago
Do you have more details or examples on how to downsample the context in the encoder? I treat the encoder as an opaque block, so I have no idea where to start.
- conjecTech 1y ago
  It's a very simple change in a vanilla python implementation. The encoder is a set of attention blocks, and the length of the attention can be changed without changing the calculation at all. Here(https://github.com/openai/whisper/blob/main/whisper/model.py#L201 https://github.com/openai/whisper/blob/main/whisper/model.py...) is the relevant code in the whisper repo. You'd just need to change the for loop to an enumerate and subsample the context along its length at the point you want. I believe it would be: for i, block in enumerate(self.blocks): x = block(x) if i==4: x = x[,,::2]

PeterStuer 1y ago

I wonder how much time and battery transcoding/uploading/downloading over coffeeshop wifi would realy save vs just running it locally through optimized Whisper.

georgemandis 1y ago
I had this same thought and won't pretend my fear was rational, haha. One thing that I thought was fairly clear in my write-up but feels a little lost in the comments: I didn't just try this with whisper. I tried it with their newer gpt-4o-transcription model, which seems considerably faster. There's no way to run that one locally.

KPennig86852 1y ago

But you know that you can run OpenAI's Whisper audio recognition model locally for free, right? It has very little GPU requirements, and the new "turbo" model works quite fast (there are also several Python libraries which make it significantly faster still).

dajonker 1y ago

Gemini 2.5 pro is, in my usage, quite superior for high quality transcriptions of phone calls, in Dutch in my case. As long as you upload the audio to GCS there you can easily process conversations of over an hour. It correctly identified and labeled speakers. The cheaper 2.5 flash made noticeably more mistakes, for example it didn't correctly output numbers while the Pro model did. As for OpenAI, their gpt-4o-transcribe model did worse than 2.5 flash, completely messing up names of places and/or people. Plus it doesn't label the conversation in turns, it just outputs a single continuous piece of text.

yashasolutions 1y ago

the question would be how to do that but also still get proper time code when using whisper to get the subtitles

ryanar 1y ago

In my experience, transcription software has no problem with transcribing sped up audio, or audio that is inaudible to humans or extremely loud (as long as not clipped), I wonder if LLM transcription works the same.

mushishi 1y ago

Do the APIs support simultaneous voice transcription in a way that different voices are tagged? (either in text or as metadata) If so: could you split the audiofile and process the latter half by pitch shifting, say an octave, and then merging them together to get shorter audiofile — then transcribe and join them back to a linear form, tagging removed. (You could insert some prerecorded voice to know at which point the second voice starts.). If pitch change is not enough, maybe manipulate it further by formants.

godot 1y ago

If you're already doing local ffmpeg stuff (i.e. pretty involved with code and scripting already) you're only a couple of steps more away from just downloading the openai-whisper models (or even the faster-whisper models which runs about two times faster). Since this looks like personal usage and not building production quality code, you can use AI (e.g. Cursor) to write a script to run the whisper model inference in seconds. Then there is no cost at all to run any length of audio. (since cost seems to be the primary factor of this article) On my m1 mac laptop it takes me about 30 seconds to run it on a 3-minute audio file. I'm guessing for a 40 minute talk it takes about 5-10 minutes to run.

Tepix 1y ago
Have you tried faster-whisper and whisper.cpp?
- godot 1y ago
  Yeah, my mentioned times are with faster-whisper, but I have not tried whisper.cpp. I just use a python script to run the model.

ta8903 1y ago

This "hack" also works in real life, youtubers low to talk slowly to increase the video runtime so I watch everything other than songs at 2x speed (and that's only because their player doesn't let you go faster).

7speter 1y ago

So wait… is whisper transcription really all that slow locally on a M3 Macbook? It’s been a while since I used whispercpp, but I seem to remember it taking maybe 20 minutes on a comparatively slowpoke (and powerhungry) i5 12600k for maybe 40 minutes of audio; it might take less time on a faster m chip (maybe I’m imagining mobile apple silicon to be more performant than even desktop intel cpus), even less if there support built in for the built in gpu cores and other ai optimized silicon? Did I miss that the task was time sensitive?

raluk 1y ago

Our team is working with soniox.com They are the most acurate model that works real time.

Nevermark 1y ago

It's also rude to talk slow to them. Unless its Siri.

another_twist 1y ago

You'd need a WER comparison to check if it really is no drop in quality. With this trick, there might be trouble if the audio is noisy, and it may. ot always be obvious whether or not to speed up.

addaidirectory 1y ago

That's a clever idea. Their are alternatives to OpenAI for audio transcription. Check them out https://www.addaidirectory.com/categories/audio https://www.addaidirectory.com/categories/audio or scroll the home page https://www.addaidirectory.com https://www.addaidirectory.com for updates