5 ms·

Wedeo – a Rust Rewrite of FFmpeg

6 points by sharifhsn 6mo ago

sharifhsn 6mo ago
After playing around with Claude Code for a bit, rewriting some Python tooling in Rust to great effect, I was interested in pushing the boundaries of what LLMs could do in terms of rewriting projects in Rust. The result is `wedeo`. For those unfamiliar, [FFmpeg](https://www.ffmpeg.org/ https://www.ffmpeg.org/) is "a complete, cross-platform solution to record, convert and stream audio and video". It is one of the most powerful and impressive pieces of open-source software and is the underlying infrastructure for a ton of A/V software. It is also, like the Linux kernel, written purely in C and Assembly. There are a couple reasons for why FFmpeg is a good candidate for this kind of project. The source code is incredibly high quality, being battle-tested and having contributions from the best experts in the area. A/V encoding and decoding is also rigorously described by specification, which is easily consumed by LLMs. The project's nature of being a bunch of codecs bundled together in a convenient interface made it easy to rewrite incrementally. C, being a systems programming language, maps nicely to Rust, and the assembly could be directly ported over. The major contribution `wedeo` currently has is an implementation of the H.264 decoder, which is around 30,000 lines of code for the scalar Rust implementation. It doesn't support some complex features like interlacing or 10-bit, but it should support 99% of H.264 encoded video. There has not been any kind of performance optimization done (except porting some assembly over and basic multithreading), so `wedeo` is much slower than FFmpeg. I expect the gap to close somewhat over time, but I doubt that even the most optimized Rust could beat FFmpeg's high quality C and Assembly. In order to get this project to the point where it can actually play video properly, I have utilized some existing libraries. The intention is for the codebase to be pure Rust, so any functionality which only exists as a non-Rust library will have to be rewritten. `symphonia` is used for audio, `rav1d` and `rav1e` for AV1, and `wgpu` and `winit` for the player. I have only confirmed that this works on MacOS M-series, so I would welcome testing on other machines. There is a lot missing from `wedeo`, in particular any video codecs besides H.264 and AV1, as well as H.264 encoding. I will be working on this as a side project, but I would also welcome contributions on these. And if anyone is interested in taking on an active role as maintainer, I'd gladly hand the reins over. # On AI Slop Although I can read and write Rust, 100% of the code in `wedeo` is AI-generated and I have not directly reviewed a single line of it, other than asking Claude to fix bugs and explain parts of it. It is intended as an experiment in pure LLM usage. I'm aware that this community and many other technical spaces online have been overwhelmed by "AI slop", bloated projects that have LOC that run in the tens of thousands, and I can see why many people might interpret `wedeo` that way. However, the amount of code in this project is similar to the equivalent amount in FFmpeg. I also think that this project is at least interesting in that this has never been attempted before, and I want to see where it can be taken. I think it's at least a better use of tokens than typical AI slop. Excited to see what the community thinks! P.S. I just saw [this tweet](https://x.com/FFmpeg/status/2039115531744334180 https://x.com/FFmpeg/status/2039115531744334180) from FFmpeg. Ironic, but I assure you this is not an April Fools' joke.
- Jyaif 6mo ago
  What's the unsafe situation like in Wedeo?
  - sharifhsn 6mo ago
    It’s used for multithreading and also for assembly loading. It’s not checked by anything other than Claude itself, so that would definitely be a strong candidate for manual review. I did have Claude ingest Rust Atomics & Locks by Mara Bos to help guide the implementation for concurrency, at least.
- fwip 6mo ago
  Thank you for being up-front in disclaiming that this project is AI-written, both here and in the Github page. I really appreciate the transparency.
- toastyavocado 6mo ago
  > 100% of the code in `wedeo` is AI-generated Thanks but no thanks
  - hk__2 6mo ago
    > It is intended as an experiment in pure LLM usage.
  - sharifhsn 6mo ago
    You don’t have to like it but you have to admit that it’s pretty impressive to pass the entire FATE test suite with purely AI-generated code. I honestly didn’t think it would be possible when I first started this.
- scottcodie 6mo ago
  Have you taken a look at the various test harnesses for video encoding/decoding, or consider publishing a standalone one yourself? I get a bit-for-bit rewrite can have some advantage but it's hard to know if the AI implementation actually covers all of the edges cases.
fishgoesblub 6mo ago
A rust rewrite that's GPL and isn't MIT or Apache licensed? I'm shocked.
- hk__2 6mo ago
  It’s generated by an AI so the "author" doesn’t care about the license.
  - sharifhsn 6mo ago
    I don’t know if I would say I don’t “care” about the license. I want to respect the FFmpeg license, and considering that my project is a straight ripoff, it would be dishonest to not have the same license. Whether AI-generated code can even be legally licensed or copyrighted is a separate question that hasn’t been decided by courts. I’ll happily comply with whatever experts recommend.
    - streaming 6mo ago
      Good answer. Asking an AI coding agent to port a C codebase to Rust is clearly and obviously creating a derivative work. You need to follow the terms of the LGPL license, and your Rust fork of FFMPEG must be licensed as LGPL. Videolan holds the copyright to FFMPEG, and that includes derivative works. LGPL allows you to do exactly what you're doing. Note that if you convert x264, x265 or any of the other GPL v2 libraries, your Rust version of those libraries will be GPL v2. When builds of FFMPEG are created with GPL v2 libraries, the entire build becomes GPL v2 (which has a more restrictive copyleft provision than LGPL).
      - sharifhsn 6mo ago
        Good to know! I have licensed as LGPL so I think that covers it. Rewriting x264 and x265 are in scope so I’ll keep that in mind. I will design the application as well so that those can easily be excluded from compilation to get a pure LGPL build.
- Hamuko 6mo ago
hk__2 6mo ago
> It provides no additional features compared to FFmpeg, and despite incorporating FFmpeg's assembly code, is significantly slower.
- Hamuko 6mo ago
  What a fantastic use of tokens. The author isn't even reviewing the code so you can't really even call it a study.
  - nh23423fefe 6mo ago
    > This codebase is AI-generated. Written by Claude (Anthropic) via Claude Code, directed and reviewed by a human.
    - sharifhsn 6mo ago
      I suppose “review” is misleading, since I’m not actually reading the code, I’m simply asking Claude to explain what the code is doing. I’ll change that so it’s expressly clear that none of this code is actually human-reviewed.
- sharifhsn 6mo ago
  These things may not be true in the future. I just meant to describe the current state of the project. With Rust being a modern language and the repository being AI-optimized, I wouldn't be surprised if it grew to have more features. And I hope that Rust can close the gap on performance, it would be a great testament to Rust's usefulness for high performance software.
hkmaxpro 6mo ago
Worth mentioning https://github.com/yazaldefilimone/ffmpreg https://github.com/yazaldefilimone/ffmpreg, another rust rewrite of ffmpeg (apparently from scratch, not translated)
- sharifhsn 6mo ago
  I found this project, but I’m not surprised that it doesn’t have much progress. Doing this kind of Rust rewrite is an absolutely gargantuan undertaking that I don’t think it would be feasible without AI. It’s not as if FFmpeg is some crufty, slow code that would seriously benefit from it.
- charlie90 6mo ago
  Why even keep working on this by hand for months when an AI can just write it fully featured in a week?
  - jdiff 6mo ago
    Because it can't. I'm aware of only two projects that have attempted this. 1. Anthropic's C compiler. This is a buggy, unoptimized, unwieldy mess. One that can compile the Linux kernel, which is no mean feat. But it is not "fully featured." And it had an incredible array of tests to start with. 2. Cursor's web browser. This is, flatly, a trainwreck. It's built on top of massive dependencies that do incredible amounts of heavy lifting despite claims to the contrary, and never seems to have ever produced a successful build.
    - sharifhsn 6mo ago
      The tests are the real clincher here. That’s the main reason that this project has made so much progress: the FATE suite highlights specific bugs to fix and tracks regressions. As for optimization, that seems to be more of a question of effort than whether it’s possible. I was able to take down the performance gap on Rust vs C (without Assembly) from 10x to 1.5x through detailed profiling and iterative improvements with Claude. It also looks like the Anthropic C compiler was built from scratch. By contrast, `wedeo` was directly based on FFmpeg’s existing code. Going by spec and test suite only would have taken a lot longer, and the quality would have been significantly lower.

6 points by sharifhsn 6mo ago

After playing around with Claude Code for a bit, rewriting some Python tooling in Rust to great effect, I was interested in pushing the boundaries of what LLMs could do in terms of rewriting projects in Rust. The result is `wedeo`. For those unfamiliar, [FFmpeg](https://www.ffmpeg.org/ https://www.ffmpeg.org/) is "a complete, cross-platform solution to record, convert and stream audio and video". It is one of the most powerful and impressive pieces of open-source software and is the underlying infrastructure for a ton of A/V software. It is also, like the Linux kernel, written purely in C and Assembly. There are a couple reasons for why FFmpeg is a good candidate for this kind of project. The source code is incredibly high quality, being battle-tested and having contributions from the best experts in the area. A/V encoding and decoding is also rigorously described by specification, which is easily consumed by LLMs. The project's nature of being a bunch of codecs bundled together in a convenient interface made it easy to rewrite incrementally. C, being a systems programming language, maps nicely to Rust, and the assembly could be directly ported over. The major contribution `wedeo` currently has is an implementation of the H.264 decoder, which is around 30,000 lines of code for the scalar Rust implementation. It doesn't support some complex features like interlacing or 10-bit, but it should support 99% of H.264 encoded video. There has not been any kind of performance optimization done (except porting some assembly over and basic multithreading), so `wedeo` is much slower than FFmpeg. I expect the gap to close somewhat over time, but I doubt that even the most optimized Rust could beat FFmpeg's high quality C and Assembly. In order to get this project to the point where it can actually play video properly, I have utilized some existing libraries. The intention is for the codebase to be pure Rust, so any functionality which only exists as a non-Rust library will have to be rewritten. `symphonia` is used for audio, `rav1d` and `rav1e` for AV1, and `wgpu` and `winit` for the player. I have only confirmed that this works on MacOS M-series, so I would welcome testing on other machines. There is a lot missing from `wedeo`, in particular any video codecs besides H.264 and AV1, as well as H.264 encoding. I will be working on this as a side project, but I would also welcome contributions on these. And if anyone is interested in taking on an active role as maintainer, I'd gladly hand the reins over. # On AI Slop Although I can read and write Rust, 100% of the code in `wedeo` is AI-generated and I have not directly reviewed a single line of it, other than asking Claude to fix bugs and explain parts of it. It is intended as an experiment in pure LLM usage. I'm aware that this community and many other technical spaces online have been overwhelmed by "AI slop", bloated projects that have LOC that run in the tens of thousands, and I can see why many people might interpret `wedeo` that way. However, the amount of code in this project is similar to the equivalent amount in FFmpeg. I also think that this project is at least interesting in that this has never been attempted before, and I want to see where it can be taken. I think it's at least a better use of tokens than typical AI slop. Excited to see what the community thinks! P.S. I just saw [this tweet](https://x.com/FFmpeg/status/2039115531744334180 https://x.com/FFmpeg/status/2039115531744334180) from FFmpeg. Ironic, but I assure you this is not an April Fools' joke.

Jyaif 6mo ago
What's the unsafe situation like in Wedeo?
- sharifhsn 6mo ago
  It’s used for multithreading and also for assembly loading. It’s not checked by anything other than Claude itself, so that would definitely be a strong candidate for manual review. I did have Claude ingest Rust Atomics & Locks by Mara Bos to help guide the implementation for concurrency, at least.
fwip 6mo ago
Thank you for being up-front in disclaiming that this project is AI-written, both here and in the Github page. I really appreciate the transparency.
toastyavocado 6mo ago
> 100% of the code in `wedeo` is AI-generated Thanks but no thanks
- hk__2 6mo ago
  > It is intended as an experiment in pure LLM usage.
- sharifhsn 6mo ago
  You don’t have to like it but you have to admit that it’s pretty impressive to pass the entire FATE test suite with purely AI-generated code. I honestly didn’t think it would be possible when I first started this.
scottcodie 6mo ago
Have you taken a look at the various test harnesses for video encoding/decoding, or consider publishing a standalone one yourself? I get a bit-for-bit rewrite can have some advantage but it's hard to know if the AI implementation actually covers all of the edges cases.

fishgoesblub 6mo ago

A rust rewrite that's GPL and isn't MIT or Apache licensed? I'm shocked.

hk__2 6mo ago
It’s generated by an AI so the "author" doesn’t care about the license.
- sharifhsn 6mo ago
  I don’t know if I would say I don’t “care” about the license. I want to respect the FFmpeg license, and considering that my project is a straight ripoff, it would be dishonest to not have the same license. Whether AI-generated code can even be legally licensed or copyrighted is a separate question that hasn’t been decided by courts. I’ll happily comply with whatever experts recommend.
  - streaming 6mo ago
    Good answer. Asking an AI coding agent to port a C codebase to Rust is clearly and obviously creating a derivative work. You need to follow the terms of the LGPL license, and your Rust fork of FFMPEG must be licensed as LGPL. Videolan holds the copyright to FFMPEG, and that includes derivative works. LGPL allows you to do exactly what you're doing. Note that if you convert x264, x265 or any of the other GPL v2 libraries, your Rust version of those libraries will be GPL v2. When builds of FFMPEG are created with GPL v2 libraries, the entire build becomes GPL v2 (which has a more restrictive copyleft provision than LGPL).
    - sharifhsn 6mo ago
      Good to know! I have licensed as LGPL so I think that covers it. Rewriting x264 and x265 are in scope so I’ll keep that in mind. I will design the application as well so that those can easily be excluded from compilation to get a pure LGPL build.
Hamuko 6mo ago

hk__2 6mo ago

> It provides no additional features compared to FFmpeg, and despite incorporating FFmpeg's assembly code, is significantly slower.

Hamuko 6mo ago
What a fantastic use of tokens. The author isn't even reviewing the code so you can't really even call it a study.
- nh23423fefe 6mo ago
  > This codebase is AI-generated. Written by Claude (Anthropic) via Claude Code, directed and reviewed by a human.
  - sharifhsn 6mo ago
    I suppose “review” is misleading, since I’m not actually reading the code, I’m simply asking Claude to explain what the code is doing. I’ll change that so it’s expressly clear that none of this code is actually human-reviewed.
sharifhsn 6mo ago
These things may not be true in the future. I just meant to describe the current state of the project. With Rust being a modern language and the repository being AI-optimized, I wouldn't be surprised if it grew to have more features. And I hope that Rust can close the gap on performance, it would be a great testament to Rust's usefulness for high performance software.

hkmaxpro 6mo ago

Worth mentioning https://github.com/yazaldefilimone/ffmpreg https://github.com/yazaldefilimone/ffmpreg, another rust rewrite of ffmpeg (apparently from scratch, not translated)

sharifhsn 6mo ago
I found this project, but I’m not surprised that it doesn’t have much progress. Doing this kind of Rust rewrite is an absolutely gargantuan undertaking that I don’t think it would be feasible without AI. It’s not as if FFmpeg is some crufty, slow code that would seriously benefit from it.
charlie90 6mo ago
Why even keep working on this by hand for months when an AI can just write it fully featured in a week?
- jdiff 6mo ago
  Because it can't. I'm aware of only two projects that have attempted this. 1. Anthropic's C compiler. This is a buggy, unoptimized, unwieldy mess. One that can compile the Linux kernel, which is no mean feat. But it is not "fully featured." And it had an incredible array of tests to start with. 2. Cursor's web browser. This is, flatly, a trainwreck. It's built on top of massive dependencies that do incredible amounts of heavy lifting despite claims to the contrary, and never seems to have ever produced a successful build.
  - sharifhsn 6mo ago
    The tests are the real clincher here. That’s the main reason that this project has made so much progress: the FATE suite highlights specific bugs to fix and tracks regressions. As for optimization, that seems to be more of a question of effort than whether it’s possible. I was able to take down the performance gap on Rust vs C (without Assembly) from 10x to 1.5x through detailed profiling and iterative improvements with Claude. It also looks like the Anthropic C compiler was built from scratch. By contrast, `wedeo` was directly based on FFmpeg’s existing code. Going by spec and test suite only would have taken a lot longer, and the quality would have been significantly lower.