7 ms·

FFmpeg-over-IP – Connect to remote FFmpeg servers

235 points by steelbrain 6mo ago

deadlyllama 6mo ago
I've been meaning to build exactly this for a while, and my waiting has been rewarded by someone else doing it! Had you thought about using FUSE on the server side, rather than patching ffmpeg? Like a reverse sshfs? That avoids patching the ffmpeg binary, which allows usage of wierd and wonderful ffmpeg binaries with other people's patches. I'd be interested in seeing how well it works with SBC GPUs - many have hardware decoding and encoding, and their vendors love to fork ffmpeg.
- steelbrain 6mo ago
  I've explored sftp since ffmpeg has built-in support for it (-i sftp://...), but the support is quite buggy in code, I hope to submit some patches upstream to be able to change it. FTP in contrast seemed much more stable, at least looking at the code. FTP had some other shortcomings that made it undesirable for my usecase. That was the one motivation, the other one was that it would require rewriting arguments going into the server. What you're describing was essentially what ffmpeg-over-ip v4 (and its earlier versions!) was, and the constant feedback I'd heard was that sharing filesystems is too much work, ssh servers on windows and macOS are a bad experience, people want to use a bundled solution. Forking ffmpeg was no easy task! Took forever to figure out the build process, eventually caved in and started using jellyfin build scripts, but that has the downside of being a few versions behind of upstream HEAD.
  - deadlyllama 6mo ago
    Sharing filesystems is hard when you make users do it in advance. I was thinking of the server end of an ffmpeg-over-ip system bringing up a FUSE filesystem backed by something similar to your VFS-served-by-the-client. Combine that either with argument rewriting, or chrooting into the FUSE filesystem. As another commenter said, where's plan 9 when you need it? If you go the FUSE route there are existing 9P implementations for both server and FUSE client you can use.
anonym29 6mo ago
Missed opportunity to call this FoIP
assimpleaspossi 6mo ago
So this is port forwarding? (I'm too tired and can't concentrate to read this and am going to bed.)
- steelbrain 6mo ago
  tl;dr: - Client makes request to server (which opens a bidirectional network socket) - Server uses that bidirectional socket, spawns a local patched ffmpeg with vfs-like characteristics - ffmpeg (using client-server bidrection socket) does input/output operations, treating client filesystem as if it was local Thus client doesn't need to open any ports, or expose its filesystem in a traditional mounting manner, and one server can handle filesystems & requests of any amount of clients.
basilikum 6mo ago
ffmpeg already has network capabilities. You can let it open a tcp socket and stream input from there and write output to another TCP socket. How is this different? Is this just a wrapper around this functionality for more convenience or does it provide any fundamentally new features?
- steelbrain 6mo ago
  To be able to use ffmpeg with its native network capabilities in a usecase of media servers, where you need to stream your input to it, and then get multiple outputs (think HLS) that are streamed back is not possible at this point in time. HTTP, FTP, SFTP, all have their limitations, some are outright broken for HLS usecases, others wont stream seeking. I would have very much loved to use the built-in capabilities instead of patching ffmpeg to add a vfs layer and spend a ton of time figuring out the build pipeline once you add all the codecs and hwaccels. I do hope to be able to change this in the future, I've identified several bugs that I intend to submit patches for.
  - halayli 6mo ago
    This is not a special case. Everything you mentioned above can actually be achieved using cli. You can create listeners, configure pipelines, and sinks(granted not ergonomic). Sinks can be HTTP post for example, and sources can be tcp listeners + protocols on top. You can also configure the buffering strategies for each pipeline.
MrBuddyCasino 6mo ago
Could this be generalized? A wrapper that intercepts I/O syscalls to the filesystem of a remote client? This would work with binaries other than ffmpeg.
- steelbrain 6mo ago
  Yes, apply the patches on any binary, build it and rename it to ffmpeg and it’ll work over the network with the server and client included
- OvervCW 6mo ago
  That's exactly what I set out to do with my pet project :) https://github.com/Overv/outrun https://github.com/Overv/outrun
steelbrain 6mo ago
Hi HN! I'm excited to show case an update to a personal project of mine. Its called ffmpeg-over-ip and it allows you connect to remote ffmpeg servers, so if you have one machine with a GPU (could be your windows gaming laptop, gaming PC, a macbook?) and a machine (or VM, docker container etc) without a GPU, you could use the remote GPU to do GPU-accelerated video conversion. The way it works is pretty neat, there are two components, a server and a client. - The server (has the GPU) comes with a patched up ffmpeg and listens on a specified port - The client (without the GPU) connects to the server, takes the file IO requests from the server and runs them locally. ffmpeg doesn't know that its not dealing with a local filesystem, so this approach works with multiple inputs or outputs like HLS, and is perfect for home media servers like Plex or Jellyfin or Emby. One server can offer up its GPU to as many clients as needed, all clients offer up their own filesystems for their requests, the server also comes with a static build of ffmpeg bundled (built from jellyfin-ffmpeg scripts) so all you have to do is create a config file, set a password and you're good to go! It's been about a year and half since this was last submitted (https://news.ycombinator.com/item?id=41743780 https://news.ycombinator.com/item?id=41743780). The feedback at the time was around the difficulty of sharing a filesystem between the machines so that should no longer be a problem. This has been really useful in my local setup, I hope you find it useful. If you have any further questions, the website has some FAQs (linked in github repo), or you could post them here and I'll answer them for you! Thanks!
- bsimpson 6mo ago
  Nice to see you here! Was really impressed by your work on Pundle. (It was an amazingly fast HMR dev environment - much like Vite today.) Felt like I was the only one using it, but it was hard to walk away from instant updates.
  - steelbrain 6mo ago
    Thanks for putting a smile on my face! I am glad you liked it! :)
- doctorpangloss 6mo ago
  What's the point of this? A single CPU core on a 9500T or a Ryzen V1500B is fast enough to real-time re-encode 60mbps 4K H264 to 1080p 5mbps h264, aka, for a core use case - transcoding for web for Jellyfin over cellular, for example - you haven't needed hardware video engines on PCs for 9 YEARS. I have no idea why people are so hung up on hardware video encoding. It's completely wrong. The quality is worse. The efficiency is a red herring - you will still use every CPU core for IO threads in ffmpeg, if you don't configure that away, which you do not. And it requires really annoying setup and premium features on stuff like Plex. It just makes no sense! If latency is important to you, well then hardware engines make sense. But you are throwing away the latency sending it over the network. The only use case (basically) is video game streaming, and in that case, you'll have a local GPU. I have never read one of these ffmpeg network hardware encode innovations to have an actual benchmark comparison to single thread software transcoding tasks. I know you mean well but really. It makes NO sense.
toomuchtodo 6mo ago
Thanks for the update and continuing to share this project. What does the roadmap look like into the future?
- steelbrain 6mo ago
  I am thinking of adding a Windows application with an installer and a tray icon that you can use for some basic settings like changing port or password, or toggling automatic startup. For linux, I am thinking of adding convenience helpers around systemd service installation
  - toomuchtodo 6mo ago
    Very cool. Peertube supports remote runners [1] [2], might take a look for inspiration. As a distributed compute enthusiast, big fan of of this model for media processing. [1] https://docs.joinpeertube.org/maintain/tools#peertube-runner https://docs.joinpeertube.org/maintain/tools#peertube-runner [2] https://docs.joinpeertube.org/admin/remote-runners https://docs.joinpeertube.org/admin/remote-runners
    - bastiao 6mo ago
      Very nice. Could this concept be applied to have ffmpeg running in webassembly, and over HTTP?
      - toomuchtodo 6mo ago
        Similar to https://github.com/ffmpegwasm/ffmpeg.wasm https://github.com/ffmpegwasm/ffmpeg.wasm ?
        bastiao 6mo ago
        Exactly - but can transcoding be remote?
    - steelbrain 6mo ago
offmycloud 6mo ago
How does this differ in performance from rffmpeg? https://github.com/joshuaboniface/rffmpeg https://github.com/joshuaboniface/rffmpeg
- steelbrain 6mo ago
  I tried to answer it about a year and half ago and that answer is still mostly correct: https://news.ycombinator.com/item?id=41743932 https://news.ycombinator.com/item?id=41743932 You can mix and match operating systems, macOS, Windows, Linux, you do not need sudo privileges. rffmpeg needs a shared file system which could be a huge pain to setup: https://github.com/joshuaboniface/rffmpeg/blob/master/docs/SETUP.md#nfs-setup https://github.com/joshuaboniface/rffmpeg/blob/master/docs/S... ffmpeg-over-ip patches ffmpeg and only needs one port open for the server, then you just run the binary, no mounts needed at all.
naikrovek 6mo ago
This is software which basically replicates what Plan9 gives you out of the box. Dammit I really wish Plan9 had taken off. It isn’t perfect but it does a much, much better job of helping me run my applications in ways that I want. If anyone doesn’t already know, one method of Plan9 remote access is to “cpu” into a remote machine which has the hardware you need. Your local filesystems go with you, and your environment on the remote machine consists of your local filesystems mounted to the remote machine, but only for you, and all applications you run in this context execute on the cpu of the remote machine and have access to the remote machines hardware, but your local filesystems. Imagine SSHing into a remote machine and your entire environment goes with you, development tools and all. That’s what Plan9 does for you. So if I “cpu” into a machine without ffmpeg, but with a GPU and I run ffmpeg, not only will it work, but I can tell ffmpeg to use a hardware encoder with a simple command line flag, and it’ll work.
- olowe 6mo ago
  The great thing about Plan 9 is it can make other distributed systems look so complicated. The worst thing about Plan 9 is that it can make other distributed systems look so complicated!
- cestith 6mo ago
  It would be nice to see http://9legacy.org/ http://9legacy.org/ pick up a bunch more development.
- arboles 6mo ago
  Or just run sshd on your client computer and mount (sshfs) a directory on the render farm server
  - naikrovek 6mo ago
    Sure that could be done but you have to do it manually every time and write a script to automate it for each host you connect to, then remember to change those when/if you have some new filesystem you want to bring over, then you have to put them in the $PATH on the remote machine every time, gracefully, so that when it comes time to disconnect, everything knows to unmount automatically before you disconnect from SSH. It’s an enormous pain in the ass to try to get SSH to do all of this correctly every time. The cpu command does it all gracefully, and it’s already written, it even works on Linux.
throwaway81523 6mo ago
This sounds like a nice project, but is it a Show HN without putting that in the title, because of the stigma that Show HN has recently acquired?
- hcfman 6mo ago
  Show HN? What is that? :-) I've been reading hacker news for ages, but somehow I missed that there was some convention relating to showing technology on hacker news.
- hcfman 6mo ago
  Ah, I see it now. It's a menu option. Never noticed :-)
- steelbrain 6mo ago
  You're going to have to chalk this up to human error, in the excitement to post it, I omitted Show HN:
longislandguido 6mo ago
On the surface this seems like a terrible idea: FFmpeg is mountains of extremely complex C code whose entire job is processing untrusted inputs. Choosing to make such code network-enabled if you can't trust your inputs, I would recommend to sandbox if at all possible. Otherwise you are asking for trouble.
- steelbrain 6mo ago
  Thank you for your comment! The usecase for something like this is when you control both sides, server & client. There is some basic HMAC auth built into each request. > I would recommend to sandbox if at all possible. Since the server is a standard binary that doesn't need any special permissions, you could create the most locked down user in your server that only has access to a limit set of files and the GPUs and it'll work just fine. This is encouraged.

235 points by steelbrain 6mo ago

I've been meaning to build exactly this for a while, and my waiting has been rewarded by someone else doing it! Had you thought about using FUSE on the server side, rather than patching ffmpeg? Like a reverse sshfs? That avoids patching the ffmpeg binary, which allows usage of wierd and wonderful ffmpeg binaries with other people's patches. I'd be interested in seeing how well it works with SBC GPUs - many have hardware decoding and encoding, and their vendors love to fork ffmpeg.

steelbrain 6mo ago
I've explored sftp since ffmpeg has built-in support for it (-i sftp://...), but the support is quite buggy in code, I hope to submit some patches upstream to be able to change it. FTP in contrast seemed much more stable, at least looking at the code. FTP had some other shortcomings that made it undesirable for my usecase. That was the one motivation, the other one was that it would require rewriting arguments going into the server. What you're describing was essentially what ffmpeg-over-ip v4 (and its earlier versions!) was, and the constant feedback I'd heard was that sharing filesystems is too much work, ssh servers on windows and macOS are a bad experience, people want to use a bundled solution. Forking ffmpeg was no easy task! Took forever to figure out the build process, eventually caved in and started using jellyfin build scripts, but that has the downside of being a few versions behind of upstream HEAD.
- deadlyllama 6mo ago
  Sharing filesystems is hard when you make users do it in advance. I was thinking of the server end of an ffmpeg-over-ip system bringing up a FUSE filesystem backed by something similar to your VFS-served-by-the-client. Combine that either with argument rewriting, or chrooting into the FUSE filesystem. As another commenter said, where's plan 9 when you need it? If you go the FUSE route there are existing 9P implementations for both server and FUSE client you can use.

anonym29 6mo ago

Missed opportunity to call this FoIP

assimpleaspossi 6mo ago

So this is port forwarding? (I'm too tired and can't concentrate to read this and am going to bed.)

steelbrain 6mo ago
tl;dr: - Client makes request to server (which opens a bidirectional network socket) - Server uses that bidirectional socket, spawns a local patched ffmpeg with vfs-like characteristics - ffmpeg (using client-server bidrection socket) does input/output operations, treating client filesystem as if it was local Thus client doesn't need to open any ports, or expose its filesystem in a traditional mounting manner, and one server can handle filesystems & requests of any amount of clients.

basilikum 6mo ago

ffmpeg already has network capabilities. You can let it open a tcp socket and stream input from there and write output to another TCP socket. How is this different? Is this just a wrapper around this functionality for more convenience or does it provide any fundamentally new features?

steelbrain 6mo ago
To be able to use ffmpeg with its native network capabilities in a usecase of media servers, where you need to stream your input to it, and then get multiple outputs (think HLS) that are streamed back is not possible at this point in time. HTTP, FTP, SFTP, all have their limitations, some are outright broken for HLS usecases, others wont stream seeking. I would have very much loved to use the built-in capabilities instead of patching ffmpeg to add a vfs layer and spend a ton of time figuring out the build pipeline once you add all the codecs and hwaccels. I do hope to be able to change this in the future, I've identified several bugs that I intend to submit patches for.
- halayli 6mo ago
  This is not a special case. Everything you mentioned above can actually be achieved using cli. You can create listeners, configure pipelines, and sinks(granted not ergonomic). Sinks can be HTTP post for example, and sources can be tcp listeners + protocols on top. You can also configure the buffering strategies for each pipeline.

MrBuddyCasino 6mo ago

Could this be generalized? A wrapper that intercepts I/O syscalls to the filesystem of a remote client? This would work with binaries other than ffmpeg.

steelbrain 6mo ago
Yes, apply the patches on any binary, build it and rename it to ffmpeg and it’ll work over the network with the server and client included
OvervCW 6mo ago
That's exactly what I set out to do with my pet project :) https://github.com/Overv/outrun https://github.com/Overv/outrun

steelbrain 6mo ago

Hi HN! I'm excited to show case an update to a personal project of mine. Its called ffmpeg-over-ip and it allows you connect to remote ffmpeg servers, so if you have one machine with a GPU (could be your windows gaming laptop, gaming PC, a macbook?) and a machine (or VM, docker container etc) without a GPU, you could use the remote GPU to do GPU-accelerated video conversion. The way it works is pretty neat, there are two components, a server and a client. - The server (has the GPU) comes with a patched up ffmpeg and listens on a specified port - The client (without the GPU) connects to the server, takes the file IO requests from the server and runs them locally. ffmpeg doesn't know that its not dealing with a local filesystem, so this approach works with multiple inputs or outputs like HLS, and is perfect for home media servers like Plex or Jellyfin or Emby. One server can offer up its GPU to as many clients as needed, all clients offer up their own filesystems for their requests, the server also comes with a static build of ffmpeg bundled (built from jellyfin-ffmpeg scripts) so all you have to do is create a config file, set a password and you're good to go! It's been about a year and half since this was last submitted (https://news.ycombinator.com/item?id=41743780 https://news.ycombinator.com/item?id=41743780). The feedback at the time was around the difficulty of sharing a filesystem between the machines so that should no longer be a problem. This has been really useful in my local setup, I hope you find it useful. If you have any further questions, the website has some FAQs (linked in github repo), or you could post them here and I'll answer them for you! Thanks!

bsimpson 6mo ago
Nice to see you here! Was really impressed by your work on Pundle. (It was an amazingly fast HMR dev environment - much like Vite today.) Felt like I was the only one using it, but it was hard to walk away from instant updates.
- steelbrain 6mo ago
  Thanks for putting a smile on my face! I am glad you liked it! :)
doctorpangloss 6mo ago
What's the point of this? A single CPU core on a 9500T or a Ryzen V1500B is fast enough to real-time re-encode 60mbps 4K H264 to 1080p 5mbps h264, aka, for a core use case - transcoding for web for Jellyfin over cellular, for example - you haven't needed hardware video engines on PCs for 9 YEARS. I have no idea why people are so hung up on hardware video encoding. It's completely wrong. The quality is worse. The efficiency is a red herring - you will still use every CPU core for IO threads in ffmpeg, if you don't configure that away, which you do not. And it requires really annoying setup and premium features on stuff like Plex. It just makes no sense! If latency is important to you, well then hardware engines make sense. But you are throwing away the latency sending it over the network. The only use case (basically) is video game streaming, and in that case, you'll have a local GPU. I have never read one of these ffmpeg network hardware encode innovations to have an actual benchmark comparison to single thread software transcoding tasks. I know you mean well but really. It makes NO sense.

toomuchtodo 6mo ago

Thanks for the update and continuing to share this project. What does the roadmap look like into the future?

steelbrain 6mo ago
I am thinking of adding a Windows application with an installer and a tray icon that you can use for some basic settings like changing port or password, or toggling automatic startup. For linux, I am thinking of adding convenience helpers around systemd service installation
- toomuchtodo 6mo ago
  Very cool. Peertube supports remote runners [1] [2], might take a look for inspiration. As a distributed compute enthusiast, big fan of of this model for media processing. [1] https://docs.joinpeertube.org/maintain/tools#peertube-runner https://docs.joinpeertube.org/maintain/tools#peertube-runner [2] https://docs.joinpeertube.org/admin/remote-runners https://docs.joinpeertube.org/admin/remote-runners
  - bastiao 6mo ago
    Very nice. Could this concept be applied to have ffmpeg running in webassembly, and over HTTP?
    - toomuchtodo 6mo ago
      Similar to https://github.com/ffmpegwasm/ffmpeg.wasm https://github.com/ffmpegwasm/ffmpeg.wasm ?
      - bastiao 6mo ago
        Exactly - but can transcoding be remote?
  - steelbrain 6mo ago

offmycloud 6mo ago

How does this differ in performance from rffmpeg? https://github.com/joshuaboniface/rffmpeg https://github.com/joshuaboniface/rffmpeg

steelbrain 6mo ago
I tried to answer it about a year and half ago and that answer is still mostly correct: https://news.ycombinator.com/item?id=41743932 https://news.ycombinator.com/item?id=41743932 You can mix and match operating systems, macOS, Windows, Linux, you do not need sudo privileges. rffmpeg needs a shared file system which could be a huge pain to setup: https://github.com/joshuaboniface/rffmpeg/blob/master/docs/SETUP.md#nfs-setup https://github.com/joshuaboniface/rffmpeg/blob/master/docs/S... ffmpeg-over-ip patches ffmpeg and only needs one port open for the server, then you just run the binary, no mounts needed at all.

naikrovek 6mo ago

This is software which basically replicates what Plan9 gives you out of the box. Dammit I really wish Plan9 had taken off. It isn’t perfect but it does a much, much better job of helping me run my applications in ways that I want. If anyone doesn’t already know, one method of Plan9 remote access is to “cpu” into a remote machine which has the hardware you need. Your local filesystems go with you, and your environment on the remote machine consists of your local filesystems mounted to the remote machine, but only for you, and all applications you run in this context execute on the cpu of the remote machine and have access to the remote machines hardware, but your local filesystems. Imagine SSHing into a remote machine and your entire environment goes with you, development tools and all. That’s what Plan9 does for you. So if I “cpu” into a machine without ffmpeg, but with a GPU and I run ffmpeg, not only will it work, but I can tell ffmpeg to use a hardware encoder with a simple command line flag, and it’ll work.

olowe 6mo ago
The great thing about Plan 9 is it can make other distributed systems look so complicated. The worst thing about Plan 9 is that it can make other distributed systems look so complicated!
cestith 6mo ago
It would be nice to see http://9legacy.org/ http://9legacy.org/ pick up a bunch more development.
arboles 6mo ago
Or just run sshd on your client computer and mount (sshfs) a directory on the render farm server
- naikrovek 6mo ago
  Sure that could be done but you have to do it manually every time and write a script to automate it for each host you connect to, then remember to change those when/if you have some new filesystem you want to bring over, then you have to put them in the $PATH on the remote machine every time, gracefully, so that when it comes time to disconnect, everything knows to unmount automatically before you disconnect from SSH. It’s an enormous pain in the ass to try to get SSH to do all of this correctly every time. The cpu command does it all gracefully, and it’s already written, it even works on Linux.

throwaway81523 6mo ago

This sounds like a nice project, but is it a Show HN without putting that in the title, because of the stigma that Show HN has recently acquired?

hcfman 6mo ago
Show HN? What is that? :-) I've been reading hacker news for ages, but somehow I missed that there was some convention relating to showing technology on hacker news.
hcfman 6mo ago
Ah, I see it now. It's a menu option. Never noticed :-)
steelbrain 6mo ago
You're going to have to chalk this up to human error, in the excitement to post it, I omitted Show HN:

longislandguido 6mo ago

On the surface this seems like a terrible idea: FFmpeg is mountains of extremely complex C code whose entire job is processing untrusted inputs. Choosing to make such code network-enabled if you can't trust your inputs, I would recommend to sandbox if at all possible. Otherwise you are asking for trouble.

steelbrain 6mo ago
Thank you for your comment! The usecase for something like this is when you control both sides, server & client. There is some basic HMAC auth built into each request. > I would recommend to sandbox if at all possible. Since the server is a standard binary that doesn't need any special permissions, you could create the most locked down user in your server that only has access to a limit set of files and the GPUs and it'll work just fine. This is encouraged.