Peertube: Support for transcoding by remote workers

[scanlime]

I'm trying to run a peertube instance with a large volume of 1080p60 content and the transcode bottleneck is significant. One potentially easy way to improve this when additional CPU power is available on the local network could be to scale out the transcode with a worker daemon.

I'm opening this in the hope of a discussion for a future enhancement, and to keep track of that work.

In my case I'm imagining the workers being located in the same data center, but it seems like this could be extended to geographically diverse or even crowdsourced transcode sites.

Personally I'd be happy with a solution where the remote workers run a userspace daemon that manages ffmpeg with little to no local storage, accessing the peertube server's disks either over an existing network storage protocol or existing media streaming protocol.

I'm curious what takes other folks have. I know this is unlikely to be interesting for people running peertube on a small VPS, unless the transcode workers can be elsewhere on cheaper hosting (like at your house, on DSL).

[Jorropo]

This is planned but in desktop version (but not yet (desktop version is in infdev (before alpha)))

[rigelk]

A simpler way to implement is for clients to transcode first and then let the server decide if the video is properly transcoded.

But adding workers is another level of implementation. It's also something we need to define, because we have multiple possibilities: parallel transcoding of chunks of the same video, or transcoding of different videos. It also means the workers have to register at the PeerTube instance, and the instance has to monitor their progress with the trancoding tasks it gives them. Arguably, that's way more work than just making the client and server agree on whether or not the video needs additional trancoding.

[scanlime]

I'm interested in client-side transcodes too, but I thought that seemed much harder?

• it means running ffmpeg or similar, efficiently, in the browser or requiring all uploads to be done via a native app

• verifying that a transcode was correct seems non-trivial, if we want to do more than just check that the format and header look okay

[rigelk]

Well, they are two different kind of "hard" :stuck_out_tongue:

I'm not sure we want to check everything either ; checking for faststart and proper codec use via ffprobe should be enough.

[utack]

Maybe this helps you
https://github.com/tdaede/dve
Might however not be a method that is usable, as it includes usage of ssh

[Nutomic]

@utack It can only generate mkv containers, which isn't very helpful for Peertube. Plus it hasn't been maintained in years.

[roipoussiere]

I totally support this issue.

The is a real use-case where a federation of video producers wants to create many PeerTube instances, but use a unique transcoding server in order to reduce instance hosting costs.

[Nutomic]

This should actually be pretty easy to implement with ssh. I'm sure there is some ssh library that could be used. Basically, Peertube would do this:

scp video-to-transcode.mp4 transcoding-host:~
ssh transcoding-host ffmpeg ...
scp transcoding-host:transcoded-video.mp4 .

[rigelk]

scp video-to-transcode.mp4 transcoding-host:~
ssh transcoding-host ffmpeg ...
scp transcoding-host:transcoded-video.mp4 .

That would require to translate all the logic we have in https://github.com/Chocobozzz/PeerTube/blob/develop/server/helpers/ffmpeg-utils.ts to bash, so not so easy. An intermediate, easier-than-translating-to-bash step before your solution becomes possible, would be to decouple that logic in a standalone module (keeping it in Typescript), so that we could install it on the remote worker and just send it the job parameters. Plex does this with https://github.com/wnielson/Plex-Remote-Transcoder because it has a re-usable binary for the transcoding part, for instance.

[Nutomic]

Doesnt the library execute an ffmpeg command via bash anyway. It should be possible to get that command out, maybe with a patch.

The other option is to install nodejs on the worker, and have it execute the same library.

[olragon]

Doesnt the library execute an ffmpeg command via bash anyway. It should be possible to get that command out, maybe with a patch.

The other option is to install nodejs on the worker, and have it execute the same library.

ffmpeg('/path/to/file.avi')
  .on('start', function(commandLine) {
    console.log('Spawned Ffmpeg with command: ' + commandLine);
  });

https://github.com/fluent-ffmpeg/node-fluent-ffmpeg#start-ffmpeg-process-started

[JohnXLivingston]

In this forum post https://framacolibri.org/t/adding-a-new-resolution-to-an-existing-video-playlist/6247 , plhardy made this script with remote transcoding: https://framagit.org/artlog/piretubehack
It still is work in progress.

[kontrollanten]

May it be a first step to just do the transcoding in the browser? https://github.com/ffmpegwasm/ffmpeg.wasm seems to be a pretty easy way to solve that. It's not a perfect user experience, but at least it's a way to avoid heavy weight on the server side.

[scanlime]

transcoding in wasm is going to be especially slow until browsers and wasm
have SIMD support, and they might need improvements in multithreading too.
setting aside efficiency concerns, there are also security and correctness
implications in taking pre-encoded video from clients. most of all though,
I don't know if folks are going to want to leave one browser tab open for
hours or possibly days in order to get a video published

On Fri, Oct 30, 2020 at 7:37 AM kontrollanten [e-mail]
wrote:

May it be a first step to just do the transcoding in the browser?
https://github.com/ffmpegwasm/ffmpeg.wasm seems to be a pretty easy way
to solve that. It's not a perfect user experience, but at least it's a way
to avoid heavy weight on the server side.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Chocobozzz/PeerTube/issues/947#issuecomment-719590269,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHBRJGXZQ6EVGJR3UHPYKTSNLFTJANCNFSM4FRBAFHQ
.

[johanfleury]

Hey, just adding my use-case to the discussion: I run Peertube in Kubernetes with a limited amount of resources and, at the moment, ffmpeg always end up OOM killed as it try to get as much memory as it think possible (regardless of its cgroup’s limit).

Having remote workers would allow me to spawn pods with much higher memory and CPU limits, or even use dedicated VMs to do the transcoding.

[zblace]

Would it be possible theoretically to imagine and technically plan use a kind of 'proxy' transcoding via proxy-partner-server...for example if I have my (public-domain) video uploaded in high resolution first to Wikimedia Commons or Archive.org, then it is shared to my (to weak to transcode) PeerTube server?