Is it currently possible, either through the Admin interface or by passing parameters in configuration files or startup parameters for a Peertube-Runner-service (all runners on separate machines), to define a preference order?
For each runner, you can already specify which types of tasks it should or should not handle. However, is it also possible to define a priority level so that certain machines are preferred to pick up tasks first, provided they are available?
I am asking this because there can be significant differences between runners in terms of CPU capacity, number of threads, and available RAM. To give a theoretical example: if there are no active tasks, I would prefer an encoding or transcription job for a 20 GB video to run on the most powerful machine available. So ideally, if available, a machine with 8 CPUs, many threads, and 128 GB of RAM would be selected before a runner that is certainly helpful during peak load, but only has 1 CPU, 2 threads, and 16 GB of RAM.. I would then put the most powerfull machines on top of that list.
Perhaps this could be implemented by assigning a priority score in the runners list within the admin settings, or by allowing administrators to drag and reorder runners according to preference.
Hello,
No sorry it’s not currently implemented.
Is there a way to submit a feature request somewhere?
I currently have multiple runners to encode larger files in parallel and handle transcription. However, from what I can see, or perhaps I’m misunderstanding how it works, in many cases an entire queue gets assigned to a single runner machine, which then processes everything at maximum capacity, while all the other runners remain idle.
From the outside, it feels a bit like a casino. Jobs seem to be assigned randomly, and it’s unclear why a job isn’t sent to a runner that is currently doing nothing.
My feature request for setups with multiple runners would be:
-
Send the job to the runner with the best available capacity. For example, allowing the administrator to define what « best » means, whether that’s CPU power, available resources, or other criteria through configurable priorities.
-
If that runner is busy or at capacity, send the job to the next runner in the priority list.
-
If all runners are busy or at capacity (which I understand is a different situation), then go through the list again and assign jobs accordingly.
Something along those lines.
At the moment, I can scale up by adding more runners, but that doesn’t actually solve the underlying problem. It only increases the cost.
Yes, on GitHub - Chocobozzz/PeerTube: ActivityPub-federated video streaming platform using P2P directly in your web browser · GitHub
Issue · GitHub seems like your feature request
job isn’t sent to a runner that is currently doing nothing
PeerTube sends a notification to all runners, the first one asking for a job is taking it. In your current setup, you may choose to limit concurrency on each runner by editing their configuration (by default it’s 2).
Thanks for the quick and helpful response!
I have now configured the runners so that the number of CPUs and cores matches the concurrency parameter. Suppose I set all runners to a maximum concurrency of 1. Does this then apply to:
- vod-web-video-transcoding
- vod-hls-transcoding
- video-transcription
- vod-audio-merge-transcoding
In other words, is every “UUID” action simply one task that counts against the runner concurrency parameter?
Or, for example, does video-transcription not fall under that parameter and operate separately?
Previously, I had 2 runners dedicated only to video-transcription, and 4 runners dedicated to vod-web-video-transcoding and vod-hls-transcoding. However, that did not improve the overall processing time, possibly because large files then still need to be distributed across multiple runners or something similar.
So the remaining sub-question I have, in order to determine whether I should set concurrency to 1 despite these being very powerful machines that should be able to handle more, is this:
Does a video-transcription task on a runner take the FFmpeg concurrency parameter into account, or is it completely separate from that?
Thanks!
concurrency says to the runner to take at most concurrency job (represented by a specific UUID in the PeerTube UI). Then, each job can spawn a process (ffmpeg, whisper) that can take multiple threads.
Does a video-transcription task on a runner take the FFmpeg concurrency parameter into account, or is it completely separate from that?
Do not confuse concurrency [job] config and threads [ffmpeg] config 
