Howdy! So I have a v4.2.0 instance configured to output HLS at 480p, 720p, and 1080p. I use object storage in the modern configuration (not s3fs), and it generally runs fine.
I have a video that failed to transcode at some point in the past that I’m trying to fix. When I use create-transcoding-job.js, each of my three resolution jobs fail because ffprobe looking for a 360p file:
Of course there’s no 360p file, because I don’t have that resolution selected. Why is it expecting it?
Here’s the thing, though: my object storage location already has a set of 1080p, 720p, and 240p video and m3u8 files in the right place, although they’re all named with the uuid of the video, which is maybe how it was done in an old version?
I really just need to get Peertube to use those instead of thinking the video needs transcoding, because this video used to work. I’m fine if that involves manually updating some values in the DB.
Maybe I have two questions:
How do I get the transcoding process to stop looking for a resolution I don’t want?
How can I get Peertube to use the files it already has in object storage?
I think these might be holdovers from the old way of doing object storage, however. I don’t know that they hurt anything.
I bet I was still doing 360p back when I first uploaded that video. I figured the site-wide resolution settings would apply to new transcoding jobs, even on old videos.
I rechecked my nginx config against the repo and switched those old rewrites out, since the object_storage config already had CDN configuration.
I saw the Github issue re: cleanly deleting specific file resolutions. Given that that’s not available, I copied the 360p file from object storage to the local disk and re-ran the transcoding jobs. It’ll be a few hours, but I’ll report how it goes. I’m still curious how I could clear that video of having a 360p entry.
Okay, this got messy. Moving the 360p mp4 and m3u8 files let the transcoding complete, but the move to object storage failed with the error message « job stalled more than allowable limit ». Unfortunately, this hung the video not in the TO_MOVE_TO_EXTERNAL_STORAGE_FAILED state, but the TO_MOVE_TO_EXTERNAL_STORAGE state, so I couldn’t just queue up another job to move it. I modified the public.video table to set the state to 8 (TO_MOVE_TO_EXTERNAL_STORAGE_FAILED) on the video, and then queued up another move.
Unfortunately, four more attempts to move the video to object storage have failed with the « job stalled » message, although some of the files have made it into object storage. I’ve rebooted the server, thinking maybe it was a memory situation; that didn’t help. So I’m stuck there, as I don’t see a way to increase the maxStalledCount, which is possibly the culprit.
It seems like there’s a legitimate bug around the failure state of that move to object storage, and I’ll put an issue into Github for it later. I’m hesitant to add anything about the un-graceful treatment of missing resolutions, since there’s already a feature request to allow removing selected resolutions.
To close the loop on this, this is fully resolved, and did involve some manual database work, unfortunately. I do not know when and how this video broke, but it must have been a combination of upgrades, old object storage configuration issues, plus the switch to the new object storage methods.
The failure state of the move to object storage job was cleaned up in the fix to the bug I logged, although I didn’t pull that code down.
So here’s roughly what I did after I got all resolutions transcoded, in case someone else finds themself in this sad situation. I do not recommend digging into the database unless you’re comfortable doing so. (My experience is in mySQL and SQL Server, so forgive me for the desperate and inconsistent column quoting here; postgresql is weird in what it forced me to quote.)
The repeated « job stalled more than the allowable limit » loop
Decreased my object_storage.max_upload_part from 512MB to 256MB,
Made sure that all files were copied down from Wasabi to my server and named correctly (each file did have to be renamed to a different guid),
Set the video’s state to 8 with UPDATE PUBLIC.video SET "state" = 8 WHERE uuid = '<uuid>';, and
That should complete successfully. If not, decrease the max_upload_part further and redo the loop.
videoStreamingPlaylist table updates
Even when that move video to object storage completed successfully, however, the videoStreamingPlaylist database table did not get updated with the new locations, and the storage column did not flip to « 1 » to reflect the object storage status. I honestly don’t know why. It seems like this is or should be part of the « move to object storage » job, but I’d rerun that so many times that I don’t think I could provide reliable steps for a bug repro.
Using the video’s ID, I ran a query like this to see this to see the state of the playlist URL and segments file: SELECT * FROM PUBLIC."videoStreamingPlaylist" WHERE "storage" != 0 OR "videoId" = <videoid>; I found that the values for my broken video were still set to « irrsinn.video » (non-object storage) URLs and the storage column was 0 instead of 1. Including my other videos just let me have some working examples to borrow from.
I made sure that the master.m3u8 and segments-sha256.json files in my object storage location were correctly named. Mine used those base names, but they need a GUID in front of them. I think I pulled that from the original results of the above query.
Then I ran a query like this to update the record for the video to use object storage:
UPDATE PUBLIC."videoStreamingPlaylist"
SET
"playlistUrl" = 'https://<object_storage_domain>/streaming-playlists/hls/<video_uuid>/<playlist_uuid>-master.m3u8',
"segmentsSha256Url" = 'https://<object_storage_domain>/streaming-playlists/hls/<video_uuid>/<segments_uuid>-segments-sha256.json'
WHERE "videoId" = <videoid>;
And that got the video in the state I wanted it in.
Moving the 360p mp4 and m3u8 files let the transcoding complete, but the move to object storage failed with the error message « job stalled more than allowable limit ». Unfortunately, this hung the video not in the