To close the loop on this, this is fully resolved, and did involve some manual database work, unfortunately. I do not know when and how this video broke, but it must have been a combination of upgrades, old object storage configuration issues, plus the switch to the new object storage methods.
The failure state of the move to object storage job was cleaned up in the fix to the bug I logged, although I didn’t pull that code down.
So here’s roughly what I did after I got all resolutions transcoded, in case someone else finds themself in this sad situation. I do not recommend digging into the database unless you’re comfortable doing so. (My experience is in mySQL and SQL Server, so forgive me for the desperate and inconsistent column quoting here; postgresql is weird in what it forced me to quote.)
The repeated « job stalled more than the allowable limit » loop
- Decreased my
object_storage.max_upload_part
from 512MB to 256MB, - Made sure that all files were copied down from Wasabi to my server and named correctly (each file did have to be renamed to a different guid),
- Set the video’s
state
to 8 withUPDATE PUBLIC.video SET "state" = 8 WHERE uuid = '<uuid>';
, and - Queued up a create-move-video-storage-job.js for the video.
That should complete successfully. If not, decrease the max_upload_part
further and redo the loop.
videoStreamingPlaylist
table updates
Even when that move video to object storage completed successfully, however, the videoStreamingPlaylist
database table did not get updated with the new locations, and the storage
column did not flip to « 1 » to reflect the object storage status. I honestly don’t know why. It seems like this is or should be part of the « move to object storage » job, but I’d rerun that so many times that I don’t think I could provide reliable steps for a bug repro.
- Using the video’s ID, I ran a query like this to see this to see the state of the playlist URL and segments file:
SELECT * FROM PUBLIC."videoStreamingPlaylist" WHERE "storage" != 0 OR "videoId" = <videoid>;
I found that the values for my broken video were still set to « irrsinn.video » (non-object storage) URLs and thestorage
column was 0 instead of 1. Including my other videos just let me have some working examples to borrow from. - I made sure that the
master.m3u8
andsegments-sha256.json
files in my object storage location were correctly named. Mine used those base names, but they need a GUID in front of them. I think I pulled that from the original results of the above query. - Then I ran a query like this to update the record for the video to use object storage:
UPDATE PUBLIC."videoStreamingPlaylist"
SET
"playlistUrl" = 'https://<object_storage_domain>/streaming-playlists/hls/<video_uuid>/<playlist_uuid>-master.m3u8',
"segmentsSha256Url" = 'https://<object_storage_domain>/streaming-playlists/hls/<video_uuid>/<segments_uuid>-segments-sha256.json'
WHERE "videoId" = <videoid>;
And that got the video in the state I wanted it in.