Hi,
After upgrading to Peertube 6.0.2, only the new video upload occuring after the upgrade will trigger the generation of storyboard thumbnail.
With the following command, I can force the generation, but I have to specify the video (by the way, how ? with the UUID ?) :
npm run create-generate-storyboard-job
Is there a command to force the generation for ALL videos ?
Thanks for your help.
You can use the --all-videos option.
Hi tcit, and thanks,
I had searched the doc before it was updated (or I’m totally blind 
)
Thanks again !
After running the command, Peertube successfully complete 4 or 5 tasks, but after that the next one fail.
Here are for exemple the log of one of this failure :
Job: 382
Type: generate-video-storyboard
Processed on 08/01/24 14:02:23,597
Finished on 08/01/24 14:02:23,701
{
"videoUUID": "7642a0c7-b762-4e7a-8073-ff2958816af8",
"federate": true
}
NoSuchKey: The specified key does not exist.
at de_NoSuchKeyRes (/var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:6082:23)
at de_GetObjectCommandError (/var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4327:25)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:57:20
at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
at async makeAvailable (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/object-storage/shared/object-storage-helpers.js:110:22)
at async makeHLSFileAvailable (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/object-storage/videos.js:76:5)
at async VideoPathManager.makeAvailableFactory (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/video-path-manager.js:82:29)
at async Object.processGenerateStoryboard [as generate-video-storyboard] (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/job-queue/handlers/generate-storyboard.js:29:9)
at async Object.wrapPromiseFun (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/plugins/hooks.js:8:24)
at async Worker.processJob (/var/www/peertube/versions/peertube-v6.0.2/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
at async Worker.retryIfFailed (/var/www/peertube/versions/peertube-v6.0.2/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
Any clue in it ? Thanks for your advice.
You only have 1 video where the storyboard is not generated?
Can you check this video is broken? Seems like peertube think it’s in object storage
Hi Chocobozz, no, when I (to be honnest my colleague) run the command, we get 5 videos successfully updated with their storyboard’s thumbnails (the 5 top lines of my screencapture below, the last 2 are related to a video I uploaded manually just before, and then replace manually to test the new replace feature of 6.0.2) :
For those 5 top lines, I’ve checked and the storyboard’s thumbnail show up as expected.
And if I go back to the log, after this 5 first successfull tasks, the next tasks start failing :
but Peertube seems to only try those 10 failing tasks (from n°382 to 391) and then stop trying to handle the other videos (I’ve around 300 videos on this test instance, and only 2 uploaded since the update from 5.2.1 to 6.0.2).
And the error in the log are the same for the 10 failure : « NoSuchKey: The specified key does not exist… »
…and to answer your question Chocobozzz, I’ve checked one of the failed video, and it isn’t broken, I can play it as expected, and there is indeed no storyboard thumbnails at all (not even a black frame).
Can you paste the video URL?
corresponding to this log :
and another working video (with storyboard successfully generated by the command) :
Thanks, can you also provide peertube error logs corresponding to the job error? We may have more information inside.
In the WebUI I don’t see anything more (there is a dozen of similar error for a dozen of failed job, even if there is hundreds of videos without storyboard on this test instance).
My colleague developper just launch again the command for all vidos, and he gets the same error. Here are the logs he sends to me :
{"level":"info","message":"Processing generate storyboard of 7642a0c7-b762-4e7a-8073-ff2958816af8 in job 770.","label":"tubetest.reseau-canope.fr:443","tags":["storyboard","7642a0c7-b762-4e7a-8073-ff2958816af8"],"timestamp":"2024-01-10T09:41:30.809Z"}
{"level":"info","message":"Fetching HLS file hls/7642a0c7-b762-4e7a-8073-ff2958816af8/df5003ba-de16-4470-8822-f7f23fe324b0-480-fragmented.mp4 from object storage to /var/www/peertube/storage/tmp/376e7be8-aae9-4936-899a-a528e592757d.mp4.","label":"tubetest.reseau-canope.fr:443","tags":["object-storage"],"timestamp":"2024-01-10T09:41:30.831Z"}
{"level":"error","message":"Cannot execute job 770 in queue generate-video-storyboard.","label":"tubetest.reseau-canope.fr:443","payload":{"videoUUID":"7642a0c7-b762-4e7a-8073-ff2958816af8","federate":true},"err":{"stack":"NoSuchKey: The specified key does not exist.\n at de_NoSuchKeyRes (/var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:6082:23)\n at de_GetObjectCommandError (/var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4327:25)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20\n at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46\n at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:57:20\n at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24\n at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20\n at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26\n at async makeAvailable (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/object-storage/shared/object-storage-helpers.js:110:22)\n at async makeHLSFileAvailable (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/object-storage/videos.js:76:5)\n at async VideoPathManager.makeAvailableFactory (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/video-path-manager.js:82:29)\n at async Object.processGenerateStoryboard [as generate-video-storyboard] (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/job-queue/handlers/generate-storyboard.js:29:9)\n at async Object.wrapPromiseFun (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/plugins/hooks.js:8:24)\n at async Worker.processJob (/var/www/peertube/versions/peertube-v6.0.2/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)\n at async Worker.retryIfFailed (/var/www/peertube/versions/peertube-v6.0.2/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)","name":"NoSuchKey","$fault":"client","$metadata":{"httpStatusCode":404,"requestId":"txe40d048ac21e4fa2b67de-00659e664a","extendedRequestId":"txe40d048ac21e4fa2b67de-00659e664a","attempts":1,"totalRetryDelay":0},"Code":"NoSuchKey","RequestId":"txe40d048ac21e4fa2b67de-00659e664a","Key":"streaming-playlistshls/7642a0c7-b762-4e7a-8073-ff2958816af8/df5003ba-de16-4470-8822-f7f23fe324b0-480-fragmented.mp4","message":"The specified key does not exist.","$response":{"statusCode":404,"reason":"Not Found","headers":{"content-type":"application/xml","x-amz-id-2":"txe40d048ac21e4fa2b67de-00659e664a","x-amz-request-id":"txe40d048ac21e4fa2b67de-00659e664a","x-trans-id":"txe40d048ac21e4fa2b67de-00659e664a","x-openstack-request-id":"txe40d048ac21e4fa2b67de-00659e664a","date":"Wed, 10 Jan 2024 09:41:30 GMT","transfer-encoding":"chunked","connection":"keep-alive"},"body":{"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":null,"tail":null,"length":0},"length":0,"pipes":[],"flowing":false,"ended":true,"endEmitted":true,"reading":false,"constructed":true,"sync":false,"needReadable":false,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"errorEmitted":false,"emitClose":true,"autoDestroy":true,"destroyed":true,"errored":null,"closed":true,"closeEmitted":true,"defaultEncoding":"utf8","awaitDrainWriters":null,"multiAwaitDrain":false,"readingMore":false,"dataEmitted":true,"decoder":null,"encoding":null},"_events":{},"_eventsCount":2,"socket":null,"httpVersionMajor":1,"httpVersionMinor":1,"httpVersion":"1.1","complete":true,"rawHeaders":["Content-Type","application/xml","x-amz-id-2","txe40d048ac21e4fa2b67de-00659e664a","x-amz-request-id","txe40d048ac21e4fa2b67de-00659e664a","X-Trans-Id","txe40d048ac21e4fa2b67de-00659e664a","X-Openstack-Request-Id","txe40d048ac21e4fa2b67de-00659e664a","Date","Wed, 10 Jan 2024 09:41:30 GMT","Transfer-Encoding","chunked","Connection","keep-alive"],"rawTrailers":[],"aborted":false,"upgrade":false,"url":"","method":null,"statusCode":404,"statusMessage":"Not Found","client":
Thanks, from the error log I can see PeerTube tries to get the following object storage key: streaming-playlistshls/a97f16df-ddb4-45e7-9db8-e9de81e47e01/a97f16df-ddb4-45e7-9db8-e9de81e47e01-720-fragmented.mp4
From what I understand you have set prefix: 'streaming-playlist' in your configuration, but from the HTTP request I see in my web browser the video doesn’t seem to have an object storage prefix (https://s3.reseau-canope.fr/peertubetest/hls/a97f16df-ddb4-45e7-9db8-e9de81e47e01/a97f16df-ddb4-45e7-9db8-e9de81e47e01-720-fragmented.mp4 instead of https://s3.reseau-canope.fr/peertubetest/streaming-playlistshls/a97f16df-ddb4-45e7-9db8-e9de81e47e01/a97f16df-ddb4-45e7-9db8-e9de81e47e01-720-fragmented.mp4)
I’m not sure to understand this prefix info (I’ll pass your answer to my colleague).
But I’m wondering why the command works for some video, and not for other’s one.
Could it be that the difference comes from the instant we move to object storage and fix this prefix ? (and for example the working video is one uploaded before or after this instant, and a contrario for the failing one ?)
Yes it is: the video that works has been uploaded with your the prefix defined in your current configuration whereas the video that doesn’t work has not. Since your current configuration contains the prefix, PeerTube cannot access videos that were uploaded without the prefix. So you changed your S3 configuration but did not migrate S3 objects.
Ok, so if we remove temporarly the prefix in the config file, could we run the command and get the problematic videos treated ? And then put the prefix back, and rerun the command for the other videos ?
And then after that, every new video uploaded in the future will be correctly treated too ?
Yes I think it would work, but IMHO you should just fix objects location in S3 directly (moving old videos to the new prefix) and update PeerTube DB URLs
Here is the files on S3 :
A : streaming-playlistshls
The actual prefix set in the config.
This folder contains a minority of videos (because peertubetest is our test instance, and I upload only a feww files at each update to test new features)
/streaming-playlistshls/ Server-side modified: 07/04/2023 - 11/01/2024
From this timestamp, I assume this folder is posterior at the S3 migration.
B : streaming-playlistshls:
If I remember, it was the default prefix in the config file. My developper colleague choose to delete the colon at the end.
(it contains maybe a few file, I forgot to check)
C : hls
The majority of our files is in the hls folder
/hls/ Server-side modified: 09/03/2023 - 24/03/2023
From this timestamp, I assume this is the moment we migrate to S3.
D
Do you know why there is some files at the first level ?
So to be sure to understand what to do to correct this, your advice is to :
move all files (from B, C and D) inside A folder (the actual prefix). (could I delete the empty folders after the copy ?)
update PeerTube DB URLs (could you tell me how to do this step ? I guess a command to pass on database, but I can’t find anything in the « CLI tools » section of the documentation ?)
Is that right ?
Thanks.
hi @Chocobozzz , do you think these sql query can do the job :
UPDATE videoStreamingPlaylist
SET playlistUrl = REPLACE(playlistUrl, ‹ /hls/ ›, ‹ /streaming-playlistshls/ ›)
WHERE playlistUrl LIKE ‹ %/hls/% ›;
and this one :
UPDATE videoStreamingPlaylist
SET segmentsSha256Url = REPLACE(segmentsSha256Url, ‹ /hls/ ›, ‹ /streaming-playlistshls/ ›)
WHERE segmentsSha256Url LIKE ‹ %/hls/% ›;
and :
UPDATE « videoFile »
SET « fileUrl » = REPLACE(« fileUrl », ‹ /hls/ ›, ‹ /streaming-playlistshls/ ›)
WHERE « fileUrl » like ‹ %/hls/% ›;
Then we have to move all video s3 folder from /hls/ to /streaming-playlisthls/
and correct the acl on each video file for public read (acl are not copied when we copy or move object)
Thanks for your help.