Newly transcoded videos stuck in the state of "to move to an external storage"

TankieTanuki · Octobre 23, 2024, 3:46

Transcoded videos on my instance are typically transferred to object storage within a couple minutes.

However, since about a day ago, all transcoded videos seem to be stuck in the « to move » state.

I tried to force one of the videos to move by issuing a CLI command, but it simply said This video is already being moved to external storage/file system.

There doesn’t seem to be a corresponding error in the logs. It just never attempted to move them.

Chocobozzz · Octobre 23, 2024, 5:19

Hi,

Please enable debug logs and paste the output of move-object-to-storage jobs stuck until Peertube restarted · Issue #6686 · Chocobozzz/PeerTube · GitHub after a video upload

somebody · Octobre 24, 2024, 5:14

Hi. When the issue happen, does a Peertube restart fix those videos? Or are they still stuck?

TankieTanuki · Octobre 31, 2024, 12:11

Yes, a restart fixed them.

somebody · Novembre 1, 2024, 4:23

probably same issue as this?

github.com/Chocobozzz/PeerTube

move-object-to-storage jobs stuck until Peertube restarted

opened 02:45AM - 22 Oct 24 UTC

dausruddin

Status: Waiting for answer

### Describe the current behavior Sometimes, unpredictable, `move-to-object-sto…rage` job got stuck, then new jobs after that will just queuing indefinitely until I restarted Peertube. ![image](https://github.com/user-attachments/assets/196a25f6-7eee-48e2-84c7-620a094c7791) This is the log for job 8514: ``` Job: 8514 Type: move-to-object-storage Processed on 10/22/24, 5:57:36.476 AM Finished on - { "videoUUID": "f0a01bf5-f9da-4ad3-9362-01c90232e83e", "isNewVideo": true, "previousVideoState": 2 } Error: Timeout at Timeout.<anonymous> (file:///var/www/peertube/versions/peertube-v6.3.2/packages/core-utils/dist/common/promises.js:12:42) at listOnTimeout (node:internal/timers:581:17) at process.processTimers (node:internal/timers:519:7) ``` ### Steps to reproduce 1. Keep uploading once every few hours until the issue happen 2. Restart Peertube to fix the issue temporarily, until it happens again ### Describe the expected behavior `move-to-object-storage` successfully executed, and video published. ### Additional information * PeerTube instance: * URL: private * Version: 6.3.2 * NodeJS version: v20.18.0 * Ffmpeg version: 5.1.6-0+deb12u1 * Browser name, version and platforms on which you could reproduce the bug: Firefox 131.0.2 (64-bit), ArchLinux * Link to browser console log if relevant: N/A * Link to server log if relevant (`journalctl` or `/var/www/peertube/storage/logs/`): N/A

fediversetv · Juillet 2, 2025, 6:47

I have a similar problem. In the videos section, some videos appear with the label « To move to an external storage », but several days have passed and they still haven’t been moved to S3.

When checking the logs, I found this error:

Job: 5917
Type: move-to-object-storage
Processed on 7/1/25, 6:23:45.517 PM
Finished on 7/1/25, 6:23:45.665 PM

{
  "videoUUID": "6e75fcf1-a138-4b49-b067-9da7a8898fd9",
  "isNewVideo": true,
  "previousVideoState": 2
}

InvalidArgument: Unsupported value for canned acl 'private'
    at throwDefaultError (/app/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
    ...

After researching in the forum and checking some issues, it seems that the « Unsupported value for canned acl ‹ private › » error is related to the upload_acl configuration. Currently, I have it set like this:

upload_acl:
  # ACL for public/unlisted videos
  public: null
  # ACL for private/internal videos
  private: null

I’ve read in some threads that it’s recommended to set both upload_acl.public and upload_acl.private to null if your S3 provider does not support object ACLs, which seems to be my case. However, all the jobs that fail show the same error: « Unsupported value for canned acl ‹ private › ».

My questions are:

Should I explicitly set upload_acl.private to null, or just leave it undefined?
Why do other videos get moved to S3 without any issues, and only some of them fail? Could it be because those videos are set as private?

If anyone has encountered the same problem and found a solution, I’d really appreciate it if you could share your experience. Thanks!

Chocobozzz · Juillet 5, 2025, 1:04

Your configuration should work

How did you deploy peertube?

fediversetv · Juillet 5, 2025, 2:07

We using PeerTube with Docker and docker-compose. If you need more details, please let me know. Our current configuration works correctly: when a video is uploaded, it gets transcoded and then successfully uploaded to S3 storage without issues.

However, we have noticed that some videos are not being uploaded.

We have tried running the create-move-video-storage-job to regenerate the job and move the video, but we get the following message:

« This video is already being moved to external storage/file system »

Despite this, the video does not have the « object storage » tag and still shows the « To move to an external storage » tag.

Additionally, we have observed that videos uploaded as private are never moved to S3 and always remain with the « To move to an external storage » tag and the job in failed state.

Is our understanding correct that videos marked as private are never uploaded to S3? Could you please clarify this behavior in more detail?

Finally, is there an option to disable private video uploads for users? We need to ensure that absolutely all content is uploaded to S3 and nothing remains on the local server disk.

Chocobozzz · Juillet 5, 2025, 2:19

If you use Docker, you must comment these env variables: PeerTube/support/docker/production/.env at develop · Chocobozzz/PeerTube · GitHub

fediversetv · Juillet 5, 2025, 2:33

AAAAAH Damn it! That was exactly it. Thanks for everything! Now it looks like everything is being uploaded correctly.

One question: for the videos that haven’t been uploaded to object storage yet and are still pending, will they eventually be uploaded automatically, or do we need to do something manually on our side?