Transcoding of big videos is failing

PeerTube
1

Hi, a user uploaded a 60GB video file and the transcoding job is always failing. I see ffmpeg finishing in the external runner but after that the jobs is failing with:

superagent: double callback bug
[17:48:33.453] ERROR (29): Cannot process job
    err: {
      "type": "Error",
      "message": "read ECONNRESET",
      "stack":
          Error: read ECONNRESET
              at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
      "errno": -104,
      "code": "ECONNRESET",
      "syscall": "read"
    }

On Peertube side I only see

[makertube.net:443] 2025-03-26 17:48:33.465 error: Remote runner tube1 had an error with job 8e32f492-3929-4921-9dd5-0808a807182b (vod-hls-transcoding) {
  "errorMessage": "read ECONNRESET",
  "totalFailures": 1
}

Is there something I can tweak? Maybe some timeout config? Thanks!

2

The shell command before was:

[17:42:51.060] DEBUG (29): FFmpeg command ended.
    stdout: ""
    stderr: "frame=731672 fps= 29 q=36.0 size=N/A time=03:23:14.62 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731683 fps= 29 q=39.0 size=N/A time=03:23:15.00 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731701 fps= 29 q=37.0 size=N/A time=03:23:15.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731713 fps= 29 q=36.0 size=N/A time=03:23:15.37 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731726 fps= 29 q=39.0 size=N/A time=03:23:15.62 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731739 fps= 29 q=35.0 size=N/A time=03:23:15.62 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731748 fps= 29 q=37.0 size=N/A time=03:23:16.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731762 fps= 29 q=36.0 size=N/A time=03:23:16.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731773 fps= 29 q=36.0 size=N/A time=03:23:16.16 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731789 fps= 29 q=37.0 size=N/A time=03:23:16.62 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731803 fps= 29 q=36.0 size=N/A time=03:23:16.62 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731814 fps= 29 q=36.0 size=N/A time=03:23:17.14 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731831 fps= 29 q=37.0 size=N/A time=03:23:17.14 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731845 fps= 29 q=38.0 size=N/A time=03:23:17.63 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731857 fps= 29 q=37.0 size=N/A time=03:23:17.63 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731874 fps= 29 q=36.0 size=N/A time=03:23:18.12 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731892 fps= 29 q=35.0 size=N/A time=03:23:18.12 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731909 fps= 29 q=35.0 size=N/A time=03:23:18.63 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731923 fps= 29 q=34.0 size=N/A time=03:23:18.76 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731942 fps= 29 q=35.0 size=N/A time=03:23:19.12 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731960 fps= 29 q=35.0 size=N/A time=03:23:19.63 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731977 fps= 29 q=35.0 size=N/A time=03:23:19.63 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=731997 fps= 29 q=37.0 size=N/A time=03:23:20.12 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732013 fps= 29 q=38.0 size=N/A time=03:23:20.44 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732038 fps= 29 q=36.0 size=N/A time=03:23:20.64 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732055 fps= 29 q=36.0 size=N/A time=03:23:21.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732073 fps= 29 q=34.0 size=N/A time=03:23:21.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732087 fps= 29 q=35.0 size=N/A time=03:23:21.64 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732096 fps= 29 q=34.0 size=N/A time=03:23:21.64 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732111 fps= 29 q=31.0 size=N/A time=03:23:22.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732124 fps= 29 q=31.0 size=N/A time=03:23:22.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732140 fps= 29 q=31.0 size=N/A time=03:23:22.64 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732157 fps= 29 q=31.0 size=N/A time=03:23:22.64 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732174 fps= 29 q=31.0 size=N/A time=03:23:23.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732189 fps= 29 q=31.0 size=N/A time=03:23:23.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732208 fps= 29 q=36.0 size=N/A time=03:23:23.64 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732229 fps= 29 q=34.0 size=N/A time=03:23:24.13 bitrate=N/A dup=731 drop=0 speed=0.486x    \nframe=732251 fps= 29 q=33.0 size=N/A time=03:23:24.13 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732271 fps= 29 q=34.0 size=N/A time=03:23:24.62 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732289 fps= 29 q=33.0 size=N/A time=03:23:25.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732306 fps= 29 q=33.0 size=N/A time=03:23:25.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732319 fps= 29 q=33.0 size=N/A time=03:23:25.63 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732336 fps= 29 q=33.0 size=N/A time=03:23:25.63 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732351 fps= 29 q=34.0 size=N/A time=03:23:26.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732372 fps= 29 q=34.0 size=N/A time=03:23:26.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732390 fps= 29 q=33.0 size=N/A time=03:23:26.63 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732404 fps= 29 q=34.0 size=N/A time=03:23:26.99 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732426 fps= 29 q=34.0 size=N/A time=03:23:27.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732444 fps= 29 q=36.0 size=N/A time=03:23:27.63 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732464 fps= 29 q=35.0 size=N/A time=03:23:27.63 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732483 fps= 29 q=34.0 size=N/A time=03:23:28.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732500 fps= 29 q=35.0 size=N/A time=03:23:28.64 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732521 fps= 29 q=36.0 size=N/A time=03:23:28.64 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732540 fps= 29 q=36.0 size=N/A time=03:23:29.15 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732560 fps= 29 q=36.0 size=N/A time=03:23:29.64 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732577 fps= 29 q=36.0 size=N/A time=03:23:29.64 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732596 fps= 29 q=37.0 size=N/A time=03:23:30.15 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732611 fps= 29 q=38.0 size=N/A time=03:23:30.15 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732625 fps= 29 q=36.0 size=N/A time=03:23:30.64 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732642 fps= 29 q=37.0 size=N/A time=03:23:30.64 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732657 fps= 29 q=38.0 size=N/A time=03:23:31.13 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732674 fps= 29 q=34.0 size=N/A time=03:23:31.13 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732689 fps= 29 q=37.0 size=N/A time=03:23:31.64 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732701 fps= 29 q=38.0 size=N/A time=03:23:31.64 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732717 fps= 29 q=36.0 size=N/A time=03:23:32.13 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732729 fps= 29 q=37.0 size=N/A time=03:23:32.13 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732742 fps= 29 q=34.0 size=N/A time=03:23:32.65 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732756 fps= 29 q=34.0 size=N/A time=03:23:32.65 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732764 fps= 29 q=34.0 size=N/A time=03:23:33.01 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732781 fps= 29 q=34.0 size=N/A time=03:23:33.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732794 fps= 29 q=35.0 size=N/A time=03:23:33.16 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732809 fps= 29 q=36.0 size=N/A time=03:23:33.65 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732821 fps= 29 q=36.0 size=N/A time=03:23:33.65 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732834 fps= 29 q=34.0 size=N/A time=03:23:34.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732852 fps= 29 q=35.0 size=N/A time=03:23:34.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732867 fps= 29 q=33.0 size=N/A time=03:23:34.65 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732881 fps= 29 q=33.0 size=N/A time=03:23:34.65 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732892 fps= 29 q=36.0 size=N/A time=03:23:35.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732909 fps= 29 q=33.0 size=N/A time=03:23:35.14 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732924 fps= 29 q=37.0 size=N/A time=03:23:35.48 bitrate=N/A dup=732 drop=0 speed=0.486x    \nframe=732932 fps= 29 q=-1.0 Lsize=N/A time=03:23:35.55 bitrate=N/A dup=732 drop=0 speed=0.486x    \nvideo:18242042kB audio:351984kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown\n[libx264 @ 0x55c212cb5ac0] frame I:6134  Avg QP:32.47  size:251362\n[libx264 @ 0x55c212cb5ac0] frame P:218230 Avg QP:35.30  size: 58504\n[libx264 @ 0x55c212cb5ac0] frame B:508568 Avg QP:38.39  size:  8594\n[libx264 @ 0x55c212cb5ac0] consecutive B-frames:  5.9% 14.5% 14.7% 27.4% 11.2% 12.2%  2.7%  2.3%  2.2%  1.2%  1.0%  0.8%  0.6%  0.6%  0.8%  0.2%  1.7%\n[libx264 @ 0x55c212cb5ac0] mb I  I16..4:  5.1% 29.7% 65.3%\n[libx264 @ 0x55c212cb5ac0] mb P  I16..4:  1.8%  9.2%  1.6%  P16..4: 35.0% 23.9% 15.7%  0.0%  0.0%    skip:12.7%\n[libx264 @ 0x55c212cb5ac0] mb B  I16..4:  0.3%  1.1%  0.0%  B16..8: 16.9%  5.8%  0.5%  direct: 3.4%  skip:72.0%  L0:31.5% L1:42.8% BI:25.7%\n[libx264 @ 0x55c212cb5ac0] 8x8 transform intra:67.2% inter:34.9%\n[libx264 @ 0x55c212cb5ac0] coded y,uvDC,uvAC intra: 71.8% 37.8% 5.6% inter: 12.1% 3.0% 0.0%\n[libx264 @ 0x55c212cb5ac0] i16 v,h,dc,p: 28% 22% 36% 15%\n[libx264 @ 0x55c212cb5ac0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 14% 25%  7%  8%  7%  8%  6%  9%\n[libx264 @ 0x55c212cb5ac0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 13% 14% 16%  9% 11%  8% 11%  7% 10%\n[libx264 @ 0x55c212cb5ac0] i8c dc,h,v,p: 73% 13% 11%  3%\n[libx264 @ 0x55c212cb5ac0] Weighted P-Frames: Y:11.9% UV:2.4%\n[libx264 @ 0x55c212cb5ac0] kb/s:12233.51\n[aac @ 0x55c212cb49c0] Qavg: 65535.629\n"
    shellCommand: "ffmpeg -n 19 /usr/bin/ffmpeg -i /root/.cache/peertube-runner-nodejs/default/transcoding/07aae8f7-9255-4518-9a67-bdaf2cfded1c -y -acodec aac -vcodec libx264 -threads 4 -f mp4 -movflags faststart -max_muxing_queue_size 1024 -map_metadata -1 -pix_fmt yuv420p -channel_layout stereo -b:a 384k -vf scale=w=-2:h=1080 -preset veryfast -maxrate:v 12441755 -bufsize:v 24883510 -b_strategy 1 -bf 16 -r 60 -g:v 120 -hls_time 4 -hls_list_size 0 -hls_playlist_type vod -hls_segment_filename /root/.cache/peertube-runner-nodejs/default/transcoding/0cd1bc61-a3cf-4881-9cd8-3717641c08bf-1080-fragmented.mp4 -hls_segment_type fmp4 -f hls -hls_flags single_file /root/.cache/peertube-runner-nodejs/default/transcoding/0cd1bc61-a3cf-4881-9cd8-3717641c08bf-1080.m3u8"

Update: or is this a problem on the video file after all?

3

Hi,

Can you check on the server if the reverse proxy or PeerTube closed the HTTP connection because of a timeout?

4

I thought of this as well but I’m going directly to the service endpoint in Kubernetes. Peertube and the runner are on the same node. There should be no reverse proxy in the way.

5

Can you paste logs of server between 17:42:51 and 17:48:33?

6

I only found this, looks like the times are adjusted:

error[3/26/2025, 6:41:52 PM] Error in controller.

{
  "err": {
    "stack": "Error: Request aborted\n    at onaborted (/app/node_modules/express/lib/response.js:1062:15)\n    at Immediate._onImmediate (/app/node_modules/express/lib/response.js:1104:9)\n    at process.processImmediate (node:internal/timers:483:21)",
    "message": "Request aborted",
    "code": "ECONNABORTED"
  },
  "url": "/lazy-static/thumbnails/0587a653-b17f-4b89-bac6-f1e33cd00cfc.jpg"
}

error[3/26/2025, 6:43:55 PM] Error in controller.

{
  "err": {
    "stack": "Error: Request aborted\n    at onaborted (/app/node_modules/express/lib/response.js:1062:15)\n    at Immediate._onImmediate (/app/node_modules/express/lib/response.js:1104:9)\n    at process.processImmediate (node:internal/timers:483:21)",
    "message": "Request aborted",
    "code": "ECONNABORTED"
  },
  "url": "/lazy-static/avatars/a2f897a0-c0bf-4ed4-abad-ef01697d84ee.jpg"
}

warn[3/26/2025, 6:44:55 PM] Incorrect request parameters

{
  "path": "/.well-known/webfinger?resource=acct:peertube@www.makertube.net",
  "err": {
    "resource": {
      "type": "field",
      "value": "acct:peertube@www.makertube.net",
      "msg": "Invalid value",
      "path": "resource",
      "location": "query"
    }
  }
}

warn[3/26/2025, 6:45:01 PM] Cannot execute with many retries.

{
  "err": {
    "stack": "Error\n    at Query.run (/app/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)\n    at /app/node_modules/sequelize/lib/sequelize.js:315:28\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryInterface.select (/app/node_modules/sequelize/lib/dialects/abstract/query-interface.js:407:12)\n    at async RunnerJobModel.findAll (/app/node_modules/sequelize/lib/model.js:1140:21)\n    at async RunnerJobModel.findOne (/app/node_modules/sequelize/lib/model.js:1240:12)\n    at async self.reload (/app/node_modules/sequelize/lib/model.js:2562:22)\n    at async file:///app/dist/core/controllers/api/runners/jobs.js:47:13\n    at async /app/node_modules/sequelize/lib/sequelize.js:507:18",
    "message": "could not serialize access due to read/write dependencies among transactions",
    "name": "SequelizeDatabaseError",
    "parent": {
      "stack": "error: could not serialize access due to read/write dependencies among transactions\n    at Parser.parseErrorMessage (/app/node_modules/pg-protocol/dist/parser.js:283:98)\n    at Parser.handlePacket (/app/node_modules/pg-protocol/dist/parser.js:122:29)\n    at Parser.parse (/app/node_modules/pg-protocol/dist/parser.js:35:38)\n    at TLSSocket.<anonymous> (/app/node_modules/pg-protocol/dist/index.js:11:42)\n    at TLSSocket.emit (node:events:524:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Readable.push (node:internal/streams/readable:392:5)\n    at TLSWrap.onStreamRead (node:internal/stream_base_commons:191:23)",
      "message": "could not serialize access due to read/write dependencies among transactions",
      "length": 278,
      "name": "error",
      "severity": "ERROR",
      "code": "40001",
      "detail": "Reason code: Canceled on conflict out to pivot 35314482, during read.",
      "hint": "The transaction might succeed if retried.",
      "file": "predicate.c",
      "line": "4614",
      "routine": "OnConflict_CheckForSerializationFailure",
      "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51607;"
    },
    "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51607;",
    "parameters": {}
  }
}

error[3/26/2025, 6:45:01 PM] Error in controller.

{
  "err": {
    "stack": "Error\n    at Query.run (/app/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)\n    at /app/node_modules/sequelize/lib/sequelize.js:315:28\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryInterface.select (/app/node_modules/sequelize/lib/dialects/abstract/query-interface.js:407:12)\n    at async RunnerJobModel.findAll (/app/node_modules/sequelize/lib/model.js:1140:21)\n    at async RunnerJobModel.findOne (/app/node_modules/sequelize/lib/model.js:1240:12)\n    at async self.reload (/app/node_modules/sequelize/lib/model.js:2562:22)\n    at async file:///app/dist/core/controllers/api/runners/jobs.js:47:13\n    at async /app/node_modules/sequelize/lib/sequelize.js:507:18",
    "message": "could not serialize access due to read/write dependencies among transactions",
    "name": "SequelizeDatabaseError",
    "parent": {
      "stack": "error: could not serialize access due to read/write dependencies among transactions\n    at Parser.parseErrorMessage (/app/node_modules/pg-protocol/dist/parser.js:283:98)\n    at Parser.handlePacket (/app/node_modules/pg-protocol/dist/parser.js:122:29)\n    at Parser.parse (/app/node_modules/pg-protocol/dist/parser.js:35:38)\n    at TLSSocket.<anonymous> (/app/node_modules/pg-protocol/dist/index.js:11:42)\n    at TLSSocket.emit (node:events:524:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Readable.push (node:internal/streams/readable:392:5)\n    at TLSWrap.onStreamRead (node:internal/stream_base_commons:191:23)",
      "message": "could not serialize access due to read/write dependencies among transactions",
      "length": 278,
      "name": "error",
      "severity": "ERROR",
      "code": "40001",
      "detail": "Reason code: Canceled on conflict out to pivot 35314482, during read.",
      "hint": "The transaction might succeed if retried.",
      "file": "predicate.c",
      "line": "4614",
      "routine": "OnConflict_CheckForSerializationFailure",
      "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51607;"
    },
    "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51607;",
    "parameters": {}
  },
  "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51607;",
  "url": "/api/v1/runners/jobs/23b4213c-a40f-4244-b8ba-b3709733072c/accept"
}

warn[3/26/2025, 6:47:22 PM] Cannot execute with many retries.

{
  "err": {
    "stack": "Error\n    at Query.run (/app/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)\n    at /app/node_modules/sequelize/lib/sequelize.js:315:28\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryInterface.insert (/app/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n    at async RunnerJobModel.save (/app/node_modules/sequelize/lib/model.js:2490:35)\n    at async file:///app/dist/core/helpers/database-utils.js:33:17\n    at async /app/node_modules/sequelize/lib/sequelize.js:507:18",
    "message": "could not serialize access due to read/write dependencies among transactions",
    "name": "SequelizeDatabaseError",
    "parent": {
      "stack": "error: could not serialize access due to read/write dependencies among transactions\n    at Parser.parseErrorMessage (/app/node_modules/pg-protocol/dist/parser.js:283:98)\n    at Parser.handlePacket (/app/node_modules/pg-protocol/dist/parser.js:122:29)\n    at Parser.parse (/app/node_modules/pg-protocol/dist/parser.js:35:38)\n    at TLSSocket.<anonymous> (/app/node_modules/pg-protocol/dist/index.js:11:42)\n    at TLSSocket.emit (node:events:524:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Readable.push (node:internal/streams/readable:392:5)\n    at TLSWrap.onStreamRead (node:internal/stream_base_commons:191:23)",
      "message": "could not serialize access due to read/write dependencies among transactions",
      "length": 393,
      "name": "error",
      "severity": "ERROR",
      "code": "40001",
      "detail": "Reason code: Canceled on conflict out to pivot 35314914, during read.",
      "hint": "The transaction might succeed if retried.",
      "where": "SQL statement \"SELECT 1 FROM ONLY \"public\".\"runnerJob\" x WHERE \"id\" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x\"",
      "file": "predicate.c",
      "line": "4614",
      "routine": "OnConflict_CheckForSerializationFailure",
      "sql": "INSERT INTO \"runnerJob\" (\"id\",\"uuid\",\"type\",\"payload\",\"privatePayload\",\"state\",\"failures\",\"priority\",\"dependsOnRunnerJobId\",\"createdAt\",\"updatedAt\") VALUES (DEFAULT,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10) RETURNING \"id\",\"uuid\",\"type\",\"payload\",\"privatePayload\",\"state\",\"failures\",\"error\",\"priority\",\"processingJobToken\",\"progress\",\"startedAt\",\"finishedAt\",\"dependsOnRunnerJobId\",\"runnerId\",\"createdAt\",\"updatedAt\";",
      "parameters": [
        "84749b4a-804f-4f53-ae16-f00a8d213a9f",
        "vod-hls-transcoding",
        "{\"input\":{\"videoFileUrl\":\"https://makertube.net/api/v1/runners/jobs/84749b4a-804f-4f53-ae16-f00a8d213a9f/files/videos/484aaf3d-d348-4675-8c9c-d1897622ba09/max-quality\",\"separatedAudioFileUrl\":[]},\"output\":{\"resolution\":720,\"fps\":30,\"separatedAudio\":false}}",
        "{\"isNewVideo\":true,\"deleteWebVideoFiles\":false,\"videoUUID\":\"484aaf3d-d348-4675-8c9c-d1897622ba09\"}",
        5,
        0,
        252,
        51617,
        "2025-03-26 17:47:22.445 +00:00",
        "2025-03-26 17:47:22.465 +00:00"
      ]
    },
    "sql": "INSERT INTO \"runnerJob\" (\"id\",\"uuid\",\"type\",\"payload\",\"privatePayload\",\"state\",\"failures\",\"priority\",\"dependsOnRunnerJobId\",\"createdAt\",\"updatedAt\") VALUES (DEFAULT,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10) RETURNING \"id\",\"uuid\",\"type\",\"payload\",\"privatePayload\",\"state\",\"failures\",\"error\",\"priority\",\"processingJobToken\",\"progress\",\"startedAt\",\"finishedAt\",\"dependsOnRunnerJobId\",\"runnerId\",\"createdAt\",\"updatedAt\";"
  }
}

error[3/26/2025, 6:47:22 PM] Error in controller.

{
  "err": {
    "stack": "Error: ENOENT: no such file or directory, stat '/data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4'",
    "message": "ENOENT: no such file or directory, stat '/data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4'",
    "errno": -2,
    "code": "ENOENT",
    "syscall": "stat",
    "path": "/data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4",
    "expose": false,
    "statusCode": 404,
    "status": 404
  },
  "url": "/api/v1/runners/jobs/786eda35-9b73-4a1a-b2da-2dc9657b66ce/files/videos/484aaf3d-d348-4675-8c9c-d1897622ba09/max-quality"
}

error[3/26/2025, 6:47:22 PM] Error in ffmpeg.

{
  "err": {
    "stack": "Error: ffmpeg exited with code 1: /data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4: No such file or directory\n\n    at ChildProcess.<anonymous> (/app/node_modules/fluent-ffmpeg/lib/processor.js:180:22)\n    at ChildProcess.emit (node:events:524:28)\n    at ChildProcess._handle.onexit (node:internal/child_process:293:12)",
    "message": "ffmpeg exited with code 1: /data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4: No such file or directory\n"
  },
  "stdout": "",
  "stderr": "ffmpeg version 5.1.6-0+deb12u1 Copyright (c) 2000-2024 the FFmpeg developers\n  built with gcc 12 (Debian 12.2.0-14)\n  configuration: --prefix=/usr --extra-version=0+deb12u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared\n  libavutil      57. 28.100 / 57. 28.100\n  libavcodec     59. 37.100 / 59. 37.100\n  libavformat    59. 27.100 / 59. 27.100\n  libavdevice    59.  7.100 / 59.  7.100\n  libavfilter     8. 44.100 /  8. 44.100\n  libswscale      6.  7.100 /  6.  7.100\n  libswresample   4.  7.100 /  4.  7.100\n  libpostproc    56.  6.100 / 56.  6.100\n/data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4: No such file or directory\n",
  "shellCommand": "ffmpeg -n 10 /usr/bin/ffmpeg -i /data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4 -y -filter_complex setpts='N/FRAME_RATE/TB',select='isnan(prev_selected_t)+gte(t-prev_selected_t,2)',scale=192:108,tile=layout=11x9 -frames:v 1 -q:v 2 /data/storyboards/72e23439-40b8-441f-9445-0f51e872a37a.jpg",
  "tags": [
    "ffmpeg"
  ]
}

error[3/26/2025, 6:47:22 PM] Cannot execute job 18994 in queue generate-video-storyboard.

{
  "payload": {
    "videoUUID": "484aaf3d-d348-4675-8c9c-d1897622ba09",
    "federate": true
  },
  "err": {
    "stack": "Error: ffmpeg exited with code 1: /data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4: No such file or directory\n\n    at ChildProcess.<anonymous> (/app/node_modules/fluent-ffmpeg/lib/processor.js:180:22)\n    at ChildProcess.emit (node:events:524:28)\n    at ChildProcess._handle.onexit (node:internal/child_process:293:12)",
    "message": "ffmpeg exited with code 1: /data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4: No such file or directory\n",
    "stdout": "",
    "stderr": "ffmpeg version 5.1.6-0+deb12u1 Copyright (c) 2000-2024 the FFmpeg developers\n  built with gcc 12 (Debian 12.2.0-14)\n  configuration: --prefix=/usr --extra-version=0+deb12u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared\n  libavutil      57. 28.100 / 57. 28.100\n  libavcodec     59. 37.100 / 59. 37.100\n  libavformat    59. 27.100 / 59. 27.100\n  libavdevice    59.  7.100 / 59.  7.100\n  libavfilter     8. 44.100 /  8. 44.100\n  libswscale      6.  7.100 /  6.  7.100\n  libswresample   4.  7.100 /  4.  7.100\n  libpostproc    56.  6.100 / 56.  6.100\n/data/web-videos/bed590f3-8f3b-4c59-b3e1-65e5ee66e751-1080.mp4: No such file or directory\n"
  }
}

error[3/26/2025, 6:47:22 PM] Remote runner runner4 had an error with job 786eda35-9b73-4a1a-b2da-2dc9657b66ce (vod-web-video-transcoding)

{
  "errorMessage": "Not Found",
  "totalFailures": 1,
  "tags": [
    "api",
    "runner",
    "runner4",
    "786eda35-9b73-4a1a-b2da-2dc9657b66ce",
    "vod-web-video-transcoding"
  ]
}

warn[3/26/2025, 6:47:23 PM] Catch error in video import to send value to parent job.

{
  "payload": {
    "type": "youtube-dl",
    "videoImportId": 11795,
    "fileExt": ".mkv",
    "generateTranscription": true,
    "preventException": true
  },
  "err": {
    "stack": "Error\n    at Query.run (/app/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)\n    at /app/node_modules/sequelize/lib/sequelize.js:315:28\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryInterface.insert (/app/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n    at async RunnerJobModel.save (/app/node_modules/sequelize/lib/model.js:2490:35)\n    at async file:///app/dist/core/helpers/database-utils.js:33:17\n    at async /app/node_modules/sequelize/lib/sequelize.js:507:18",
    "message": "could not serialize access due to read/write dependencies among transactions",
    "name": "SequelizeDatabaseError",
    "parent": {
      "stack": "error: could not serialize access due to read/write dependencies among transactions\n    at Parser.parseErrorMessage (/app/node_modules/pg-protocol/dist/parser.js:283:98)\n    at Parser.handlePacket (/app/node_modules/pg-protocol/dist/parser.js:122:29)\n    at Parser.parse (/app/node_modules/pg-protocol/dist/parser.js:35:38)\n    at TLSSocket.<anonymous> (/app/node_modules/pg-protocol/dist/index.js:11:42)\n    at TLSSocket.emit (node:events:524:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Readable.push (node:internal/streams/readable:392:5)\n    at TLSWrap.onStreamRead (node:internal/stream_base_commons:191:23)",
      "message": "could not serialize access due to read/write dependencies among transactions",
      "length": 393,
      "name": "error",
      "severity": "ERROR",
      "code": "40001",
      "detail": "Reason code: Canceled on conflict out to pivot 35314914, during read.",
      "hint": "The transaction might succeed if retried.",
      "where": "SQL statement \"SELECT 1 FROM ONLY \"public\".\"runnerJob\" x WHERE \"id\" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x\"",
      "file": "predicate.c",
      "line": "4614",
      "routine": "OnConflict_CheckForSerializationFailure",
      "sql": "INSERT INTO \"runnerJob\" (\"id\",\"uuid\",\"type\",\"payload\",\"privatePayload\",\"state\",\"failures\",\"priority\",\"dependsOnRunnerJobId\",\"createdAt\",\"updatedAt\") VALUES (DEFAULT,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10) RETURNING \"id\",\"uuid\",\"type\",\"payload\",\"privatePayload\",\"state\",\"failures\",\"error\",\"priority\",\"processingJobToken\",\"progress\",\"startedAt\",\"finishedAt\",\"dependsOnRunnerJobId\",\"runnerId\",\"createdAt\",\"updatedAt\";",
      "parameters": [
        "84749b4a-804f-4f53-ae16-f00a8d213a9f",
        "vod-hls-transcoding",
        "{\"input\":{\"videoFileUrl\":\"https://makertube.net/api/v1/runners/jobs/84749b4a-804f-4f53-ae16-f00a8d213a9f/files/videos/484aaf3d-d348-4675-8c9c-d1897622ba09/max-quality\",\"separatedAudioFileUrl\":[]},\"output\":{\"resolution\":720,\"fps\":30,\"separatedAudio\":false}}",
        "{\"isNewVideo\":true,\"deleteWebVideoFiles\":false,\"videoUUID\":\"484aaf3d-d348-4675-8c9c-d1897622ba09\"}",
        5,
        0,
        252,
        51617,
        "2025-03-26 17:47:22.445 +00:00",
        "2025-03-26 17:47:22.465 +00:00"
      ]
    },
    "sql": "INSERT INTO \"runnerJob\" (\"id\",\"uuid\",\"type\",\"payload\",\"privatePayload\",\"state\",\"failures\",\"priority\",\"dependsOnRunnerJobId\",\"createdAt\",\"updatedAt\") VALUES (DEFAULT,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10) RETURNING \"id\",\"uuid\",\"type\",\"payload\",\"privatePayload\",\"state\",\"failures\",\"error\",\"priority\",\"processingJobToken\",\"progress\",\"startedAt\",\"finishedAt\",\"dependsOnRunnerJobId\",\"runnerId\",\"createdAt\",\"updatedAt\";"
  }
}

error[3/26/2025, 6:47:24 PM] Remote runner runner4 had an error with job 786eda35-9b73-4a1a-b2da-2dc9657b66ce (vod-web-video-transcoding)

{
  "errorMessage": "Not Found",
  "totalFailures": 2,
  "tags": [
    "api",
    "runner",
    "runner4",
    "786eda35-9b73-4a1a-b2da-2dc9657b66ce",
    "vod-web-video-transcoding"
  ]
}

error[3/26/2025, 6:47:25 PM] Remote runner runner4 had an error with job 786eda35-9b73-4a1a-b2da-2dc9657b66ce (vod-web-video-transcoding)

{
  "errorMessage": "Not Found",
  "totalFailures": 3,
  "tags": [
    "api",
    "runner",
    "runner4",
    "786eda35-9b73-4a1a-b2da-2dc9657b66ce",
    "vod-web-video-transcoding"
  ]
}

error[3/26/2025, 6:47:28 PM] Remote runner runner4 had an error with job 786eda35-9b73-4a1a-b2da-2dc9657b66ce (vod-web-video-transcoding)

{
  "errorMessage": "Not Found",
  "totalFailures": 4,
  "tags": [
    "api",
    "runner",
    "runner4",
    "786eda35-9b73-4a1a-b2da-2dc9657b66ce",
    "vod-web-video-transcoding"
  ]
}

error[3/26/2025, 6:47:28 PM] Remote runner runner4 had an error with job 786eda35-9b73-4a1a-b2da-2dc9657b66ce (vod-web-video-transcoding)

{
  "errorMessage": "Not Found",
  "totalFailures": 5,
  "tags": [
    "api",
    "runner",
    "runner4",
    "786eda35-9b73-4a1a-b2da-2dc9657b66ce",
    "vod-web-video-transcoding"
  ]
}

warn[3/26/2025, 6:47:50 PM] Cannot execute processUpdateVideo with many retries.

{
  "err": {
    "stack": "RequestError: Timeout awaiting 'request' for 7000ms\n    at ClientRequest.<anonymous> (file:///app/node_modules/got/dist/source/core/index.js:673:61)\n    at Object.onceWrapper (node:events:639:26)\n    at ClientRequest.emit (node:events:536:35)\n    at emitErrorEvent (node:_http_client:101:11)\n    at TLSSocket.socketErrorListener (node:_http_client:504:5)\n    at TLSSocket.emit (node:events:524:28)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at Timeout._onTimeout (file:///app/node_modules/got/dist/source/core/timed-out.js:39:33)\n    at listOnTimeout (node:internal/timers:581:17)\n    at process.processTimers (node:internal/timers:519:7)",
    "message": "Timeout awaiting 'request' for 7000ms",
    "name": "TimeoutError",
    "requestHeaders": {
      "user-agent": "PeerTube/7.1.0 (+https://makertube.net)",
      "date": "Wed, 26 Mar 2025 17:47:26 GMT",
      "accept": "application/activity+json, application/ld+json",
      "accept-encoding": "gzip, deflate, br",
      "host": "peertube.tv",
      "signature": "keyId=\"https://makertube.net/accounts/peertube#main-key\",algorithm=\"rsa-sha256\",headers=\"(request-target) host date\",signature=\"TtcSh3brD8XAELaOfmtfnZBPTBjUQVScb0CoShPbXgw1BIA4QXmATMPQJJyMqM2yQetiNSYLogUHHbSiZOXFnEerv37BU44lUszbz7/L8M+F5R+ae7u1hkr5cdaAZX0hJBs++qqrWED8jGf9wiK1ruevtFf557O8lMtpIFh2BloiMFzsCx9k5C3ndDW2/MMdWpQIYer2hPBq3YdYURV0rPrpEXWo/WwuPR/GwKznWVemYpYvlPQvcv5bKwpwfp/UTUeJiRzEMFhHlwYvV4l2mMua3n024WvAk1x284pVwsFpOc8+0tMuDUoyd4ct+x9Ia7mnNI3wNggzuEkii4/Jgg==\""
    },
    "requestUrl": "https://peertube.tv/accounts/transportevolved",
    "requestMethod": "GET"
  }
}

warn[3/26/2025, 6:47:50 PM] Cannot process activity Update.

{
  "err": {
    "stack": "RequestError: Timeout awaiting 'request' for 7000ms\n    at ClientRequest.<anonymous> (file:///app/node_modules/got/dist/source/core/index.js:673:61)\n    at Object.onceWrapper (node:events:639:26)\n    at ClientRequest.emit (node:events:536:35)\n    at emitErrorEvent (node:_http_client:101:11)\n    at TLSSocket.socketErrorListener (node:_http_client:504:5)\n    at TLSSocket.emit (node:events:524:28)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at Timeout._onTimeout (file:///app/node_modules/got/dist/source/core/timed-out.js:39:33)\n    at listOnTimeout (node:internal/timers:581:17)\n    at process.processTimers (node:internal/timers:519:7)",
    "message": "Timeout awaiting 'request' for 7000ms",
    "name": "TimeoutError",
    "requestHeaders": {
      "user-agent": "PeerTube/7.1.0 (+https://makertube.net)",
      "date": "Wed, 26 Mar 2025 17:47:26 GMT",
      "accept": "application/activity+json, application/ld+json",
      "accept-encoding": "gzip, deflate, br",
      "host": "peertube.tv",
      "signature": "keyId=\"https://makertube.net/accounts/peertube#main-key\",algorithm=\"rsa-sha256\",headers=\"(request-target) host date\",signature=\"TtcSh3brD8XAELaOfmtfnZBPTBjUQVScb0CoShPbXgw1BIA4QXmATMPQJJyMqM2yQetiNSYLogUHHbSiZOXFnEerv37BU44lUszbz7/L8M+F5R+ae7u1hkr5cdaAZX0hJBs++qqrWED8jGf9wiK1ruevtFf557O8lMtpIFh2BloiMFzsCx9k5C3ndDW2/MMdWpQIYer2hPBq3YdYURV0rPrpEXWo/WwuPR/GwKznWVemYpYvlPQvcv5bKwpwfp/UTUeJiRzEMFhHlwYvV4l2mMua3n024WvAk1x284pVwsFpOc8+0tMuDUoyd4ct+x9Ia7mnNI3wNggzuEkii4/Jgg==\""
    },
    "requestUrl": "https://peertube.tv/accounts/transportevolved",
    "requestMethod": "GET"
  }
}

error[3/26/2025, 6:48:33 PM] Remote runner tube1 had an error with job 8e32f492-3929-4921-9dd5-0808a807182b (vod-hls-transcoding)

{
  "errorMessage": "read ECONNRESET",
  "totalFailures": 1,
  "tags": [
    "api",
    "runner",
    "tube1",
    "8e32f492-3929-4921-9dd5-0808a807182b",
    "vod-hls-transcoding"
  ]
}

warn[3/26/2025, 6:48:46 PM] Cannot execute with many retries.

{
  "err": {
    "stack": "Error\n    at Query.run (/app/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)\n    at /app/node_modules/sequelize/lib/sequelize.js:315:28\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryInterface.select (/app/node_modules/sequelize/lib/dialects/abstract/query-interface.js:407:12)\n    at async RunnerJobModel.findAll (/app/node_modules/sequelize/lib/model.js:1140:21)\n    at async RunnerJobModel.findOne (/app/node_modules/sequelize/lib/model.js:1240:12)\n    at async self.reload (/app/node_modules/sequelize/lib/model.js:2562:22)\n    at async file:///app/dist/core/controllers/api/runners/jobs.js:47:13\n    at async /app/node_modules/sequelize/lib/sequelize.js:507:18",
    "message": "could not serialize access due to read/write dependencies among transactions",
    "name": "SequelizeDatabaseError",
    "parent": {
      "stack": "error: could not serialize access due to read/write dependencies among transactions\n    at Parser.parseErrorMessage (/app/node_modules/pg-protocol/dist/parser.js:283:98)\n    at Parser.handlePacket (/app/node_modules/pg-protocol/dist/parser.js:122:29)\n    at Parser.parse (/app/node_modules/pg-protocol/dist/parser.js:35:38)\n    at TLSSocket.<anonymous> (/app/node_modules/pg-protocol/dist/index.js:11:42)\n    at TLSSocket.emit (node:events:524:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Readable.push (node:internal/streams/readable:392:5)\n    at TLSWrap.onStreamRead (node:internal/stream_base_commons:191:23)",
      "message": "could not serialize access due to read/write dependencies among transactions",
      "length": 278,
      "name": "error",
      "severity": "ERROR",
      "code": "40001",
      "detail": "Reason code: Canceled on conflict out to pivot 35315157, during read.",
      "hint": "The transaction might succeed if retried.",
      "file": "predicate.c",
      "line": "4614",
      "routine": "OnConflict_CheckForSerializationFailure",
      "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51626;"
    },
    "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51626;",
    "parameters": {}
  }
}

error[3/26/2025, 6:48:46 PM] Error in controller.

{
  "err": {
    "stack": "Error\n    at Query.run (/app/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)\n    at /app/node_modules/sequelize/lib/sequelize.js:315:28\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryInterface.select (/app/node_modules/sequelize/lib/dialects/abstract/query-interface.js:407:12)\n    at async RunnerJobModel.findAll (/app/node_modules/sequelize/lib/model.js:1140:21)\n    at async RunnerJobModel.findOne (/app/node_modules/sequelize/lib/model.js:1240:12)\n    at async self.reload (/app/node_modules/sequelize/lib/model.js:2562:22)\n    at async file:///app/dist/core/controllers/api/runners/jobs.js:47:13\n    at async /app/node_modules/sequelize/lib/sequelize.js:507:18",
    "message": "could not serialize access due to read/write dependencies among transactions",
    "name": "SequelizeDatabaseError",
    "parent": {
      "stack": "error: could not serialize access due to read/write dependencies among transactions\n    at Parser.parseErrorMessage (/app/node_modules/pg-protocol/dist/parser.js:283:98)\n    at Parser.handlePacket (/app/node_modules/pg-protocol/dist/parser.js:122:29)\n    at Parser.parse (/app/node_modules/pg-protocol/dist/parser.js:35:38)\n    at TLSSocket.<anonymous> (/app/node_modules/pg-protocol/dist/index.js:11:42)\n    at TLSSocket.emit (node:events:524:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Readable.push (node:internal/streams/readable:392:5)\n    at TLSWrap.onStreamRead (node:internal/stream_base_commons:191:23)",
      "message": "could not serialize access due to read/write dependencies among transactions",
      "length": 278,
      "name": "error",
      "severity": "ERROR",
      "code": "40001",
      "detail": "Reason code: Canceled on conflict out to pivot 35315157, during read.",
      "hint": "The transaction might succeed if retried.",
      "file": "predicate.c",
      "line": "4614",
      "routine": "OnConflict_CheckForSerializationFailure",
      "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51626;"
    },
    "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51626;",
    "parameters": {}
  },
  "sql": "SELECT \"RunnerJobModel\".\"id\", \"RunnerJobModel\".\"uuid\", \"RunnerJobModel\".\"type\", \"RunnerJobModel\".\"payload\", \"RunnerJobModel\".\"privatePayload\", \"RunnerJobModel\".\"state\", \"RunnerJobModel\".\"failures\", \"RunnerJobModel\".\"error\", \"RunnerJobModel\".\"priority\", \"RunnerJobModel\".\"processingJobToken\", \"RunnerJobModel\".\"progress\", \"RunnerJobModel\".\"startedAt\", \"RunnerJobModel\".\"finishedAt\", \"RunnerJobModel\".\"dependsOnRunnerJobId\", \"RunnerJobModel\".\"runnerId\", \"RunnerJobModel\".\"createdAt\", \"RunnerJobModel\".\"updatedAt\", \"Runner\".\"id\" AS \"Runner.id\", \"Runner\".\"runnerToken\" AS \"Runner.runnerToken\", \"Runner\".\"name\" AS \"Runner.name\", \"Runner\".\"description\" AS \"Runner.description\", \"Runner\".\"lastContact\" AS \"Runner.lastContact\", \"Runner\".\"ip\" AS \"Runner.ip\", \"Runner\".\"runnerRegistrationTokenId\" AS \"Runner.runnerRegistrationTokenId\", \"Runner\".\"createdAt\" AS \"Runner.createdAt\", \"Runner\".\"updatedAt\" AS \"Runner.updatedAt\" FROM \"runnerJob\" AS \"RunnerJobModel\" LEFT OUTER JOIN \"runner\" AS \"Runner\" ON \"RunnerJobModel\".\"runnerId\" = \"Runner\".\"id\" WHERE \"RunnerJobModel\".\"id\" = 51626;",
  "url": "/api/v1/runners/jobs/642f128d-b1da-4071-8513-6e8f45cba825/accept"
}
7

Thanks, can you also provide info level logs?

8

I’ll try, but it’s a mess when I turn that on.