Cannot set headers after they are sent to the client

PeerTube
1

Hi.

Recently we are getting a fatal error on our peertube-server that causes a restart of the application.

Seems to be triggered at some cases when the video-watch page is opened. Not anywhere else.

I’m not able to reproduce the problem on a local installation.

Anyone with ideas on how to solve this?

Apr  5 15:51:01 video peertube[28083]: [video.playmaker.ai:443] 2023-04-05 15:51:01.747 #033[32minfo#033[39m: 85.195.33.11 - - [05/Apr/2023:15:51:01 +0000] "GET /object-storage-proxy/streaming-playlists/hls/private/833fda45-2309-483c-ba86-781ff036f83f/d5e9845f-883f-4d92-891d-88287d473395-720-fragmented.mp4 HTTP/1.0" 206 - "https://video.playmaker.ai/w/hd29Hb2nkA1kxcwUsLmhYP" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
Apr  5 15:51:01 video peertube[28083]: [video.playmaker.ai:443] 2023-04-05 15:51:01.753 #033[31merror#033[39m: uncaughtException: Cannot set headers after they are sent to the client
Apr  5 15:51:01 video peertube[28083]: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
Apr  5 15:51:01 video peertube[28083]:     at new NodeError (internal/errors.js:322:7)
Apr  5 15:51:01 video peertube[28083]:     at ServerResponse.setHeader (_http_outgoing.js:561:11)
Apr  5 15:51:01 video peertube[28083]:     at ServerResponse.res.fail (/var/www/peertube/versions/peertube-v5.0.1/dist/server/middlewares/error.js:11:13)
Apr  5 15:51:01 video peertube[28083]:     at handleObjectStorageFailure (/var/www/peertube/versions/peertube-v5.0.1/dist/server/controllers/object-storage-proxy.js:69:16)
Apr  5 15:51:01 video peertube[28083]:     at /var/www/peertube/versions/peertube-v5.0.1/dist/server/controllers/object-storage-proxy.js:56:17
Apr  5 15:51:01 video peertube[28083]:     at internal/util.js:435:14
Apr  5 15:51:01 video peertube[28083]:     at finish (internal/streams/pipeline.js:162:7)
Apr  5 15:51:01 video peertube[28083]:     at internal/util.js:435:14
Apr  5 15:51:01 video peertube[28083]:     at internal/streams/pipeline.js:73:5
Apr  5 15:51:01 video peertube[28083]:     at finish (internal/streams/pipeline.js:158:23)
Apr  5 15:51:01 video peertube[28083]:     at internal/util.js:435:14
Apr  5 15:51:01 video peertube[28083]:     at internal/streams/pipeline.js:73:5
Apr  5 15:51:01 video peertube[28083]:     at finish (internal/streams/pipeline.js:158:23)
Apr  5 15:51:01 video peertube[28083]:     at internal/util.js:435:14
Apr  5 15:51:01 video peertube[28083]:     at ServerResponse.<anonymous> (internal/streams/pipeline.js:65:7)
Apr  5 15:51:01 video peertube[28083]:     at ServerResponse.<anonymous> (internal/util.js:435:14)
Apr  5 15:51:01 video peertube[28083]:     at ServerResponse.onclose (internal/streams/end-of-stream.js:121:25)
Apr  5 15:51:01 video peertube[28083]:     at ServerResponse.emit (events.js:412:35)
Apr  5 15:51:01 video peertube[28083]:     at Socket.onServerResponseClose (_http_server.js:221:44)
Apr  5 15:51:01 video peertube[28083]:     at Socket.emit (events.js:412:35)
Apr  5 15:51:01 video peertube[28083]:     at TCP.<anonymous> (net.js:686:12) {
Apr  5 15:51:01 video peertube[28083]:   "error": {
Apr  5 15:51:01 video peertube[28083]:     "stack": "Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n    at new NodeError (internal/errors.js:322:7)\n    at ServerResponse.setHeader (_http_outgoing.js:561:11)\n    at ServerResponse.res.fail (/var/www/peertube/versions/peertube-v5.0.1/dist/server/middlewares/error.js:11:13)\n    at handleObjectStorageFailure (/var/www/peertube/versions/peertube-v5.0.1/dist/server/controllers/object-storage-proxy.js:69:16)\n    at /var/www/peertube/versions/peertube-v5.0.1/dist/server/controllers/object-storage-proxy.js:56:17\n    at internal/util.js:435:14\n    at finish (internal/streams/pipeline.js:162:7)\n    at internal/util.js:435:14\n    at internal/streams/pipeline.js:73:5\n    at finish (internal/streams/pipeline.js:158:23)\n    at internal/util.js:435:14\n    at internal/streams/pipeline.js:73:5\n    at finish (internal/streams/pipeline.js:158:23)\n    at internal/util.js:435:14\n    at ServerResponse.<anonymous> (internal/streams/pipeline.js:65:7)\n    at ServerResponse.<anonymous> (internal/util.js:435:14)\n    at ServerResponse.onclose (internal/streams/end-of-stream.js:121:25)\n    at ServerResponse.emit (events.js:412:35)\n    at Socket.onServerResponseClose (_http_server.js:221:44)\n    at Socket.emit (events.js:412:35)\n    at TCP.<anonymous> (net.js:686:12)",
Apr  5 15:51:01 video peertube[28083]:     "message": "Cannot set headers after they are sent to the client",
Apr  5 15:51:01 video peertube[28083]:     "code": "ERR_HTTP_HEADERS_SENT"
Apr  5 15:51:01 video peertube[28083]:   },
Apr  5 15:51:01 video peertube[28083]:   "stack": "Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n    at new NodeError (internal/errors.js:322:7)\n    at ServerResponse.setHeader (_http_outgoing.js:561:11)\n    at ServerResponse.res.fail (/var/www/peertube/versions/peertube-v5.0.1/dist/server/middlewares/error.js:11:13)\n    at handleObjectStorageFailure (/var/www/peertube/versions/peertube-v5.0.1/dist/server/controllers/object-storage-proxy.js:69:16)\n    at /var/www/peertube/versions/peertube-v5.0.1/dist/server/controllers/object-storage-proxy.js:56:17\n    at internal/util.js:435:14\n    at finish (internal/streams/pipeline.js:162:7)\n    at internal/util.js:435:14\n    at internal/streams/pipeline.js:73:5\n    at finish (internal/streams/pipeline.js:158:23)\n    at internal/util.js:435:14\n    at internal/streams/pipeline.js:73:5\n    at finish (internal/streams/pipeline.js:158:23)\n    at internal/util.js:435:14\n    at ServerResponse.<anonymous> (internal/streams/pipeline.js:65:7)\n    at ServerResponse.<anonymous> (internal/util.js:435:14)\n    at ServerResponse.onclose (internal/streams/end-of-stream.js:121:25)\n    at ServerResponse.emit (events.js:412:35)\n    at Socket.onServerResponseClose (_http_server.js:221:44)\n    at Socket.emit (events.js:412:35)\n    at TCP.<anonymous> (net.js:686:12)",
Apr  5 15:51:01 video peertube[28083]:   "exception": true,
Apr  5 15:51:01 video peertube[28083]:   "date": "Wed Apr 05 2023 15:51:01 GMT+0000 (Coordinated Universal Time)",
Apr  5 15:51:01 video peertube[28083]:   "process": {
Apr  5 15:51:01 video peertube[28083]:     "pid": 28083,
Apr  5 15:51:01 video peertube[28083]:     "uid": 1000,
Apr  5 15:51:01 video peertube[28083]:     "gid": 1000,
Apr  5 15:51:01 video peertube[28083]:     "cwd": "/var/www/peertube/versions/peertube-v5.0.1",
Apr  5 15:51:01 video peertube[28083]:     "execPath": "/usr/bin/node",
Apr  5 15:51:01 video peertube[28083]:     "version": "v14.19.3",
Apr  5 15:51:01 video peertube[28083]:     "argv": [
Apr  5 15:51:01 video peertube[28083]:       "/usr/bin/node",
Apr  5 15:51:01 video peertube[28083]:       "/var/www/peertube/versions/peertube-v5.0.1/dist/server"
Apr  5 15:51:01 video peertube[28083]:     ],
Apr  5 15:51:01 video peertube[28083]:     "memoryUsage": {
Apr  5 15:51:01 video peertube[28083]:       "rss": 178421760,
Apr  5 15:51:01 video peertube[28083]:       "heapTotal": 117354496,
Apr  5 15:51:01 video peertube[28083]:       "heapUsed": 111070048,
Apr  5 15:51:01 video peertube[28083]:       "external": 2851463,
Apr  5 15:51:01 video peertube[28083]:       "arrayBuffers": 716645
Apr  5 15:51:01 video peertube[28083]:     }
Apr  5 15:51:01 video peertube[28083]:   },
Apr  5 15:51:01 video peertube[28083]:   "os": {
Apr  5 15:51:01 video peertube[28083]:     "loadavg": [
Apr  5 15:51:01 video peertube[28083]:       0.05,
Apr  5 15:51:01 video peertube[28083]:       0.05,
Apr  5 15:51:01 video peertube[28083]:       0.07
Apr  5 15:51:01 video peertube[28083]:     ],
Apr  5 15:51:01 video peertube[28083]:     "uptime": 90189.83
Apr  5 15:51:01 video peertube[28083]:   },
Apr  5 15:51:01 video peertube[28083]:   "trace": [
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 7,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/errors.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "new NodeError",
Apr  5 15:51:01 video peertube[28083]:       "line": 322,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 11,
Apr  5 15:51:01 video peertube[28083]:       "file": "_http_outgoing.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "ServerResponse.setHeader",
Apr  5 15:51:01 video peertube[28083]:       "line": 561,
Apr  5 15:51:01 video peertube[28083]:       "method": "setHeader",
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 13,
Apr  5 15:51:01 video peertube[28083]:       "file": "/var/www/peertube/versions/peertube-v5.0.1/dist/server/middlewares/error.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "ServerResponse.res.fail",
Apr  5 15:51:01 video peertube[28083]:       "line": 11,
Apr  5 15:51:01 video peertube[28083]:       "method": "fail",
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 16,
Apr  5 15:51:01 video peertube[28083]:       "file": "/var/www/peertube/versions/peertube-v5.0.1/dist/server/controllers/object-storage-proxy.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "handleObjectStorageFailure",
Apr  5 15:51:01 video peertube[28083]:       "line": 69,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 17,
Apr  5 15:51:01 video peertube[28083]:       "file": "/var/www/peertube/versions/peertube-v5.0.1/dist/server/controllers/object-storage-proxy.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 56,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 14,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/util.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 435,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 7,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/streams/pipeline.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "finish",
Apr  5 15:51:01 video peertube[28083]:       "line": 162,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 14,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/util.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 435,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 5,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/streams/pipeline.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 73,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 23,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/streams/pipeline.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "finish",
Apr  5 15:51:01 video peertube[28083]:       "line": 158,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 14,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/util.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 435,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 5,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/streams/pipeline.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 73,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 23,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/streams/pipeline.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "finish",
Apr  5 15:51:01 video peertube[28083]:       "line": 158,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 14,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/util.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 435,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 7,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/streams/pipeline.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 65,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 14,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/util.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 435,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 25,
Apr  5 15:51:01 video peertube[28083]:       "file": "internal/streams/end-of-stream.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "ServerResponse.onclose",
Apr  5 15:51:01 video peertube[28083]:       "line": 121,
Apr  5 15:51:01 video peertube[28083]:       "method": "onclose",
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 35,
Apr  5 15:51:01 video peertube[28083]:       "file": "events.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "ServerResponse.emit",
Apr  5 15:51:01 video peertube[28083]:       "line": 412,
Apr  5 15:51:01 video peertube[28083]:       "method": "emit",
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 44,
Apr  5 15:51:01 video peertube[28083]:       "file": "_http_server.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "Socket.onServerResponseClose",
Apr  5 15:51:01 video peertube[28083]:       "line": 221,
Apr  5 15:51:01 video peertube[28083]:       "method": "onServerResponseClose",
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 35,
Apr  5 15:51:01 video peertube[28083]:       "file": "events.js",
Apr  5 15:51:01 video peertube[28083]:       "function": "Socket.emit",
Apr  5 15:51:01 video peertube[28083]:       "line": 412,
Apr  5 15:51:01 video peertube[28083]:       "method": "emit",
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     },
Apr  5 15:51:01 video peertube[28083]:     {
Apr  5 15:51:01 video peertube[28083]:       "column": 12,
Apr  5 15:51:01 video peertube[28083]:       "file": "net.js",
Apr  5 15:51:01 video peertube[28083]:       "function": null,
Apr  5 15:51:01 video peertube[28083]:       "line": 686,
Apr  5 15:51:01 video peertube[28083]:       "method": null,
Apr  5 15:51:01 video peertube[28083]:       "native": false
Apr  5 15:51:01 video peertube[28083]:     }
Apr  5 15:51:01 video peertube[28083]:   ]
Apr  5 15:51:01 video peertube[28083]: }
2

the problem seem to happen when a video is resumed at some point.

so to reproduce the error I can scroll to a certain time in the playing video and then reload the page

this seem to happen at the same time

Apr 5 16:12:01 video peertube[28683]: [video.playmaker.ai:443] 2023-04-05 16:12:01.419 #033[32minfo#033[39m: 85.195.33.11 - - [05/Apr/2023:16:12:01 +0000] “GET /object-storage-proxy/streaming-playlists/hls/private/833fda45-2309-483c-ba86-781ff036f83f/7d92850d-21a8-4858-8860-50d9a2cae657-1080-fragmented.mp4 HTTP/1.0” 206 - “https://video.playmaker.ai/w/hd29Hb2nkA1kxcwUsLmhYP” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36”
Apr 5 16:12:02 video peertube[28683]: [video.playmaker.ai:443] 2023-04-05 16:12:02.658 #033[33mwarn#033[39m: Too much watch section to store for a viewer, skipping this one {
Apr 5 16:12:02 video peertube[28683]:  “currentTime”: 5295
Apr 5 16:12:02 video peertube[28683]: }
Apr 5 16:12:02 video peertube[28683]: [video.playmaker.ai:443] 2023-04-05 16:12:02.662 #033[32minfo#033[39m: Creating job to send viewer of https://video.playmaker.ai/videos/watch/833fda45-2309-483c-ba86-781ff036f83f.
Apr 5 16:12:02 video peertube[28683]: [video.playmaker.ai:443] 2023-04-05 16:12:02.680 #033[32minfo#033[39m: 85.195.33.11 - - [05/Apr/2023:16:12:02 +0000] “POST /api/v1/videos/833fda45-2309-483c-ba86-781ff036f83f/views HTTP/1.0” 204 - “https://video.playmaker.ai/w/hd29Hb2nkA1kxcwUsLmhYP” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36”
Apr 5 16:12:05 video peertube[28683]: [video.playmaker.ai:443] 2023-04-05 16:12:05.568 #033[33mwarn#033[39m: Invalid end watch section 0. Last start record was at 499.
Apr 5 16:12:05 video peertube[28683]: [video.playmaker.ai:443] 2023-04-05 16:12:05.571 #033[32minfo#033[39m: Creating job to send viewer of https://video.playmaker.ai/videos/watch/933b4741-8c3e-4a14-9a49-c25870a3635c.
3

This log seems related to object-storage. Have you also a S3 service configured on your local installation?

I’m not using object storage, so I’m not competent to analyse the situation. But I guess it could be because Peertube tries to set headers in addition to the S3 headers, but too late (as the error is in object-storage-proxy.js).

4

Oh, it seems your Peertube version is 5.0.1. I think there were some issues with this version and object storage. Can you try to update to the v5.1.0?

5

Hi, I think there is a bug to correctly catch the error. I don’t know what is the root cause of your issue, but I’ll fix the uncaughtException (our error handler throws an error).

6

I updated to 5.1.0 and get the same problem :frowning:

7

great!

also could this be a hint of why this happens?

Apr  6 11:02:02 video peertube[44093]: [video.playmaker.ai:443 uploadx] 2023-04-06 11:02:02.243 #033[31merror#033[39m: uncaughtException: Cannot set headers after they are sent to the client
Apr  6 11:02:02 video peertube[44093]: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
Apr  6 11:02:02 video peertube[44093]:     at new NodeError (internal/errors.js:322:7)
Apr  6 11:02:02 video peertube[44093]:     at ServerResponse.setHeader (_http_outgoing.js:561:11)
Apr  6 11:02:02 video peertube[44093]:     at ServerResponse.res.fail (/var/www/peertube/versions/peertube-v5.1.0/dist/server/middlewares/error.js:11:13)
Apr  6 11:02:02 video peertube[44093]:     at handleObjectStorageFailure (/var/www/peertube/versions/peertube-v5.1.0/dist/server/controllers/object-storage-proxy.js:69:16)
Apr  6 11:02:02 video peertube[44093]:     at /var/www/peertube/versions/peertube-v5.1.0/dist/server/controllers/object-storage-proxy.js:56:17
Apr  6 11:02:02 video peertube[44093]:     at internal/util.js:435:14
Apr  6 11:02:02 video peertube[44093]:     at finish (internal/streams/pipeline.js:162:7)
Apr  6 11:02:02 video peertube[44093]:     at internal/util.js:435:14
Apr  6 11:02:02 video peertube[44093]:     at internal/streams/pipeline.js:73:5
Apr  6 11:02:02 video peertube[44093]:     at finish (internal/streams/pipeline.js:158:23)
Apr  6 11:02:02 video peertube[44093]:     at internal/util.js:435:14
Apr  6 11:02:02 video peertube[44093]:     at internal/streams/pipeline.js:73:5
Apr  6 11:02:02 video peertube[44093]:     at finish (internal/streams/pipeline.js:158:23)
Apr  6 11:02:02 video peertube[44093]:     at internal/util.js:435:14
Apr  6 11:02:02 video peertube[44093]:     at ServerResponse.<anonymous> (internal/streams/pipeline.js:65:7)
Apr  6 11:02:02 video peertube[44093]:     at ServerResponse.<anonymous> (internal/util.js:435:14)
Apr  6 11:02:02 video peertube[44093]:     at ServerResponse.onclose (internal/streams/end-of-stream.js:121:25)
Apr  6 11:02:02 video peertube[44093]:     at ServerResponse.emit (events.js:412:35)
Apr  6 11:02:02 video peertube[44093]:     at Socket.onServerResponseClose (_http_server.js:221:44)
Apr  6 11:02:02 video peertube[44093]:     at Socket.emit (events.js:412:35)
Apr  6 11:02:02 video peertube[44093]:     at TCP.<anonymous> (net.js:686:12) {
Apr  6 11:02:02 video peertube[44093]:   "error": {
8

I also have updated redis and node now to versions 6 and 18

9

I got it not dying and restarting by changing this in object-storage-proxy.ts

 return pipeline(
      stream,
      streamReplacer,
      res,
      err => {
        return err
      }
    )

from

 return pipeline(
      stream,
      streamReplacer,
      res,
      err => {
        if (!err) return

        handleObjectStorageFailure(res, err)
      }
    )

Im not sure if this is a bug?

In the logs now if it may be a hint is the following:

I get the feeling it is all related to jumping into a previous time when watching videos. That’s when it previously failed and killed the server.

Apr  6 17:21:01 video peertube[9887]: [video.playmaker.ai:443] 2023-04-06 17:21:01.521 #033[33mwarn#033[39m: Client log: HLS.js error: mediaError - fatal: false - bufferStalledError {
Apr  6 17:21:01 video peertube[9887]:   "username": "root",
Apr  6 17:21:01 video peertube[9887]:   "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
Apr  6 17:21:01 video peertube[9887]:   "url": "https://video.playmaker.ai/w/ctGLbsKTTyBbRTdnUZqbeQ"
Apr  6 17:21:01 video peertube[9887]: }