Federation sync drop out periodically [Logs attached]

Hello,

we have been working with Peertube for a few years and follow the project with great interest. We run a few instances ourselves and currently have a problem with our federation.
The error manifests itself in such a way that after some time the videos and metadata (e.g. views) are no longer synchronized correctly between the instances and then on the public instance, new videos are missing and the views of the videos are no longer updated.
Since we have been investigating the problem for some time now, we have made various attempts to solve the problem, even reinstalling the instances and establishing new (empty) content instances to which the new videos were then uploaded, but after some time the synchronization problems occurred again.
Since we can’t exclude the possibility that someone is creating this error condition from the outside and sending requests to our servers, which then disrupt the federation as a whole, we wanted to ask here if someone can help us with these error messages we have received.

Nov 30 14:06:07 vid4 peertube[97994]: [vid4.domain.nx:443] 2021-11-30 14:06:07.182 warn: Warning in tracker. {
Nov 30 14:06:07 vid4 peertube[97994]:   "err": {
Nov 30 14:06:07 vid4 peertube[97994]:     "stack": "Error: no peer with that `to_peer_id`\n    at /var/www/peertube/versions/peertube-v3.4.1/node_modules/bittorrent-tracker/server.js:554:41\n    at /var/www/peertube/versions/peertube-v3.4.1/node_modules/bittorrent-tracker/server.js:364:7\n    at processTicksAndRejections (internal/process/task_queues.js:77:11)",
Nov 30 14:06:07 vid4 peertube[97994]:     "message": "no peer with that `to_peer_id`"
Nov 30 14:06:07 vid4 peertube[97994]:   }
Nov 30 14:06:07 vid4 peertube[97994]: }

Question i.) Where does this message come from and which « peer_id » is meant here. Further information is not available in the message. Occurs on a standalone instance that used to be federated, but is not anymore. Occurs regularly again and again.

Nov 30 14:05:23 vid4 peertube[97994]: [vid4.domain.nx:443] 2021-11-30 14:05:23.628 warn: Warning in tracker. {
Nov 30 14:05:23 vid4 peertube[97994]:   "err": {
Nov 30 14:05:23 vid4 peertube[97994]:     "stack": "Error: Unknown infoHash 6f06afe55524d66b68ea25be56dd1f97e919c152 requested by ip x.y.z.xx\n    at Server.<anonymous> (/var/www/peertube/versions/peertube-v3.4.1/dist/server/controllers/tracker.js:52:20)\n    at Generator.next (<anonymous>)\n    at fulfilled (/var/www/peertube/versions/peertube-v3.4.1/node_modules/tslib/tslib.js:114:62)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
Nov 30 14:05:23 vid4 peertube[97994]:     "message": "Unknown infoHash 6f06afe55524d66b68ea25be56dd1f97e919c152 requested by ip x.y.z.xx"
Nov 30 14:05:23 vid4 peertube[97994]:   }
Nov 30 14:05:23 vid4 peertube[97994]: }

Question ii.) Where do these requests come from? A video with this « infoHash » does not exist on the instance and never did. Occurs on a standalone instance that used to be federated, but is no longer. Occurs regularly over and over again with different infoHashes

While troubleshooting federation, we received mostly activitypub errors.

Nov 29 11:46:18 vid peertube[23018]: [vid.domain.nx:443] 2021-11-29 11:46:18.750 error: Cannot execute job 183810 in queue activitypub-http-unicast. {
Nov 29 11:46:18 vid peertube[23018]:   "payload": {
Nov 29 11:46:18 vid peertube[23018]:     "uri": "URL//inbox",
Nov 29 11:46:18 vid peertube[23018]:     "signatureActorId": 1,
Nov 29 11:46:18 vid peertube[23018]:     "body": {
Nov 29 11:46:18 vid peertube[23018]:       "id": "URL//accounts/peertube/views/videos/2550/2021-11-29T11:45:53.345Z",
Nov 29 11:46:18 vid peertube[23018]:       "type": "View",
Nov 29 11:46:18 vid peertube[23018]:       "actor": "URL//accounts/peertube",
Nov 29 11:46:18 vid peertube[23018]:       "object": "URL//videos/watch/d8cbbdc1-a575-4e27-b568-c38e6328d700",
Nov 29 11:46:18 vid peertube[23018]:       "to": [
Nov 29 11:46:18 vid peertube[23018]:         "URL//accounts/redakteur"
Nov 29 11:46:18 vid peertube[23018]:       ],
Nov 29 11:46:18 vid peertube[23018]:       "cc": [
Nov 29 11:46:18 vid peertube[23018]:         "URL//video-channels/redaktion/followers",
Nov 29 11:46:18 vid peertube[23018]:         "URL//accounts/peertube/followers",
Nov 29 11:46:18 vid peertube[23018]:         "URL//accounts/redakteur/followers"
Nov 29 11:46:18 vid peertube[23018]:       ]
Nov 29 11:46:18 vid peertube[23018]:     },
Nov 29 11:46:18 vid peertube[23018]:     "contextType": "View"
Nov 29 11:46:18 vid peertube[23018]:   },
Nov 29 11:46:18 vid peertube[23018]:   "err": {
Nov 29 11:46:18 vid peertube[23018]:     "stack": "RequestError: queryA ESERVFAIL vid4.domain.nx\n    at ClientRequest.<anonymous> (/var/www/peertube/versions/peertube-v3.4.1/node_modules/got/dist/source/core/index.js:956:111)\n    at Object.onceWrapper (events.js:520:26)\n    at ClientRequest.emit (events.js:412:35)\n    at ClientRequest.emit (domain.js:475:12)\n    at ClientRequest.origin.emit (/var/www/peertube/versions/peertube-v3.4.1/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at TLSSocket.socketErrorListener (_http_client.js:475:9)\n    at TLSSocket.emit (events.js:400:28)\n    at TLSSocket.emit (domain.js:475:12)\n    at emitErrorNT (internal/streams/destroy.js:106:8)\n    at emitErrorCloseNT (internal/streams/destroy.js:74:3)\n    at processTicksAndRejections (internal/process/task_queues.js:82:21)\n    at QueryReqWrap.onresolve [as oncomplete] (internal/dns/promises.js:172:17)",
Nov 29 11:46:18 vid peertube[23018]:     "message": "queryA ESERVFAIL vid4.domain.nx",
Nov 29 11:46:18 vid peertube[23018]:     "name": "RequestError"
Nov 29 11:46:18 vid peertube[23018]:   }
Nov 29 11:46:18 vid peertube[23018]: }


Nov 29 12:42:22 vid peertube[126684]: [vid.domain.nx:443] 2021-11-29 12:42:22.539 error: Cannot execute job 184138 in queue activitypub-http-unicast. {
Nov 29 12:42:22 vid peertube[126684]:   "payload": {
Nov 29 12:42:22 vid peertube[126684]:     "uri": "URL//inbox",
Nov 29 12:42:22 vid peertube[126684]:     "signatureActorId": 1,
Nov 29 12:42:22 vid peertube[126684]:     "body": {
Nov 29 12:42:22 vid peertube[126684]:       "id": "URL//accounts/peertube/views/videos/2549/2021-11-29T12:41:12.761Z",
Nov 29 12:42:22 vid peertube[126684]:       "type": "View",
Nov 29 12:42:22 vid peertube[126684]:       "actor": "URL//accounts/peertube",
Nov 29 12:42:22 vid peertube[126684]:       "object": "URL//videos/watch/1bc808bf-5f36-4a59-9eb4-948f17a27ff1",
Nov 29 12:42:22 vid peertube[126684]:       "to": [
Nov 29 12:42:22 vid peertube[126684]:         "URL//accounts/redakteur"
Nov 29 12:42:22 vid peertube[126684]:       ],
Nov 29 12:42:22 vid peertube[126684]:       "cc": [
Nov 29 12:42:22 vid peertube[126684]:         "URL//video-channels/redaktion/followers",
Nov 29 12:42:22 vid peertube[126684]:         "URL//accounts/peertube/followers",
Nov 29 12:42:22 vid peertube[126684]:         "URL//accounts/redakteur/followers"
Nov 29 12:42:22 vid peertube[126684]:       ]
Nov 29 12:42:22 vid peertube[126684]:     },
Nov 29 12:42:22 vid peertube[126684]:     "contextType": "View"
Nov 29 12:42:22 vid peertube[126684]:   },
Nov 29 12:42:22 vid peertube[126684]:   "err": {
Nov 29 12:42:22 vid peertube[126684]:     "stack": "jsonld.InvalidUrl: Dereferencing a URL did not result in a valid JSON-LD object. Possible causes are an inaccessible URL perhaps due to a same-origin policy (ensure the server uses CORS if you are using client-side JavaScript), too many redirects, a non-JSON response, or more than one HTTP Link Header was provided for a remote context.\n    at ContextResolver._fetchContext (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/ContextResolver.js:173:13)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async ContextResolver._resolveRemoteContext (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/ContextResolver.js:117:34)\n    at async ContextResolver.resolve (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/ContextResolver.js:50:22)\n    at async api.process (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/context.js:65:20)\n    at async api.expand (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/expand.js:211:17)\n    at async Function.jsonld.expand (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/jsonld.js:325:18)\n    at async Function.jsonld.toRDF (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/jsonld.js:671:16)\n    at async Function.jsonld.normalize.jsonld.canonize (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/jsonld.js:580:19)\n    at async Promise.all (index 0)",
Nov 29 12:42:22 vid peertube[126684]:     "message": "Dereferencing a URL did not result in a valid JSON-LD object. Possible causes are an inaccessible URL perhaps due to a same-origin policy (ensure the server uses CORS if you are using client-side JavaScript), too many redirects, a non-JSON response, or more than one HTTP Link Header was provided for a remote context.",
Nov 29 12:42:22 vid peertube[126684]:     "name": "jsonld.InvalidUrl",
Nov 29 12:42:22 vid peertube[126684]:     "details": {
Nov 29 12:42:22 vid peertube[126684]:       "code": "loading remote context failed",
Nov 29 12:42:22 vid peertube[126684]:       "url": "URL//ns/activitystreams",
Nov 29 12:42:22 vid peertube[126684]:       "cause": {
Nov 29 12:42:22 vid peertube[126684]:         "stack": "jsonld.LoadDocumentError: URL could not be dereferenced, an error occurred.\n    at _fetch (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/documentLoaders/node.js:184:11)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async loadDocument (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/documentLoaders/node.js:79:25)\n    at async RequestQueue.add (/var/www/peertube/versions/peertube-v3.4.1/node_modules/jsonld/lib/RequestQueue.js:33:14)",
Nov 29 12:42:22 vid peertube[126684]:         "message": "URL could not be dereferenced, an error occurred.",
Nov 29 12:42:22 vid peertube[126684]:         "name": "jsonld.LoadDocumentError",
Nov 29 12:42:22 vid peertube[126684]:         "details": {
Nov 29 12:42:22 vid peertube[126684]:           "code": "loading document failed",
Nov 29 12:42:22 vid peertube[126684]:           "url": "URL//ns/activitystreams",
Nov 29 12:42:22 vid peertube[126684]:           "cause": {
Nov 29 12:42:22 vid peertube[126684]:             "stack": "TimeoutError: Request timed out\n    at Timeout.<anonymous> (/var/www/peertube/versions/peertube-v3.4.1/node_modules/ky/umd.js:182:12)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)",
Nov 29 12:42:22 vid peertube[126684]:             "message": "Request timed out",
Nov 29 12:42:22 vid peertube[126684]:             "name": "TimeoutError",
Nov 29 12:42:22 vid peertube[126684]:             "request": {
Nov 29 12:42:22 vid peertube[126684]:               "size": 0,
Nov 29 12:42:22 vid peertube[126684]:               "follow": 20,
Nov 29 12:42:22 vid peertube[126684]:               "compress": true,
Nov 29 12:42:22 vid peertube[126684]:               "counter": 0,
Nov 29 12:42:22 vid peertube[126684]:               "agent": {
Nov 29 12:42:22 vid peertube[126684]:                 "_events": {},
Nov 29 12:42:22 vid peertube[126684]:                 "_eventsCount": 2,
Nov 29 12:42:22 vid peertube[126684]:                 "defaultPort": 443,
Nov 29 12:42:22 vid peertube[126684]:                 "protocol": "https:",
Nov 29 12:42:22 vid peertube[126684]:                 "options": {
Nov 29 12:42:22 vid peertube[126684]:                   "rejectUnauthorized": true,
Nov 29 12:42:22 vid peertube[126684]:                   "path": null
Nov 29 12:42:22 vid peertube[126684]:                 },
Nov 29 12:42:22 vid peertube[126684]:                 "requests": {},
Nov 29 12:42:22 vid peertube[126684]:                 "sockets": {},
Nov 29 12:42:22 vid peertube[126684]:                 "freeSockets": {},
Nov 29 12:42:22 vid peertube[126684]:                 "keepAliveMsecs": 1000,
Nov 29 12:42:22 vid peertube[126684]:                 "keepAlive": false,
Nov 29 12:42:22 vid peertube[126684]:                 "maxSockets": null,
Nov 29 12:42:22 vid peertube[126684]:                 "maxFreeSockets": 256,
Nov 29 12:42:22 vid peertube[126684]:                 "scheduling": "lifo",
Nov 29 12:42:22 vid peertube[126684]:                 "maxTotalSockets": null,
Nov 29 12:42:22 vid peertube[126684]:                 "totalSocketCount": 0,
Nov 29 12:42:22 vid peertube[126684]:                 "maxCachedSessions": 100,
Nov 29 12:42:22 vid peertube[126684]:                 "_sessionCache": {
Nov 29 12:42:22 vid peertube[126684]:                   "map": {},
Nov 29 12:42:22 vid peertube[126684]:                   "list": []
Nov 29 12:42:22 vid peertube[126684]:                 }
Nov 29 12:42:22 vid peertube[126684]:               },
Nov 29 12:42:22 vid peertube[126684]:               "highWaterMark": 16384,
Nov 29 12:42:22 vid peertube[126684]:               "insecureHTTPParser": false
Nov 29 12:42:22 vid peertube[126684]:             }
Nov 29 12:42:22 vid peertube[126684]:           }
Nov 29 12:42:22 vid peertube[126684]:         }
Nov 29 12:42:22 vid peertube[126684]:       }
Nov 29 12:42:22 vid peertube[126684]:     }
Nov 29 12:42:22 vid peertube[126684]:   }
Nov 29 12:42:22 vid peertube[126684]: }



Nov 29 14:56:27 vid peertube[2707]: [vid.domain.nx:443] 2021-11-29 14:56:27.643 error: Cannot execute job 184195 in queue activitypub-http-unicast. {
Nov 29 14:56:27 vid peertube[2707]:   "payload": {
Nov 29 14:56:27 vid peertube[2707]:     "uri": "URL//accounts/peertube/inbox",
Nov 29 14:56:27 vid peertube[2707]:     "signatureActorId": 1,
Nov 29 14:56:27 vid peertube[2707]:     "body": {
Nov 29 14:56:27 vid peertube[2707]:       "type": "Follow",
Nov 29 14:56:27 vid peertube[2707]:       "id": "URL//accounts/peertube/follows/4",
Nov 29 14:56:27 vid peertube[2707]:       "actor": "URL//accounts/peertube",
Nov 29 14:56:27 vid peertube[2707]:       "object": "URL//accounts/peertube"
Nov 29 14:56:27 vid peertube[2707]:     }
Nov 29 14:56:27 vid peertube[2707]:   },
Nov 29 14:56:27 vid peertube[2707]:   "err": {
Nov 29 14:56:27 vid peertube[2707]:     "stack": "HTTPError: Response code 403 (Forbidden)\n    at Request.<anonymous> (/var/www/peertube/versions/peertube-v3.4.1/node_modules/got/dist/source/as-promise/index.js:117:42)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
Nov 29 14:56:27 vid peertube[2707]:     "message": "Response code 403 (Forbidden)",
Nov 29 14:56:27 vid peertube[2707]:     "name": "HTTPError",
Nov 29 14:56:27 vid peertube[2707]:     "responseBody": "{\"type\":\"about:blank\",\"title\":\"Forbidden\",\"detail\":\"ActivityPub signature could not be checked\",\"status\":403,\"error\":\"ActivityPub signature could not be checked\"}",
Nov 29 14:56:27 vid peertube[2707]:     "statusCode": 403
Nov 29 14:56:27 vid peertube[2707]:   }
Nov 29 14:56:27 vid peertube[2707]: }


Nov 22 21:56:21 vid3 peertube[483651]: [vid3.domain.nx:443] 2021-11-22 21:56:21.339 info: 46.182.0.0 - - [22/Nov/2021:21:56:21 +0000] "GET /download/streaming-playlists/hls/videos/f421688c-6b9a-4d2c-b9b6-771dc128d934-1080-fragmented.mp4 HTTP/1.0" 200 52787789 "URL//" "Chrome Mobile"
Nov 22 21:56:24 vid3 peertube[483651]: [vid3.domain.nx:443] 2021-11-22 21:56:24.079 error: Cannot execute job 129673 in queue activitypub-http-broadcast. {
Nov 22 21:56:24 vid3 peertube[483651]:   "payload": {
Nov 22 21:56:24 vid3 peertube[483651]:     "uris": [
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:       <snip>
Nov 22 21:56:24 vid3 peertube[483651]:     ],
Nov 22 21:56:24 vid3 peertube[483651]:     "body": {
Nov 22 21:56:24 vid3 peertube[483651]:       "id": "URL//accounts/peertube/views/videos/2477/2021-11-22T16:50:54.715Z",
Nov 22 21:56:24 vid3 peertube[483651]:       "type": "View",
Nov 22 21:56:24 vid3 peertube[483651]:       "actor": "URL//accounts/peertube",
Nov 22 21:56:24 vid3 peertube[483651]:       "object": "URL//videos/watch/fe76a4df-7078-4662-8ba7-50a5e2dea3fa",
Nov 22 21:56:24 vid3 peertube[483651]:       "to": [
Nov 22 21:56:24 vid3 peertube[483651]:         "URL//accounts/redakteur"
Nov 22 21:56:24 vid3 peertube[483651]:       ],
Nov 22 21:56:24 vid3 peertube[483651]:       "cc": [
Nov 22 21:56:24 vid3 peertube[483651]:         "URL//video-channels/redaktion/followers",
Nov 22 21:56:24 vid3 peertube[483651]:         "URL//accounts/peertube/followers",
Nov 22 21:56:24 vid3 peertube[483651]:         "URL//accounts/redakteur/followers"
Nov 22 21:56:24 vid3 peertube[483651]:       ],
Nov 22 21:56:24 vid3 peertube[483651]:       "@context": [
Nov 22 21:56:24 vid3 peertube[483651]:         "URL//ns/activitystreams",
Nov 22 21:56:24 vid3 peertube[483651]:         "URL//security/v1",
Nov 22 21:56:24 vid3 peertube[483651]:         {
Nov 22 21:56:24 vid3 peertube[483651]:           "RsaSignature2017": "URL//security#RsaSignature2017"
Nov 22 21:56:24 vid3 peertube[483651]:         }
Nov 22 21:56:24 vid3 peertube[483651]:       ],
Nov 22 21:56:24 vid3 peertube[483651]:       "signature": {
Nov 22 21:56:24 vid3 peertube[483651]:         "type": "RsaSignature2017",
Nov 22 21:56:24 vid3 peertube[483651]:         "creator": "URL//accounts/peertube",
Nov 22 21:56:24 vid3 peertube[483651]:         "created": "2021-11-22T16:50:54.721Z",
Nov 22 21:56:24 vid3 peertube[483651]:         "signatureValue": "YLonRu3VOJaUYFyqp/pn2+kePOdJYFfydCZTufMEdCDL3YyM8agV1z2GiDmz3fRSXZvRfovU6mNfME7WNqZl29CAZXgTpU+mi9kjtbipLkdSOgFWzz6kRpooVmbpSWyJvtAWO8vyKKvjnxHQIty5d8n0WtVQ0F32/0FUlgRuQm6ZgWup8IFf0Hrc0Mjgv93ABeR96ZdTXNzzrRBk7lWJXiF4Q0/xNd2U7gjxzlXQLzPImwMLKGZ3TPX3CD+Q4AAZcgfIb/I8+Qi/d95XGXBR2icJPRKV5Temdqivld/CHEf8dvk8IvYOMzJlWVFw+zPl2+9es3pnMP/CoX1HI7AXcg=="
Nov 22 21:56:24 vid3 peertube[483651]:       }
Nov 22 21:56:24 vid3 peertube[483651]:     }
Nov 22 21:56:24 vid3 peertube[483651]:   },
Nov 22 21:56:24 vid3 peertube[483651]:   "err": {
Nov 22 21:56:24 vid3 peertube[483651]:     "stack": "TimeoutError: Promise timed out after 600000 milliseconds\n    at Timeout._onTimeout (/var/www/peertube/versions/peertube-v3.4.1/node_modules/p-timeout/index.js:34:63)\n    at listOnTimeout (node:internal/timers:557:17)\n    at processTimers (node:internal/timers:500:7)",
Nov 22 21:56:24 vid3 peertube[483651]:     "message": "Promise timed out after 600000 milliseconds",
Nov 22 21:56:24 vid3 peertube[483651]:     "name": "TimeoutError"
Nov 22 21:56:24 vid3 peertube[483651]:   }
Nov 22 21:56:24 vid3 peertube[483651]: }



Nov 23 09:17:01 vid3 peertube[2083]:   "err": {
Nov 23 09:17:01 vid3 peertube[2083]:     "stack": "TimeoutError: Promise timed out after 600000 milliseconds\n    at Timeout._onTimeout (/var/www/peertube/versions/peertube-v3.4.1/node_modules/p-timeout/index.js:34:63)\n    at listOnTimeout (node:internal/timers:557:17)\n    at processTimers (node:internal/timers:500:7)",
Nov 23 09:17:01 vid3 peertube[2083]:     "message": "Promise timed out after 600000 milliseconds",
Nov 23 09:17:01 vid3 peertube[2083]:     "name": "TimeoutError"
Nov 23 09:17:01 vid3 peertube[2083]:   }
Nov 23 09:17:01 vid3 peertube[2083]: }
Nov 23 09:17:01 vid3 peertube[2083]: [vid3.domain.nx:443] 2021-11-23 09:17:01.052 info: Processing ActivityPub broadcast in job 129751.
Nov 23 09:17:22 vid3 peertube[2083]: [vid3.domain.nx:443] 2021-11-23 09:17:22.938 warn: Warning in tracker. {
Nov 23 09:17:22 vid3 peertube[2083]:   "err": {
Nov 23 09:17:22 vid3 peertube[2083]:     "stack": "Error: no peer with that `to_peer_id`\n    at /var/www/peertube/versions/peertube-v3.4.1/node_modules/bittorrent-tracker/server.js:554:41\n    at /var/www/peertube/versions/peertube-v3.4.1/node_modules/bittorrent-tracker/server.js:364:7\n    at processTicksAndRejections (node:internal/process/task_queues:78:11)",
Nov 23 09:17:22 vid3 peertube[2083]:     "message": "no peer with that `to_peer_id`"
Nov 23 09:17:22 vid3 peertube[2083]:   }
Nov 23 09:17:22 vid3 peertube[2083]: }


Nov 24 20:27:28 vid3 peertube[39515]: [vid3.domain.nx:443] 2021-11-24 20:27:28.631 info: 94.134.109.201 - - [24/Nov/2021:20:27:28 +0000] "POST /api/v1/videos/098ec799-e70a-42a8-bb83-18b5f50d8183/views HTTP/1.0" 204 - "URL//videos/embed/098ec799-e70a-42a8-bb83-18b5f50d8183" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 Edg/96.0.1054.29"
Nov 24 20:27:28 vid3 peertube[39515]: [vid3.domain.nx:443] 2021-11-24 20:27:28.636 info: Processing ActivityPub broadcast in job 152405.
Nov 24 20:27:28 vid3 peertube[39515]: [vid3.domain.nx:443] 2021-11-24 20:27:28.658 error: Cannot execute job 152405 in queue activitypub-http-broadcast. {
Nov 24 20:27:28 vid3 peertube[39515]:   "payload": {
Nov 24 20:27:28 vid3 peertube[39515]:     "uris": [
Nov 24 20:27:28 vid3 peertube[39515]:       "URL//inbox"
Nov 24 20:27:28 vid3 peertube[39515]:     ],
Nov 24 20:27:28 vid3 peertube[39515]:     "signatureActorId": 1,
Nov 24 20:27:28 vid3 peertube[39515]:     "body": {
Nov 24 20:27:28 vid3 peertube[39515]:       "id": "URL//accounts/peertube/views/videos/2498/2021-11-24T20:27:28.621Z",
Nov 24 20:27:28 vid3 peertube[39515]:       "type": "View",
Nov 24 20:27:28 vid3 peertube[39515]:       "actor": "URL//accounts/peertube",
Nov 24 20:27:28 vid3 peertube[39515]:       "object": "URL//videos/watch/098ec799-e70a-42a8-bb83-18b5f50d8183",
Nov 24 20:27:28 vid3 peertube[39515]:       "to": [
Nov 24 20:27:28 vid3 peertube[39515]:         "URL//ns/activitystreams#Public",
Nov 24 20:27:28 vid3 peertube[39515]:         "URL//video-channels/redaktion/followers",
Nov 24 20:27:28 vid3 peertube[39515]:         "URL//accounts/peertube/followers",
Nov 24 20:27:28 vid3 peertube[39515]:         "URL//accounts/redakteur/followers"
Nov 24 20:27:28 vid3 peertube[39515]:       ],
Nov 24 20:27:28 vid3 peertube[39515]:       "cc": []
Nov 24 20:27:28 vid3 peertube[39515]:     },
Nov 24 20:27:28 vid3 peertube[39515]:     "contextType": "View"
Nov 24 20:27:28 vid3 peertube[39515]:   },
Nov 24 20:27:28 vid3 peertube[39515]:   "err": {
Nov 24 20:27:28 vid3 peertube[39515]:     "stack": "Error: error:0909006C:PEM routines:get_name:no start line\n    at Sign.sign (internal/crypto/sig.js:110:29)\n    at Object.<anonymous> (/var/www/peertube/versions/peertube-v3.4.1/dist/server/helpers/peertube-crypto.js:88:37)\n    at Generator.next (<anonymous>)\n    at fulfilled (/var/www/peertube/versions/peertube-v3.4.1/node_modules/tslib/tslib.js:114:62)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
Nov 24 20:27:28 vid3 peertube[39515]:     "message": "error:0909006C:PEM routines:get_name:no start line",
Nov 24 20:27:28 vid3 peertube[39515]:     "library": "PEM routines",
Nov 24 20:27:28 vid3 peertube[39515]:     "function": "get_name",
Nov 24 20:27:28 vid3 peertube[39515]:     "reason": "no start line",
Nov 24 20:27:28 vid3 peertube[39515]:     "code": "ERR_OSSL_PEM_NO_START_LINE"
Nov 24 20:27:28 vid3 peertube[39515]:   }
Nov 24 20:27:28 vid3 peertube[39515]: }


Nov 25 14:21:16 vid peertube[23018]: [vid.domain.nx:443] 2021-11-25 14:21:16.531 info: Processing ActivityPub follow in job 23.
Nov 25 14:21:16 vid peertube[23018]: [vid.domain.nx:443] 2021-11-25 14:21:16.656 info: Fetching remote actor URL//accounts/peertube.
Nov 25 14:21:16 vid peertube[23018]: [vid.domain.nx:443] 2021-11-25 14:21:16.679 error: Cannot execute job 23 in queue activitypub-follow. {
Nov 25 14:21:16 vid peertube[23018]:   "payload": {
Nov 25 14:21:16 vid peertube[23018]:     "host": "vid3.domain.nx:443",
Nov 25 14:21:16 vid peertube[23018]:     "name": "peertube",
Nov 25 14:21:16 vid peertube[23018]:     "followerActorId": 1
Nov 25 14:21:16 vid peertube[23018]:   },
Nov 25 14:21:16 vid peertube[23018]:   "err": {
Nov 25 14:21:16 vid peertube[23018]:     "stack": "Error: Cannot fetch remote actor URL//accounts/peertube\n    at Object.<anonymous> (/var/www/peertube/versions/peertube-v3.4.1/dist/server/lib/activitypub/actors/get.js:21:23)\n    at Generator.next (<anonymous>)\n    at fulfilled (/var/www/peertube/versions/peertube-v3.4.1/node_modules/tslib/tslib.js:114:62)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
Nov 25 14:21:16 vid peertube[23018]:     "message": "Cannot fetch remote actor URL//accounts/peertube"
Nov 25 14:21:16 vid peertube[23018]:   }
Nov 25 14:21:16 vid peertube[23018]: }

The federation was reset several times during the debugging. We also removed old federation settings from the databases to make sure that the federation is initialized correctly.

Our installation is running on Ubuntu 20.04.3 LTS with npm 6.14.15 (we tried downgrading from npm 8.2.0), yarn 1.22.15, redis 5.0.7, nginx 1.18.0 and postgresql 12.9.

Any hints would be greatly appreciated.

Greetings

iTap

Hello,

It’s coming from the bittorrent tracker and is not very important. See https://github.com/webtorrent/bittorrent-tracker/issues/384

Question ii.) Where do these requests come from? A video with this « infoHash » does not exist on the instance and never did. Occurs on a standalone instance that used to be federated, but is no longer. Occurs regularly over and over again with different infoHashes

From the bittorrent tracker. When someone else tries to use your tracker with files your server does not know, or could also be sent by valid peertube clients when the video infohash changes (before/after transcoding for example).

Regarding your federation issues, it seems your server has connectivity problems:

  • ESERVFAIL means there is an issue reaching the DNS for this domain
  • jsonld.InvalidUrl means your server cannot reach the URL in the details (it could also be the server behind URL//ns/activitystreams that have issues, but it’s rare)
  • TimeoutError is a timeout error :o) But try peertube v4, we decreased timeout values