I want to start this post by saying thanks to everyone working on this great project. I also want to come with a couple disclaimers:
- I didn’t follow the official tutorial exactly as I already have a nginx reverse proxy and a certbot Docker instance.
- I am doing this completely on a hobby basis for the fun of it, and do not have a public instance.
That said, my issue is the following: I set up a couple local machines as remote runners, and a lot of times the job gets stuck because the runner downloads it, transcodes successfully but then fails to upload it fully, and then the job gets stuck just saying « PROCESSING ».
The log of prunner.service then says
Sep 15 14:45:40 oldie prunner[15981]: "message": "connect ENETUNREACH [IPv6 address here]",
Sep 15 14:45:40 oldie prunner[15981]: "stack":
Sep 15 14:45:40 oldie prunner[15981]: Error: connect ENETUNREACH [IPv6 address here]
Sep 15 14:45:40 oldie prunner[15981]: at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1549:16)
Sep 15 14:45:40 oldie prunner[15981]: "errno": -101,
Sep 15 14:45:40 oldie prunner[15981]: "code": "ENETUNREACH",
Sep 15 14:45:40 oldie prunner[15981]: "syscall": "connect",
Sep 15 14:45:40 oldie prunner[15981]: "address": "[IPv6 address here]",
Sep 15 14:45:40 oldie prunner[15981]: "port": 443
Sep 15 14:45:40 oldie prunner[15981]: }
The strange thing is that sometimes it does go through, so if I restart prunner.service enough times I can get it to finish. It does take a lot of time though because it redoes downloading and transcoding too because it doesn’t keep the job until it is confirmed uploaded successfully. The actual issue seems to be the following:
That is a screenshot of both of my local runners, and they’re both on the same network. However, randomly PeerTube thinks the runners are on the Docker gateway network and randomly not, so whenever I refresh that page I never know whether it’s going to list the real outbound IP for the runner or the internal Docker gateway IP. I suspect that’s why the uploads fail, because somehow the nginx reverse proxy doesn’t proxy correctly at all times.
This is what I think are the relevant portions of my nginx config, just switched out the domain name to example.com to anonymize it:
# example.com/
upstream example.com {
# Container: docker-peertube-1
# networks:
# docker_default (reachable)
# IP address: 172.18.0.42
# exposed ports (first ten): 1935/tcp 9000/tcp
# default port: 80
# using port: 9000
# /!\ WARNING: Virtual port published on host. Clients
# might be able to bypass nginx-proxy and
# access the container's server directly.
server 172.18.0.42:9000;
}
server {
server_name example.com;
access_log /var/log/nginx/access.log vhost;
listen 80 ;
# Do not HTTPS redirect Let's Encrypt ACME challenge
location ^~ /.well-known/acme-challenge/ {
auth_basic off;
auth_request off;
allow all;
root /usr/share/nginx/html;
try_files $uri =404;
break;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
server_name example.com;
access_log /var/log/nginx/access.log vhost;
http2 on;
listen 443 ssl ;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_certificate /etc/nginx/certs/example.com.crt;
ssl_certificate_key /etc/nginx/certs/example.com.key;
ssl_dhparam /etc/nginx/certs/example.com.dhparam.pem;
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/certs/example.com.chain.pem;
set $sts_header "";
if ($https) {
set $sts_header "max-age=31536000";
}
add_header Strict-Transport-Security $sts_header always;
include /etc/nginx/vhost.d/example.com;
location / {
proxy_pass http://example.com;
set $upstream_keepalive false;
}
}
# configuration file /etc/nginx/vhost.d/example.com:
# Minimum Nginx version required: 1.13.0 (released Apr 25, 2017)
# Please check your Nginx installation features the following modules via 'nginx -V':
# STANDARD HTTP MODULES: Core, Proxy, Rewrite, Access, Gzip, Headers, HTTP/2, Log, Real IP, SSL, Thread Pool, Upstream, AIO Multithreading.
# THIRD PARTY MODULES: None.
##
# Performance optimizations
# For extra performance please refer to https://github.com/denji/nginx-tuning
##
# root /var/www/peertube/storage;
#
# Enable compression for JS/CSS/HTML, for improved client load times.
# It might be nice to compress JSON/XML as returned by the API, but
# leaving that out to protect against potential BREACH attack.
gzip on;
gzip_vary on;
gzip_types # text/html is always compressed by HttpGzipModule
text/css
application/javascript
font/truetype
font/opentype
application/vnd.ms-fontobject
image/svg+xml;
gzip_min_length 1000; # default is 20 bytes
gzip_buffers 16 8k;
gzip_comp_level 2; # default is 1
client_body_timeout 30s; # default is 60
client_header_timeout 10s; # default is 60
send_timeout 10s; # default is 60
keepalive_timeout 10s; # default is 75
resolver_timeout 10s; # default is 30
reset_timedout_connection on;
proxy_ignore_client_abort on;
tcp_nopush on; # send headers in one piece
tcp_nodelay on; # don't buffer data sent, good for small data bursts in real time
##
# Application
##
location @api {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
client_max_body_size 100k; # default is 1M
proxy_connect_timeout 10m;
proxy_send_timeout 10m;
proxy_read_timeout 10m;
send_timeout 10m;
proxy_pass http://example.com;
}
location ~ ^/api/v1/videos/(upload-resumable|([^/]+/source/replace-resumable))$ {
client_max_body_size 0;
proxy_request_buffering off;
try_files /dev/null @api;
}
location ~ ^/api/v1/users/[^/]+/imports/import-resumable$ {
client_max_body_size 0;
proxy_request_buffering off;
try_files /dev/null @api;
}
location ~ ^/api/v1/videos/(upload|([^/]+/studio/edit))$ {
limit_except POST HEAD { deny all; }
# This is the maximum upload size, which roughly matches the maximum size of a video file.
# Note that temporary space is needed equal to the total size of all concurrent uploads.
# This data gets stored in /var/lib/nginx by default, so you may want to put this directory
# on a dedicated filesystem.
client_max_body_size 12G; # default is 1M
add_header X-File-Maximum-Size 8G always; # inform backend of the set value in bytes before mime-encoding (x * 1.4 >= client_max_body_size)
try_files /dev/null @api;
}
location ~ ^/api/v1/runners/jobs/[^/]+/(update|success)$ {
client_max_body_size 12G; # default is 1M
add_header X-File-Maximum-Size 8G always; # inform backend of the set value in bytes before mime-encoding (x * 1.4 >= client_max_body_size)
try_files /dev/null @api;
}
location ~ ^/api/v1/(videos|video-playlists|video-channels|users/me) {
client_max_body_size 6M; # default is 1M
add_header X-File-Maximum-Size 4M always; # inform backend of the set value in bytes before mime-encoding (x * 1.4 >= client_max_body_size)
try_files /dev/null @api;
}
##
# Websocket
##
location @api_websocket {
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://example.com;
}
location /socket.io {
try_files /dev/null @api_websocket;
}
location /tracker/socket {
# Peers send a message to the tracker every 15 minutes
# Don't close the websocket before then
proxy_read_timeout 15m; # default is 60s
try_files /dev/null @api_websocket;
}
# Plugin websocket routes
location ~ ^/plugins/[^/]+(/[^/]+)?/ws/ {
try_files /dev/null @api_websocket;
}
# configuration file /etc/nginx/conf.d/my_custom_proxy_settings.conf:
client_max_body_size 4096m;