Peertube Remote storage (S3)

It works, but I think it’s not yet optimal.

I ran a live for several hours and checking what’s happening here and there. The live was normal without saving for playback.

I’ve noticed that .ts files stay in the bucket for a minute before being deleted. In my current test, there is always the last 15 .ts files in the bucket which is the data received in the last minute. Not caching .ts files at all generates unnecessary traffic between the bucket and the reverse proxy. But still not figured out how to create two different caches…
0.m3u8 and segments-sha256.json will be modified in the bucket at each new .ts, so if we want to cache it, not more than three or four secends max (but not tested)!
master.m3u8 will never change after the live strated, we can cache it.

Still working on it…

Cache won’t work with the config of my previous posts.
Here is the correction :

proxy_cache_path /var/cache/s3 levels=1:2 keys_zone=CACHE-S3:100m inactive=48h max_size=10G;
proxy_cache_path /var/cache/s3-ts levels=1:2 keys_zone=CACHE-S3-TS:10m inactive=60s max_size=1G;

server {
  listen 80;
  server_name peertube.tld;
  root /var/www/html;
  location / { return 301 https://$host$request_uri; }
}

server {
  listen 443 ssl;
  http2 on;
  server_name peertube.tld;

  access_log /var/log/nginx/medias.access.log; # reduce I/0 with buffer=10m flush=5m
  error_log  /var/log/nginx/medias.error.log;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;
  ssl_session_tickets off;

  ssl_certificate     /etc/letsencrypt/live/peertube.tld/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/peertube.tld/privkey.pem;

  root /var/www/html;

  keepalive_timeout 30;

  location = / {
    index index.html;
  }

  location / {
    try_files $uri @s3;
  }

  location ~ \.ts$ {
    try_files $uri @s3-ts;
  }

  location ~ \.(json|m3u8)$ {
    try_files $uri @s3_nocache;
  }

  set $s3_backend 'https://my-bucket.s3.bhs.perf.cloud.ovh.net';

  location @s3 {
    limit_except GET OPTIONS {
        deny all;
    }

    resolver 1.1.1.1 8.8.8.8 208.67.222.222 208.67.220.220;
    proxy_set_header Host my-bucket.s3.bhs.perf.cloud.ovh.net;
    proxy_set_header Connection '';
    proxy_set_header Authorization '';
    proxy_set_header Range $slice_range;
    proxy_hide_header Set-Cookie;
    proxy_hide_header 'Access-Control-Allow-Origin';
    proxy_hide_header 'Access-Control-Allow-Methods';
    proxy_hide_header 'Access-Control-Allow-Headers';
    proxy_hide_header x-amz-id-2;
    proxy_hide_header x-amz-request-id;
    proxy_hide_header x-amz-meta-server-side-encryption;
    proxy_hide_header x-amz-server-side-encryption;
    proxy_hide_header x-amz-bucket-region;
    proxy_hide_header x-amzn-requestid;
    proxy_ignore_headers Set-Cookie;
    proxy_pass $s3_backend$uri;
    proxy_intercept_errors off;

    proxy_cache CACHE-S3;
    proxy_cache_valid 200 206 48h;     # <---- Needs 206 too
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    slice              1m;
    proxy_cache_key    $host$uri$is_args$args$slice_range;
    proxy_http_version 1.1;

    expires 1y;
    add_header Cache-Control public;
    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS';
    add_header 'Access-Control-Allow-Headers' 'Range,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;
    add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
  }

location @s3-ts {
    limit_except GET OPTIONS {
        deny all;
    }

    resolver 1.1.1.1 8.8.8.8 208.67.222.222 208.67.220.220;
    proxy_set_header Host my-bucket.s3.bhs.perf.cloud.ovh.net;
    proxy_set_header Connection '';
    proxy_set_header Authorization '';
    proxy_set_header Range $slice_range;
    proxy_hide_header Set-Cookie;
    proxy_hide_header 'Access-Control-Allow-Origin';
    proxy_hide_header 'Access-Control-Allow-Methods';
    proxy_hide_header 'Access-Control-Allow-Headers';
    proxy_hide_header x-amz-id-2;
    proxy_hide_header x-amz-request-id;
    proxy_hide_header x-amz-meta-server-side-encryption;
    proxy_hide_header x-amz-server-side-encryption;
    proxy_hide_header x-amz-bucket-region;
    proxy_hide_header x-amzn-requestid;
    proxy_ignore_headers Set-Cookie;
    proxy_pass $s3_backend$uri;
    proxy_intercept_errors off;

    proxy_cache CACHE-S3-TS;
    proxy_cache_valid 200 206 2m;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    slice              1m;
    proxy_cache_key    $host$uri$is_args$args$slice_range;
    proxy_http_version 1.1;

    expires 1y;
    add_header Cache-Control public;
    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS';
    add_header 'Access-Control-Allow-Headers' 'Range,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;
    add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
  }

  location @s3_nocache {
    limit_except GET OPTIONS {
        deny all;
    }

    resolver 1.1.1.1 8.8.8.8 208.67.222.222 208.67.220.220;
    proxy_set_header Host my-bucket.s3.bhs.perf.cloud.ovh.net;
    proxy_set_header Connection '';
    proxy_set_header Authorization '';
    proxy_set_header Range $http_range;
    proxy_hide_header Set-Cookie;
    proxy_hide_header 'Access-Control-Allow-Origin';
    proxy_hide_header 'Access-Control-Allow-Methods';
    proxy_hide_header 'Access-Control-Allow-Headers';
    proxy_hide_header x-amz-id-2;
    proxy_hide_header x-amz-request-id;
    proxy_hide_header x-amz-meta-server-side-encryption;
    proxy_hide_header x-amz-server-side-encryption;
    proxy_hide_header x-amz-bucket-region;
    proxy_hide_header x-amzn-requestid;
    proxy_ignore_headers Set-Cookie;
    proxy_pass $s3_backend$uri;
    proxy_intercept_errors off;

    expires 0;
    proxy_cache off;

    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS';
    add_header 'Access-Control-Allow-Headers' 'Range,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;
    add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
  }
}

Hi, I am still stuck with my issue: can not play a video a video ( size >1G, stored in S3 object storage )

I found your post, and it contains the same key words: forwards Range HTTP header . I added your line:

  location {
    proxy_set_header Range $http_range;
}

Then my server is down.

I am using yunohost to host peertube and wasabi S3. And I got two nginx files:


map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    listen [::]:80;
    server_name qien.tv;

    access_by_lua_file /usr/share/ssowat/access.lua;

    include /etc/nginx/conf.d/acme-challenge.conf.inc;

    location ^~ '/.well-known/ynh-diagnosis/' {
        alias /var/www/.well-known/ynh-diagnosis/;
    }


    location ^~ '/.well-known/autoconfig/mail/' {
        alias /var/www/.well-known/qien.tv/autoconfig/mail/;
    }




    location / {
        return 301 https://$host$request_uri;
    }



    include /etc/nginx/conf.d/yunohost_http_errors.conf.inc;

    access_log /var/log/nginx/qien.tv-access.log;
    error_log /var/log/nginx/qien.tv-error.log;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

  GNU nano 5.4                            qien.tv.conf                                      
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name qien.tv;

    include /etc/nginx/conf.d/security.conf.inc;

    ssl_certificate /etc/yunohost/certs/qien.tv/crt.pem;
    ssl_certificate_key /etc/yunohost/certs/qien.tv/key.pem;


    more_set_headers "Strict-Transport-Security : max-age=63072000; includeSubDomains; prel>


    # OCSP settings
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/yunohost/certs/qien.tv/crt.pem;
    resolver 1.1.1.1 9.9.9.9 valid=300s;
    resolver_timeout 5s;



    location ^~ '/.well-known/autoconfig/mail/' {
        alias /var/www/.well-known/qien.tv/autoconfig/mail/;
    }


    access_by_lua_file /usr/share/ssowat/access.lua;

    include /etc/nginx/conf.d/qien.tv.d/*.conf;

    include /etc/nginx/conf.d/yunohost_sso.conf.inc;
    include /etc/nginx/conf.d/yunohost_admin.conf.inc;
    include /etc/nginx/conf.d/yunohost_api.conf.inc;
    include /etc/nginx/conf.d/yunohost_http_errors.conf.inc;

    access_log /var/log/nginx/qien.tv-access.log;
    error_log /var/log/nginx/qien.tv-error.log;

and I found another nginx config file here:

more_set_headers "X-Frame-Options : ALLOWALL";                                              
                                                                                            
##                                                                                          
# Application                                                                               
##                                                                                          
                                                                                            
location @api {                                                                             
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;                              
  proxy_set_header Host            $host;                                                   
  proxy_set_header X-Real-IP       $remote_addr;                                            
                                                                                            
  client_max_body_size  100k; # default is 1M                                               
                                                                                            
  proxy_connect_timeout 10m;                                                                
  proxy_send_timeout    10m;                                                                
  proxy_read_timeout    10m;                                                                
  send_timeout          10m;                                                                
                                                                                            
  proxy_pass http://127.0.0.1:8095;                                                         
}                                                                                           
                                                                                            
location = /api/v1/videos/upload-resumable {                                                
  client_max_body_size    0;                                                                
  proxy_request_buffering off;                                                              
                                                                                            
  try_files /dev/null @api;                                                                 
}                                                                                           
                                                                                            
location / {                                                                                
  try_files /dev/null @api;                                                                 
  # Include SSOWAT user panel.                                                              
  include conf.d/yunohost_panel.conf.inc;                                                   
}                                                                                           
                                                                                            
location ~ ^/api/v1/videos/(upload|([^/]+/studio/edit))$ {                                  
  limit_except POST HEAD { deny all; }                                                      
                                                                                            
  # This is the maximum upload size, which roughly matches the maximum size of a video file.
  # Note that temporary space is needed equal to the total size of all concurrent uploads.  
  # This data gets stored in /var/lib/nginx by default, so you may want to put this directo>

  # on a dedicated filesystem.                                                              
  client_max_body_size                      24G; # default is 1M                            
  more_set_headers "X-File-Maximum-Size : 16G always"; # inform backend of the set value in>
                                                                                            
  try_files /dev/null @api;                                                                 
}                                                                                           
                                                                                            
location ~ ^/api/v1/runners/jobs/[^/]+/(update|success)$ {                                  
  client_max_body_size                      12G; # default is 1M                            
  more_set_headers "X-File-Maximum-Size : 8G always"; # inform backend of the set value in >
                                                                                            
  try_files /dev/null @api;                                                                 
}                                                                                           
                                                                                            
location ~ ^/api/v1/(videos|video-playlists|video-channels|users/me) {                      
  client_max_body_size                      6M; # default is 1M                             
  more_set_headers "X-File-Maximum-Size : 4M always"; # inform backend of the set value in >
                                                                                            
  try_files /dev/null @api;                                                                 
}                                                                                           
                                                                                            
##                                                                                          
# Websocket                                                                                 
##                                                                                          
                                                                                            
location @api_websocket {                                                                   
  proxy_http_version 1.1;                                                                   
  proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;                            
  proxy_set_header   Host            $host;                                                 
  proxy_set_header   X-Real-IP       $remote_addr;                                          
  proxy_set_header   Upgrade         $http_upgrade;                                         
  proxy_set_header   Connection      "upgrade";                                             
                                                                                            
  proxy_pass http://127.0.0.1:8095;                                                         
}                                                                                           
                                                                                            
location /socket.io {                                                                       
  try_files /dev/null @api_websocket;                                                       


}                                                                                           
                                                                                            
# Plugin websocket routes                                                                   
location ~ ^/plugins/[^/]+(/[^/]+)?/ws/ {                                                   
  try_files /dev/null @api_websocket;                                                       
}                                                                                           
                                                                                            
##                                                                                          
# Performance optimizations                                                                 
# For extra performance please refer to https://github.com/denji/nginx-tuning               
##                                                                                          
                                                                                            
root /home/yunohost.app/peertube/storage;                                                   
                                                                                            
# Enable compression for JS/CSS/HTML, for improved client load times.                       
# It might be nice to compress JSON/XML as returned by the API, but                         
# leaving that out to protect against potential BREACH attack.                              
# gzip              on;                                                                     
gzip_vary         on;                                                                       
gzip_types        # text/html is always compressed by HttpGzipModule                        
                  text/css                                                                  
                  application/javascript                                                    
                  font/truetype                                                             
                  font/opentype                                                             
                  application/vnd.ms-fontobject                                             
                  image/svg+xml;                                                            
gzip_min_length   1000; # default is 20 bytes                                               
gzip_buffers      16 8k;                                                                    
gzip_comp_level   2; # default is 1                                                         
                                                                                            
client_body_timeout       30s; # default is 60                                              
client_header_timeout     10s; # default is 60                                              
send_timeout              10s; # default is 60                                              
keepalive_timeout         10s; # default is 75                                              
# resolver_timeout          10s; # default is 30                                            
reset_timedout_connection on;                                                               
proxy_ignore_client_abort on;                                                               
                                                                                            
tcp_nopush                on; # send headers in one piece                                   
tcp_nodelay               on; # don't buffer data sent, good for small data bursts in real >


# If you have a small /var/lib partition, it could be interesting to store temp nginx uploa>
# See https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_temp_path        
#client_body_temp_path /var/www/peertube/storage/nginx/;                                    
                                                                                            
# Bypass PeerTube for performance reasons. Optional.                                        
# Should be consistent with client-overrides assets list in /server/controllers/client.ts   
location ~ ^/client/(assets/images/(icons/icon-36x36\.png|icons/icon-48x48\.png|icons/icon->
  more_set_headers "Cache-Control : public, max-age=31536000, immutable"; # Cache 1 year    
                                                                                            
  try_files /home/yunohost.app/peertube/storage/client-overrides/$1 /var/www/peertube/clien>
}                                                                                           
                                                                                            
# Bypass PeerTube for performance reasons. Optional.                                        
location ~ ^/client/(.*\.(js|css|png|svg|woff2|otf|ttf|woff|eot))$ {                        
  more_set_headers "Cache-Control : public, max-age=31536000, immutable"; # Cache 1 year    
                                                                                            
  alias /var/www/peertube/client/dist/$1;                                                   
}                                                                                           
                                                                                            
# Bypass PeerTube for performance reasons. Optional.                                        
location ~ ^/static/(thumbnails|avatars)/ {                                                 
  if ($request_method = 'OPTIONS') {                                                        
    more_set_headers "Access-Control-Allow-Origin : *";                                     
    more_set_headers "Access-Control-Allow-Methods : GET, OPTIONS";                         
    more_set_headers "Access-Control-Allow-Headers : Range,DNT,X-CustomHeader,Keep-Alive,Us>
    more_set_headers "Access-Control-Max-Age : 1728000"; # Preflight request can be cached >
    more_set_headers "Content-Type : text/plain charset=UTF-8";                             
    more_set_headers "Content-Length : 0";                                                  
    return 204;                                                                             
  }                                                                                         
                                                                                            
  more_set_headers "Access-Control-Allow-Origin : *";                                       
  more_set_headers "Access-Control-Allow-Methods : GET, OPTIONS";                           
  more_set_headers "Access-Control-Allow-Headers : Range,DNT,X-CustomHeader,Keep-Alive,User>
  more_set_headers "Cache-Control : public, max-age=7200"; # Cache response 2 hours         
                                                                                            
  rewrite ^/static/(.*)$ /$1 break;                                                         

  root /home/yunohost.app/peertube/storage;                                                 
                                                                                            
  try_files $uri @api;                                                                      
}                                                                                           
                                                                                            
location ~ ^(/static/(webseed|web-videos|streaming-playlists)/private/)|^/download {        
  # We can't rate limit a try_files directive, so we need to duplicate @api                 
                                                                                            
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;                              
  proxy_set_header Host            $host;                                                   
  proxy_set_header X-Real-IP       $remote_addr;                                            
                                                                                            
  proxy_limit_rate 5M;                                                                      
                                                                                            
  proxy_pass http://127.0.0.1:8095;                                                         
}                                                                                           
                                                                                            
# Bypass PeerTube for performance reasons. Optional.                                        
location ~ ^/static/(webseed|web-videos|redundancy|streaming-playlists)/ {                  
  limit_rate_after            5M;                                                           
                                                                                            
  set $peertube_limit_rate  5M;                                                             
                                                                                            
  # Use this line with nginx >= 1.17.0                                                      
  limit_rate $peertube_limit_rate;                                                          
  # Or this line with nginx < 1.17.0                                                        
  # set $limit_rate $peertube_limit_rate;                                                   
                                                                                            
  if ($request_method = 'OPTIONS') {                                                        
    more_set_headers "Access-Control-Allow-Origin : *";                                     
    more_set_headers "Access-Control-Allow-Methods : GET, OPTIONS";                         
    more_set_headers "Access-Control-Allow-Headers : Range,DNT,X-CustomHeader,Keep-Alive,Us>
    more_set_headers "Access-Control-Max-Age : 1728000"; # Preflight request can be cached >
    more_set_headers "Content-Type : text/plain charset=UTF-8";                             
    more_set_headers "Content-Length : 0";                                                  
    return 204;     

 }                                                                                         
                                                                                            
  if ($request_method = 'GET') {                                                            
    more_set_headers "Access-Control-Allow-Origin : *";                                     
    more_set_headers "Access-Control-Allow-Methods : GET, OPTIONS";                         
    more_set_headers "Access-Control-Allow-Headers : Range,DNT,X-CustomHeader,Keep-Alive,Us>
                                                                                            
    # Don't spam access log file with byte range requests                                   
    access_log off;                                                                         
  }                                                                                         
                                                                                            
  # Enabling the sendfile directive eliminates the step of copying the data into the buffer 
  # and enables direct copying data from one file descriptor to another.                    
  sendfile on;                                                                              
  sendfile_max_chunk 1M; # prevent one fast connection from entirely occupying the worker p>
  aio threads;                                                                              
                                                                                            
  # web-videos is the name of the directory mapped to the `storage.web_videos` key in your >
  rewrite ^/static/webseed/(.*)$ /web-videos/$1 break;                                      
  rewrite ^/static/(.*)$         /$1        break;                                          
                                                                                            
  root /home/yunohost.app/peertube/storage;                                                 
                                                                                            
  try_files $uri @api;                                                                      
}                                                                                           



I see that your peertube site got a video which is over 1 hour. Can you please help me?

I just checked your website. I am in Gatinue, Quebec. :slight_smile:

1 Like

Est-ce que tu as configuré object_storage.max_upload_part? : Remote storage (S3) | PeerTube documentation

Je n’utilise pas YunoHost, je ne sais pas comment sont géré les fichiers de config nginx et l’environnement (.env) de PeerTube dans YH.

Je n’utilise plus Wasabi depuis quelques années. À l’époque, le egress était gratuit, donc un reverse proxy (et serveur cache en même temps) devant le bucket S3 était inutile.

You know what I followed what you by using ChatGPT. I figure it out. Thanks a lot. :sunflower: