Nginx optimization for 1k-2k visitors for live stream

Sergiu · Septembre 9, 2022, 10:01

I have a problem when there are 1k-2k visitors to the live stream, the site does not open, it opens from the 3rd-5th page refresh. What optimizations could I do to nginx so that the site opens to more than 2k visitors?

Chocobozzz · Septembre 9, 2022, 11:40

Hello,

Can you provide some PeerTube logs? Do you know what is the bottleneck? CPU? Network? Nginx? PeerTube?

Sergiu · Septembre 9, 2022, 12:19

The logs were not saved, the processor was used somewhere at 50%, the player worked well, network traffic 60%, I intuit that the problem is in nginx because many times it shows me bad gateway and sometimes the site opens but does not display the live streams. I moved the folder /var/www/peertube/storage/streaming-playlists to ram (tmpfs) so it’s not a problem in SSD.

Chocobozzz · Septembre 9, 2022, 1:17

Okay thanks. Unfortunately I can’t help without nginx logs and peertube logs.

Sergiu · Septembre 9, 2022, 1:59

I tried with wrk to simulate several accesses to get the 500 error and here are the logs:

Chocobozzz · Septembre 9, 2022, 2:30

You need to increase nginx worker connections. See What are worker_connections in Nginx

Sergiu · Septembre 9, 2022, 4:34

Thanks for the help, I added it manually in nginx.conf

worker_processes 16;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
I hope it helps, does it make sense to add the last 2 lines?

vidbin · Septembre 10, 2022, 2:14

worker_connections 1024 is the nginx default on nginx mainline.

I would keep worker_processes on auto and set worker_connections to 5000-10000. You’ll need to modify the /etc/security/limits.conf accordingly.

Sergiu · Novembre 23, 2022, 12:23

I increased the number of worker_connections to 65535 in nginx and in the system, the errors with worker_connections disappeared from the logs, but anyway it is very difficult to display the streams at 1.5-2k views.
In htop I noticed that the problem appears when 1 or 2 cores reach 100% usage.

And when I access the website, it opens quickly, but it is very difficult to display the live streams.

I use PeerTube version 4.2.1

I don’t see any errors in nginx, but in peertube there are a lot like:

2022/11/23 13:55:58 [error] 1382#1382: *14203100 open() "/var/www/peertube/storage/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-007925.ts" failed (2: No such file or directory), client: 185.181.230.179, server: stream.trm.md, request: "GET /static/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-007925.ts HTTP/2.0", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"
2022/11/23 13:55:58 [error] 1395#1395: *14061335 open() "/var/www/peertube/storage/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-008818.ts" failed (2: No such file or directory), client: 89.32.232.82, server: stream.trm.md, request: "GET /static/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-008818.ts HTTP/2.0", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"
2022/11/23 13:55:58 [error] 1382#1382: *14183099 open() "/var/www/peertube/storage/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-012007.ts" failed (2: No such file or directory), client: 95.65.58.192, server: stream.trm.md, request: "GET /static/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-012007.ts HTTP/2.0", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c"
2022/11/23 13:55:58 [error] 1382#1382: *14069482 open() "/var/www/peertube/storage/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-008154.ts" failed (2: No such file or directory), client: 93.116.73.210, server: stream.trm.md, request: "GET /static/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-008154.ts HTTP/2.0", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"
2022/11/23 13:55:59 [crit] 1382#1382: *13506201 connect() to 127.0.0.1:9000 failed (99: Cannot assign requested address) while connecting to upstream, client: 178.168.109.69, server: stream.trm.md, request: "GET /live/segments-sha256/dd7acba9-2151-4415-8751-5d08a342340c HTTP/2.0", upstream: "http://127.0.0.1:9000/live/segments-sha256/dd7acba9-2151-4415-8751-5d08a342340c", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c"
2022/11/23 13:55:59 [crit] 1392#1392: *14105418 connect() to 127.0.0.1:9000 failed (99: Cannot assign requested address) while connecting to upstream, client: 178.168.119.65, server: stream.trm.md, request: "POST /api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views HTTP/2.0", upstream: "http://127.0.0.1:9000/api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"
2022/11/23 13:55:59 [crit] 1392#1392: *13403932 connect() to 127.0.0.1:9000 failed (99: Cannot assign requested address) while connecting to upstream, client: 178.168.121.225, server: stream.trm.md, request: "POST /api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views HTTP/2.0", upstream: "http://127.0.0.1:9000/api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"
2022/11/23 13:55:59 [crit] 1382#1382: *14254746 connect() to 127.0.0.1:9000 failed (99: Cannot assign requested address) while connecting to upstream, client: 91.211.51.119, server: stream.trm.md, request: "POST /api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views HTTP/2.0", upstream: "http://127.0.0.1:9000/api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c"
2022/11/23 13:55:59 [crit] 1382#1382: *13855722 connect() to 127.0.0.1:9000 failed (99: Cannot assign requested address) while connecting to upstream, client: 46.166.62.7, server: stream.trm.md, request: "POST /api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views HTTP/2.0", upstream: "http://127.0.0.1:9000/api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"
2022/11/23 13:55:59 [crit] 1392#1392: *14104363 connect() to 127.0.0.1:9000 failed (99: Cannot assign requested address) while connecting to upstream, client: 86.105.57.10, server: stream.trm.md, request: "POST /api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views HTTP/2.0", upstream: "http://127.0.0.1:9000/api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"
2022/11/23 13:55:59 [crit] 1392#1392: *14058239 connect() to 127.0.0.1:9000 failed (99: Cannot assign requested address) while connecting to upstream, client: 89.28.67.105, server: stream.trm.md, request: "POST /api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views HTTP/2.0", upstream: "http://127.0.0.1:9000/api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"
2022/11/23 13:55:59 [crit] 1392#1392: *14040687 connect() to 127.0.0.1:9000 failed (99: Cannot assign requested address) while connecting to upstream, client: 89.32.230.50, server: stream.trm.md, request: "POST /api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views HTTP/2.0", upstream: "http://127.0.0.1:9000/api/v1/videos/dd7acba9-2151-4415-8751-5d08a342340c/views", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"
2022/11/23 13:55:59 [error] 1392#1392: *14252025 open() "/var/www/peertube/storage/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-008094.ts" failed (2: No such file or directory), client: 185.33.106.97, server: stream.trm.md, request: "GET /static/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-008094.ts HTTP/2.0", host: "stream.trm.md", referrer: "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c?peertubeLink=0"

From changes in nginx I added this:

worker_rlimit_nofile 65535;

events {
worker_connections 65535;
use epoll;
multi_accept on;
}

Chocobozzz · Novembre 23, 2022, 1:13

Check peertube logs to see why nginx can’t connect to it

Sergiu · Novembre 23, 2022, 9:54

I updated to version 4.3.1, but now no peer appears, everything is downloaded directly from the server.
In the error logs I see this:

error[23/11/2022, 23:26:57] Client log: Cannot get sha256 segments

{
  "tags": [
    "client"
  ],
  "userAgent": "Mozilla/5.0 (Linux; Android 10; Redmi 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Mobile Safari/537.36",
  "stackTrace": "Unexpected end of JSON input\nSyntaxError: Unexpected end of JSON input",
  "url": "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c"
}

error[23/11/2022, 23:45:00] Client log: HLS.js error: mediaError - fatal: false - bufferStalledError

{
  "tags": [
    "client"
  ],
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
  "meta": "{\"data\":{\"type\":\"mediaError\",\"details\":\"bufferStalledError\",\"fatal\":false,\"buffer\":3.387165999999999}}",
  "url": "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c"
}

error[23/11/2022, 23:39:55] Client log: HLS.js error: networkError - fatal: false - fragLoadError

{
  "tags": [
    "client"
  ],
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
  "meta": "{\"data\":{\"type\":\"networkError\",\"details\":\"fragLoadError\",\"fatal\":false,\"frag\":{\"_byteRange\":null,\"_url\":\"https://stream.trm.md/static/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0-000182.ts\",\"baseurl\":\"https://stream.trm.md/static/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/0.m3u8\",\"relurl\":\"0-000182.ts\",\"elementaryStreams\":{\"audio\":null,\"video\":null,\"audiovideo\":null},\"_decryptdata\":null,\"rawProgramDateTime\":\"2022-11-23T23:39:03.680+0200\",\"programDateTime\":1669239543680,\"tagList\":[[\"INF\",\"4.760000\"],[\"PROGRAM-DATE-TIME\",\"2022-11-23T23:39:03.680+0200\"]],\"duration\":4.76,\"sn\":182,\"type\":\"main\",\"loader\":null,\"level\":0,\"cc\":0,\"start\":536.6798333333334,\"stats\":{\"loaded\":564,\"total\":564,\"aborted\":false,\"retry\":0,\"chunkCount\":0,\"bwEstimate\":0,\"loading\":{\"start\":533380.3,\"end\":0,\"first\":533380.3},\"parsing\":{\"start\":0,\"end\":0},\"buffering\":{\"start\":0,\"end\":0,\"first\":0}},\"urlId\":0,\"bitrateTest\":false,\"title\":null,\"initSegment\":null},\"response\":{\"isTrusted\":true}}}",
  "url": "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c"
}

error[23/11/2022, 23:39:55] Client log: Segment https://stream.trm.md/static/streaming-playlists/hls/dd7acba9-2151-4415-8751-5d08a342340c/master.m3u8+V0+183 error.

{
  "tags": [
    "client"
  ],
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
  "meta": "{\"isTrusted\":true}",
  "url": "https://stream.trm.md/videos/embed/dd7acba9-2151-4415-8751-5d08a342340c"
}

Sergiu · Novembre 24, 2022, 8:12

I moved the stream to another server, version 4.3.0 was installed cleanly, and I have the same problem, when I reach 2k viewers in real time on the live stream, I moved the hls folder to ram (tmpfs). When I access the site with 2k - 2.5k viewers, it gives me the error 502 (Bad Gateway):

Peertube error:

	Line 383136: 2022/11/24 21:47:22 [crit] 1323#1323: *15695864 connect() to 127.0.0.1:9000 failed (99: Unknown error) while connecting to upstream, client: 192.168.10.222, server: media.trm.md, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:9000/", host: "media.trm.md"
	Line 383269: 2022/11/24 21:47:22 [crit] 1323#1323: *15695864 connect() to 127.0.0.1:9000 failed (99: Unknown error) while connecting to upstream, client: 192.168.10.222, server: media.trm.md, request: "GET /favicon.ico HTTP/2.0", upstream: "http://127.0.0.1:9000/favicon.ico", host: "media.trm.md", referrer: "https://media.trm.md/"
	Line 386245: 2022/11/24 21:47:25 [crit] 1323#1323: *15695864 connect() to 127.0.0.1:9000 failed (99: Unknown error) while connecting to upstream, client: 192.168.10.222, server: media.trm.md, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:9000/", host: "media.trm.md"
	Line 387459: 2022/11/24 21:47:27 [crit] 1323#1323: *15695864 connect() to 127.0.0.1:9000 failed (99: Unknown error) while connecting to upstream, client: 192.168.10.222, server: media.trm.md, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:9000/", host: "media.trm.md"
	Line 388435: 2022/11/24 21:47:28 [crit] 1323#1323: *15695864 connect() to 127.0.0.1:9000 failed (99: Unknown error) while connecting to upstream, client: 192.168.10.222, server: media.trm.md, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:9000/", host: "media.trm.md"
	Line 389312: 2022/11/24 21:47:29 [crit] 1323#1323: *15695864 connect() to 127.0.0.1:9000 failed (99: Unknown error) while connecting to upstream, client: 192.168.10.222, server: media.trm.md, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:9000/", host: "media.trm.md"
	Line 390198: 2022/11/24 21:47:30 [crit] 1323#1323: *15695864 connect() to 127.0.0.1:9000 failed (99: Unknown error) while connecting to upstream, client: 192.168.10.222, server: media.trm.md, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:9000/", host: "media.trm.md"

Nginx error, it’s because of geoblocking:

2022/11/24 21:47:14 [error] 1328#1328: OCSP responder timed out (110: Unknown error) while requesting certificate status, responder: r3.o.lencr.org, peer: 84.53.161.192:80, certificate: "/etc/letsencrypt/live/media.trm.md/fullchain.pem"
2022/11/24 21:47:50 [error] 1321#1321: OCSP responder timed out (110: Unknown error) while requesting certificate status, responder: r3.o.lencr.org, peer: 84.53.161.192:80, certificate: "/etc/letsencrypt/live/media.trm.md/fullchain.pem"
2022/11/24 21:48:00 [error] 1323#1323: OCSP responder timed out (110: Unknown error) while requesting certificate status, responder: r3.o.lencr.org, peer: 84.53.161.192:80, certificate: "/etc/letsencrypt/live/media.trm.md/fullchain.pem"

I intuit that the problem is that the peertube process uses only one core, and when this process reaches 95%-98%, these problems appear for me.

The problem is not in cpu/ram/hdd/network resources, out of 4Gb peertube channel uses only 2.5-3.5Gb.

Is it somehow possible to implement a cache for displaying the site or the player? something like fastcgi cache and proxy cache?

JohnLivingston · Novembre 25, 2022, 10:45

This should already be the case, thanks to the nginx configuration.

For example:

github.com

Chocobozzz/PeerTube/blob/a5858c3e53103ff42f8a442ef4760979db87dcfc/support/nginx/peertube#L189-L193


      
          location ~ ^/client/(.*\.(js|css|png|svg|woff2|otf|ttf|woff|eot))$ {
            add_header Cache-Control "public, max-age=31536000, immutable"; # Cache 1 year
          
            alias /var/www/peertube/peertube-latest/client/dist/$1;
          }

(and other similar nginx locations)

Sergiu · Novembre 25, 2022, 2:47

So the problem is that the peertube process uses only one core and when it reaches 98%+, the player is very difficult to appear on the site, possibly after 2-3 page refreshes.
In this screen we see the resources used for 1000 live stream views, without encoding.

Chocobozzz · Novembre 28, 2022, 9:53

Can you try to profile peertube?

Restart PeerTube using --inspect option (node --inspect dist/server) on server
Run ssh -L 9229:127.0.0.1:9229 name@host on your computer
Run Chrome and go on chrome://inspect using search bar
Profile the application while you stream for a few seconds and send me by private message the profile file

Sergiu · Novembre 29, 2022, 10:00

I installed version 18 of nodejs, in the releases it says that it supports, during installation I have the following warnings: