Peertube user creating nginx process with 200% cpu usage (6.2.1)

PeerTube
1

hi!

recently (last half week or so) my vps has started to max out cpu, so I looked into it and I saw the peertube user had an nginx process that had fully utilised all cpu!

when I stop this process (kill pid) the prrocess restarts and goes to full cpu usage straight away.

this is affecting other services on the server.

any advice would be appreciated.

I could not see any notable errors in the logs for either peertube or nginx.

—edit—

I saw there was a second nginx process for peertube running, so when I stopped that the cpu-intensive process stopped restarting (it looks like) so at least my vps is not at full cpu currently. however why this is occurring and how to address it I do not know.

thanks!

2

Greetings, try updating Nginx to version 1.27 by adding the official repository, this should solve the problem. I think you have 1.22 or 1.24

3

hallo @bestperson thanks for the suggestion! I upgraded my nginx to the new version however the issue did not resolve.

if I kill the (two) nginx processes started by peertube the server appears to function (they do not restart)

however they are started on boot.

any further advice would be appreciated!

thanks!

—edit—

the two nginx processes do restart! then vps goes to high resource usage.

4

Hello, I would do it this way, I would completely reinstall peertube on a new server, copy it via rsync, and see how it goes. Ngnix is a regular proxy, it can’t raise the load, and have you changed the nginx config? And also as an option I would update peertube to the latest version as written in version 6.3.1

5

Is there anything suspicious in the nginx access logs? In the nginx error logs?
Is there many requests?

6

hallo, thanks for the suggestion. after viewing the peertube.access and peertube.error logs, I did not see any detail that looked like an issue, though I do not know exactly what I am looking for.

can you give me any advice on how too see an error in these logs?

thanks!

7

hi, I am not able to start a new instance on another server, I only have this one. I will look at updating peertube to newest version and see if that helps.

8

/var/log/nginx

9

If there are thousands of requests per second, it could explain a high CPU load.

10

If it is nginx that is using CPU, I don’t think Peertube is the problem.

The question is « what the **** is nginx doing?! ». You can maybe try to use lsof -p 123 (where 123 is the PID - Process ID - of the nginx process that uses 100%).
This will show you the list of opened file descriptors for the nginx process (files and even network streams).
It could be hard to read when you are not familiar, but maybe you could see some unusual files opened (for example, if you see hundreds of similar files, it could help you understand what is happening).

Just a question: have you any other service than Peertube on this server?

11

I run an akkoma instance, an owncast server, peertube and a static html site.

peertube ran fine for the last four months or so, upgraded it once, however now it starts to do this with nginx.

here is the lsof output for the two nginx processes:

https://pastebin.com/raw/gt0HtL4b

there are no big lists of similar files like you say, I do not know enough to understand what I am looking at.

12

Nginx has a .redtail file open. This is not normal.

A quick web search seems on « .redtail file » give some results about a malware that is mining crypto currencies.

First link i found (i haven’t read it fully): https://www.akamai.com/blog/security-research/2024-redtail-cryptominer-pan-os-cve-exploit

I think your server was compromised.
(not necessarely throught Peertube, could be another software, a compromised password or ssh key, …)