recently (last half week or so) my vps has started to max out cpu, so I looked into it and I saw the peertube user had an nginx process that had fully utilised all cpu!
when I stop this process (kill pid) the prrocess restarts and goes to full cpu usage straight away.
this is affecting other services on the server.
any advice would be appreciated.
I could not see any notable errors in the logs for either peertube or nginx.
—edit—
I saw there was a second nginx process for peertube running, so when I stopped that the cpu-intensive process stopped restarting (it looks like) so at least my vps is not at full cpu currently. however why this is occurring and how to address it I do not know.
Hello, I would do it this way, I would completely reinstall peertube on a new server, copy it via rsync, and see how it goes. Ngnix is a regular proxy, it can’t raise the load, and have you changed the nginx config? And also as an option I would update peertube to the latest version as written in version 6.3.1
hallo, thanks for the suggestion. after viewing the peertube.access and peertube.error logs, I did not see any detail that looked like an issue, though I do not know exactly what I am looking for.
can you give me any advice on how too see an error in these logs?
hi, I am not able to start a new instance on another server, I only have this one. I will look at updating peertube to newest version and see if that helps.
If it is nginx that is using CPU, I don’t think Peertube is the problem.
The question is « what the **** is nginx doing?! ». You can maybe try to use lsof -p 123 (where 123 is the PID - Process ID - of the nginx process that uses 100%).
This will show you the list of opened file descriptors for the nginx process (files and even network streams).
It could be hard to read when you are not familiar, but maybe you could see some unusual files opened (for example, if you see hundreds of similar files, it could help you understand what is happening).
Just a question: have you any other service than Peertube on this server?
so after a little reading I could not find a guide to removing this malware, so I rm’d the .redtail binary from the peertube directory. should I change my passwords or is this a vulnerability with the peertube software itself?
thanks forr the support!
—edit— if removed the miner binary is replaced and re-run within six hours or so. I do not know how to address this issue, so any help would be appreciated.
I don’t think it is related to Peertube: i manage several instances, and i never had any malware.
Maybe a malicious Peertube plugin?
But there is no single response to such question. There are litteraly thousands of way a server can be infected.
Could be a compromised software, a brute force attack on your ssh password (if you have enabled connection using passwords), an exploit if your server is not up-to-date with the latest security fix, …
It could even be your personnal computer that is compromised (for example by a cracked video game) and propagated the malware to your servers using your ssh keys.
The first question is « how the attacker was able to install the malware ». Unfortunately, i can’t give you an exact walkthrough in a forum post to find this. Could be very difficult
(if you don’t find and block the entry point of this attack, the attacker will always be able to re-install a malware).
One way to proceed, if you have backups of your data:
completly re-install the server from scratch
Re-install all used applications (in same version as previously)
Restore applications data (and only data).
And even by doing so, there is a risk that the malware comes back: if any of your application can run code from its data, then there is a risk that the malware is hidden in the data.
hallo. thank you for your reply. the virus could have installed from a peertube plugin, since I added that recently or is it possible that php on this system (debian bookworm) is vulnerable.
I do not know how to back up all my services and lack the hard disk space to do so if I did know how.
I will look around to see if there is support for removing this particular virus, though I could not see a guide when I first looked.
should the peertube user be running any of the following processes:
Peertube as only one running process at startup.
It can then launch ffmpeg, ytb-dlp. If you have the livechat plugin, you could also see some child process (prosody, node, …).
But the peertube user should not have process like cron or sd-pam.
The crontab is an usual trick for malware to hide them: i saw many times malwares creating files in /var/spool/cron/crontabs/, to re-install them when deleted.
So, you can start by looking in this directly, and other crontab dir (/etc/cron.d, /etc/crontab, …).
If all malware file you found are owned by the peertube user, this might be because of a plugin. Can you share the list of plugin you installed, so we could check?
(please note that it is not 100% sure… it could also be a brute force on the peertube user login/password, if you have set a password and allowed ssh connections using passwords).
You can try to read /var/log/auth.log (and rotated files, /var/log/auth.log.1, …), to see if you can find any authent attempt for the peertube user. (you should not have such attempts).
If you have installed Bookworm directly, maybe these files does not exists. In such case, you have to use journalctl to find such logs (read the documentation to see how).
there was a file in /var/spool/cron/crontabs, which is replaced if I remove it, however I changed the write permissions on the directory which appears to hhave prevented the virus from recreating the cron file.
I only had thhe livechat plugin, which I since disabled.
from what I saw viewing top, all cpu-intensive tasks were being run by the peertube user account. this suggests the virus was from a vulnerability in the peertube software, or the plugin I had installed.
I read there were other files this virus modifies, however since changing permissions on the /var/spool/cron/crontabs folder it has not restarted.
