Peertube server overloaded

Hey,

I’m running PeerTube on 48 vCPUs. It worked fine for the last couple months, but now at peak times the videos don’t run fluidly anymore. The shell of the Ubuntu virtual machine, where PeerTube is running, doesn’t respond well (I connect via SSH and when I enter something, it takes a second until the characters are visible in the prompt).

I noticed that one or two CPUs, interestingly always the same ones, are often close to or at 100%, while the others remain idle.

In peertube.error.log there’s the following error a couple times per second: " open() « /var/www/peertube/storage/videos/62800644-857a-4b6d-8d46-241cc09535ef-1080.mp4 » failed (2: No such file or directory)"
This file doesn’t exist obviously. Could this be the cause?

The problem appears when there is a lot of traffic, at low times it’s just fine.

If you have any idea what could be done, please help me out. Thank you very much

And what about the network bandwidth? Have you checked?

Does the video 62800644-857a-4b6d-8d46-241cc09535ef exist? (try the url https://_your_instance_domain/videos/watch/62800644-857a-4b6d-8d46-241cc09535ef )

Hey,

thank you for your reply.

According to our hosting provider, there is no limit on network bandwidth.

The video with this id does not exist. The error is caused by a different video (that plays just fine), according to the « referrer » property in the error message.

I don’t really understand how videos are stored by PeerTube. Most of the data seems to be stored in peertube/storage/streaming-playlist/hls. In peertube/storage/videos there are only a few files, although we have many videos on the instance. What does it mean?

There is necessarily a limit, if only that of the material. Have you a monitoring system that shows the used bandwidth? If not, you can try the command iftop as root, when you notice a slow-down.

Peertube changed the video format some months (years?) ago. Previous format was stored in /storage/videos. New format (hls) is in /storage/streaming-playlist/hls. Video were not migrated, so it is normal to have files in both folders.

You were right - bandwidth is limited to 5 Gbps, which we hit. I discovered that by running vnstat on the server when the problem appeared. Our solution will be setting up multiple PeerTube instances with redundant federation on different servers.

Thank you, John, for taking the time to share your knowledge!

1 « J'aime »