PeerTube Server Stops Responding (Rate Limiting?)

[Vencabot]

I just installed Peertube for the first time and set up my instance at https://tube.vencabot.com – and I’m super pleased with it! I’m having an odd problem, though, which I think may be due to rate-limiting?

If I’m playing a video on my instance and also have a few other tabs open featuring pages from my instance (the exact number of tabs seems to vary based on what the tabs are?), my new requests to the server do not get a response; they hang… UNTIL I close a tab, and then, as SOON as the tab closes, a neighboring tab will finally get a response from the server. Weird!

Now that the video has finished processing, I’m noticing two things: instead of saying ‹ HTTP ›, the player is saying ‹ Connected peers ›… and now, no matter how many tabs I open, I can’t make it freeze the way that I’ve been able to consistently test over the past two hours (while the video was processing).

The fact that closing a tab allows the others to finish connecting makes me think this is some kind of rate-limiting. I peeked at PeerTube’s rate-limiting settings in production.yaml but I don’t really know where to start.

Can someone shed light on this issue? Does it have something to do with the fact that I was watching an unprocessed video, and therefore playing via HTTP? Does the player send a crap-ton of HTTP requests in that case? Why would closing a tab allow the others to load (if I were being rate-limited, wouldn’t I have to wait for some amount of time)?

Very odd issue! And yeah, now that my video has processed at least one resolution and the player no longer says ‹ HTTP ›, I can’t reproduce it. Any ideas?

[Chocobozzz]

Hi,

It should not be related to peertube rate limits, as when you reach the rate limit then you have an error message.

Might be another issue. Do you have some web browser or server logs to provide?

[Vencabot]

Thank you for getting back to me, and for everything that you do. I do not have any logs, but I’ll continue to investigate by uploading another large file and doing some testing while it transcodes – because, like I said, I couldn’t seem to reproduce the issue once the initial transcoding was finished.

I appreciate your insight. I’m also considering, of course, that the issue could have something to do with the reverse proxy, or the server PC, or even my client-side browser / router. Who knows. I’ll see what logs I can review and what browser diagnostics I can watch that could be helpful.

Here’s how I was observing the issue before:

• I was watching a two-hour video I’d uploaded before its first transcoding was finished. The player said ‹ HTTP ›.
• I was opening up other tabs to look at various configuration pages for the server.
• Suddenly, the video stopped buffering and froze. At the same time, other pages I had loaded for my Peertube server also stopped loading new pages. They would just hang indefinitely while waiting for a response: no error message.
• At first, I thought perhaps my local internet was on the fritz, but I noticed that I could load all other websites normally – including my own pages running in other server software on the same server, behind the same reverse-proxy.
• When I closed the most recent Peertube tab I had open, the video suddenly started playing again and all of the other Peertube tabs loaded.
• When I tried opening another Peertube tab, it hanged and the video stopped buffering again. When I closed it, the video started buffering again.
• I tested this for as long as I could, opening and closing tabs and observing the relationship. Indeed, although the number of tabs I would need to open would sometimes vary, opening a small number (less than 10) would make the Peertube server seem to be unresponsive until I lowered the number of open tabs below a threshold, at which point the other hanging tabs would instantly load (as soon as I closed the tab) and the video would continue buffering and playing again.

But yeah, I’m far from convinced that this is a Peertube issue, or I would’ve opened an Issue: was just looking for insight, in case anyone here would know what I was talking about and be able to help me fix it quick! Seems like I’ll need to do more poking around.

Also, at first I thought that maybe the Peertube process was hanging because the transcode was using up so much CPU power, but of course the transcode was on its own thread, on its own core. I gave up on this idea when I noticed that the server was responding normally as long as I didn’t have too many tabs open that were making requests of it.

I’ll report back soon! Thanks again.

[Vencabot]

Tonight, I tried streaming to my Peertube server, and I’m having the same problem: the entire server has frozen up and won’t respond to any connections after I streamed for about 2 minutes or so. Just as with last time, I see that it’s using up 100% of one of my CPU’s cores. I presume that’s transcoding, but shouldn’t the peertube process running the server be on another core / process? It’s been locked up for more than 10 minutes now, and I’m waiting to see if it ever starts responding again. This time, I only had two Peertube tabs open, but it seems like maybe streaming is what overwhelmed it? I was streaming in low-latency mode, which disables p2p, etc.

I checked the logs in /var/www/peertube/storage/logs , and nothing there seems unusual. No errors or anything. Just as before, the other sites on the same box are all serving up just fine, including Owncast, where I stream normally.

The stream to Peertube was working fine, too, but then it just stopped responding. OBS started telling me that 100% of its frames were being dropped and that the server was receiving 0kbps. At that point, I stopped being able to navigate the server at all from tube.vencabot.com .

Is there another log that I should be checking for errors, somewhere? Any advice on how I could investigate what peertube is doing that seems to be freezing it / preventing it from doing anything else? I apologize that I need to brush up on my Linux skills in general; I only know that peertube is using 100% of one of my CPU cores because I’m running htop, but I don’t know how to investigate further from there.

I presume that this will be repeatable (and thus investigatable) any time I stream to Peertube.

[EDIT] I see that peertube is also running /usr/bin/node_dist/server and it’s also taking up 100% cpu or on the same core as the other peertube process which is using up 100% cpu, if I understand this UI correctly.

[EDIT] Maybe it’s bad behavior because I was trying to install the live-chat plugin while a stream is running? I see that the final messages in the log are about the live-chat plugin and launching ‹ prosody ›. The final message in the log is « Waiting for the prosody process to launch. » And that’s where it’s been stuck for 45 minutes now.

[EDIT] A few hours later, my reverse-proxy, Caddy, finally starts serving a 502 bad gateway error when trying to reach the Peertube instance. This inspired me to try connecting to it directly via the port I configured for Peertube. The connection is refused. When I look at netstat, the recv-q on Peertube’s http port is 512 and Peertube’s rtmp recv-q is 1 – so those ports are technically listening but their buffers are not being emptied. It does seem like Peertube is hanging – and still using up 100% CPU.

[Vencabot]

Unlike when I first made this post, where the server eventually sorted itself out, I’ve let the server run for 63 hours now and htop looks the same as it did before, and the peertube server still isn’t responding. I’m going to see if I can reproduce this problem and, if I can, I’ll make a formal Issue on the project page.

[Chocobozzz]

Can you enable debug logs in production.yaml, restart peertube and paste logs when peertube is jumping to 100%?

[Vencabot]

Thank you as always for your help, including detailed instructions on enabling debug logs. Debug logs are now enabled, but I was unable to reproduce my issue simply by starting another live-stream. Next time I have this issue, though, logs will be in debug mode, so hopefully that’ll help. I won’t bump this thread until then.

Here’s what I did to try to reproduce my issue, which failed:

• start a livestream with ‹ small latency (p2p disabled) ›.
• open a bunch of tabs to my Peertube server, including a bunch of viewers of the live stream, other videos, configuration pages, etc.
• Uninstalled, re-installed, and configured the livechat plugin DURING the stream (I theorized that this may have somehow caused the server to hang last time, since it was when I clicked ‹ Settings › for the plugin that I first noticed the server was unresponsive).

After all of that, the stream worked fine – and in fact, of course, it was very impressive! Transcoding works great, the latency was SUPER low (8 seconds, which is substantially lower than the lowest latency I could get with Owncast), the livechat plugin is beautiful and has excellent functionality.

If I have server hanging issues again, I’ll bump this thread with logs.

[JohnLivingston]

Oh, i did not see your edit from a previous comment, where you mention the livechat plugin (i’m the plugin’s dev).

Don’t hesitate to mention me here if you can reproduce the bug.