Problems with large user export

PeerTube
1

Hello forum! I’m really pleased to see the user import and export feature land. I run a server with a lot of disk space, which I’ve been in the slow process of winding down. It’s important to me that users can get their data if they want.

My account is the largest on the server with 1.7 TB of video total, so I started there to stress test the feature. I’ve been running into multiple problems, and there’s one bug that prevents any export over 4GB from working.

I don’t use github any more or I’d start filing issues. There are a few:

  • integer column in userExports table
  • premature cleanup of « expired » exports after they have been in progress for 1 hour
  • exports stuck in « progress » until server restart
  • older incomplete livestreams crashing the entire export because no video file path could be determined

Maybe others I haven’t understood yet. I’m still trying to get one successful export as a proof of concept.

If any devs try to export a larger account these issues should be easy to reproduce.

2

Thank you very much, I’ll fix these issues!

Can you paste PeerTube error logs so it’s easier for me to find the root cause of the bugs?

3

Hi Chocobozzz,

I’m not seeing anything useful in the logs. The postgres related integer « size » field drops a backtrace but that fix is easy:

alter table « userExport » alter « size » type bigint;

I was trying to find the video path related error I hit earlier on the incomplete livestream, but I don’t think it ended up in the log.

4

My nearly 2TB export finished, with two hacks in place. (size column data type changed to bigint, and commented out the « expired » cleanup entirely)

Now it’s downloading and the default rate limit is very painful here. It seems like user exports should have a much higher limit than regular video downloads.

1 « J'aime »
5

I personally think the rate limit is fine.

You don’t want 1-5 users overloading your bandwidth. And if you’re on a large bandwidth instance you’ve likely already tweaked the default rate limit.

6

Yes I’ve had a higher default limit on the instance for a while, but I would even still prefer a higher limit for user exports than video downloads.

Video download throttling helps mitigate logged-out crawlers, but my actual users I want to download their archive as fast as possible so the file doesn’t sit around.

7

Slow and steady, the nearly 2TB zip file has fully downloaded and it at least looks okay to « unzip -l ». I’ll need to try importing it too, when I have a server free with enough space.

1 « J'aime »
8

Thanks a lot!

I fixed the column type in Fix big user export file size · Chocobozzz/PeerTube@e7ebcb1 · GitHub

Regarding the user export expiration, you can change it in the configuration: PeerTube/config/default.yaml at develop · Chocobozzz/PeerTube · GitHub

9

Thanks for fixing the column type!

Regarding expiration, I did try changing the config but archives were being deleted an hour into the process regardless. It seemed like the expiration test was buggy but I didn’t get far enough to understand the root cause before disabling the cleanup entirely.