Best way to upload 30k videos?

Can you show your code? And what is your instance URL (or version at least?)

It’s ok now, it seems the doc is missing the info that contrary to a mere api/v1/videos/{id} call, api/v1/videos/{id}/source must be authorized, so it needs the Authorization: « Bearer » option :sweat_smile:

1 Like

Indeed, this info is not public, and only viewable by the video’s owner :slight_smile:

Great work.

Thanks! I’ll update the documentation :slight_smile:

1 Like

This was rather useless anyway since I didn’t think first « ok I can get the filename of a video I know, but I can’t get a video from its filename » :man_facepalming:. Getting ALL videos and then getting ALL sources with just the API to find which one fits a given filename just feels ridiculous. I guess I either have to query the videoSource table (and learn postgreSQL first, but shouldn’t be far from mySQL), or just stick with the xls file.

Unless of course @Chocobozzz feels like adding a query to the API returning a video by its source, or the list of sources :blush:

I’m sorry but it’s not planned, Framasoft doesn’t have time to implement this feature :slight_smile:

No pb, getting the info straight from database went smoother than I thought.

Just a little teasing of the great job of GregRc with Chocobozzz’s and JohnLivingston’s help… Still in progress :slight_smile:

2 Likes

Hi, my mass upload script works mostly but I’m having some upload errors I can’t figure, because usually I just restart the script a bit later and failed uploads then succeed, with no settings or code modification. Errors are of these types:

  • Error: write ECONNRESET
  • Error: write EPIPE
  • ENOSPC: no space left on device, write
  • SocketError: other side closed

I also get SyntaxError: Unexpected token '<', "<html><h"... is not valid JSON errors thrown because obviously html is sent instead of json, but I don’t know why or what’s in it.

It feels like some kind of traffic jam is happening. There are 101 videos involved, each a few minutes duration an a 300-700 Mb size range. At first 53 uploads went nicely and 22 failed (script ignored the remaining 26 as they were already online). A few minutes later those 22 kept failing, and a few days later they all succeeded. This happens on my local peertube with local files sent, as well as when script runs on a cdn to send files from there to a distant test peertube server. I’m still trying to figure what’s going on with those errors above, but I tell it here in case it rings a bell and someone has any clue.

edit: the full SocketError is
Peertube API callSync error "http://127.0.0.1:9001/api/v1/videos/upload": TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11457:11) after I fetch Peertube « upload » request.

Seems there is no space left. You just have to figure out where. Maybe on the partition where nginx stores incoming upload?

You have to make sure you have space left. Specifically you want to look for temporary upload locations. Nginx has one for example.

I don’t think disk space is the issue: our peertube server has dozens gigs left, plus the ENOSPC is thrown on 2 uploads only among 101 and the script goes on to upload other vidéos with no pb. E.g. the ENOSPC is thrown after calling the videos/upload API with a 186 Mb mp4 file, while larger vidéos get uploaded a few steps later with no pb. I suppose if space was the pb, it would also break following uploads.

Edit: indeed that might be a space problem, as another upload on a different server was monitored by admins who saw the free space vanishing, and then many above errors started legitimately appearing. There’s a script that frees space once a video has been transcoded, but upload went quite faster than transcoding :sweat_smile: Which would explain why another try a few days later could succeed.

1 Like

Hello thanks for sharing this information, I Sanjeev Mansotra was looking at it for a long time. Thank you for the hack!

Hi, here is a quick update.
We’re now writing the doc for this peertube-mass-upload script, and we hope to release it in the next 2 or 3 months. Once done, we will publish here the link to it.
Thanks :slight_smile:

4 Likes

Yeah! Looking forwarding to check it out. :bouquet: :smiley: