Peertube setup with minio S3 object storage. System cannot delete content from bucket

Hello,

I am testing out peertube setup on docker with minio S3 object storage configured for transcoded video storage.
Everything seems to be working except deleting videos from S3 buckets.

I also did test bucket access just in case via rlone and it had no problem with functions: listing, directory-listing, delete.
With the same credentials that peertube uses so I guess permissions should be a problem. Since key has « * » for S3 actions allowed which is given peertube.

Noticed one issue similar to mine in github, but user with that problem stopped replying before any decent info exchange…

Any ideas?

Peertube container log at that time:

[<peertube_host>:443] 2023-11-29 08:54:42.149 error: Cannot apply function on peertube-streaming/hls/730dadfa-d91c-42df-9e02-fb3a6af46af3 prefix in bucket peertube-streaming-cdn: no files listed. {
  "response": {
    "$metadata": {
      "httpStatusCode": 200,
      "requestId": "179C0C1DF8B73EBA",
      "extendedRequestId": "fe074e5c965e5bde521147179318c7e2667684bd44dc8133e770e3bdac500cd4",
      "attempts": 1,
      "totalRetryDelay": 0
    }
  }
}
[<peertube_host>:443] 2023-11-29 08:54:42.149 error: Some errors when removing files of video 730dadfa-d91c-42df-9e02-fb3a6af46af3 in before destroy hook. {
  "err": {
    "stack": "Error: Cannot apply function on peertube-streaming/hls/730dadfa-d91c-42df-9e02-fb3a6af46af3 prefix in bucket peertube-streaming-cdn: no files listed.\n    at applyOnPrefix (file:///app/dist/core/lib/object-storage/shared/object-storage-helpers.js:175:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async VideoModel.removeStreamingPlaylistFiles (file:///app/dist/core/models/video/video.js:940:17)\n    at async Promise.all (index 0)",
    "message": "Cannot apply function on peertube-streaming/hls/730dadfa-d91c-42df-9e02-fb3a6af46af3 prefix in bucket peertube-streaming-cdn: no files listed."
  }
}

Peertube compose env settings for S3

# object storage config
PEERTUBE_OBJECT_STORAGE_ENABLED=true
PEERTUBE_OBJECT_STORAGE_CREDENTIALS_ACCESS_KEY_ID="<key>"
PEERTUBE_OBJECT_STORAGE_CREDENTIALS_SECRET_ACCESS_KEY="<secret>"
PEERTUBE_OBJECT_STORAGE_ENDPOINT="<domain>"
PEERTUBE_OBJECT_STORAGE_STREAMING_PLAYLISTS_BUCKET_NAME="peertube-streaming-cdn"
PEERTUBE_OBJECT_STORAGE_STREAMING_PLAYLISTS_PREFIX="peertube-streaming/"
PEERTUBE_OBJECT_STORAGE_WEB_VIDEOS_BUCKET_NAME="peertube-videos-cdn"
PEERTUBE_OBJECT_STORAGE_WEB_VIDEOS_PREFIX="peertube-videos/"

Haproxy log at that time:

Nov 29 11:54:24 localhost haproxy[228418]: [minio_backend] client=<peertube_host>:53488 server=<minio_host>:9000 (name=minio2 host=peertube-streaming-cdn.<domain>) flags=-- read=867 upload=777 [HTTPS: TLSv1.3] 200 "GET /?list-type=2&prefix=peertube-streaming%2Fhls%2Fb8c61d2d-55c2-4f28-bb98-f43a1d110ff2 HTTP/1.1"

minio r/w key policy:

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Action": [
    "s3:*"
   ],
   "Resource": [
    "arn:aws:s3:::peertube-streaming/*",
    "arn:aws:s3:::peertube-videos/*"
   ]
  }
 ]
}

BR

Hi,

Can you list files inside peertube-streaming/hls/730dadfa-d91c-42df-9e02-fb3a6af46af3 prefix?

As mentioned, yes. I did try that with rclone with the same S3 credentials.

Here’s output from one of tries (not the same video).

[<user>@HAL3000 ~]$ rclone ls peertube-streaming-cdn.<domain>:/peertube-streaming/hls
      567 2858a48f-30a0-421e-8773-3815105fd513/01e739ad-5e48-461e-a8cc-fff77fda62d7-segments-sha256.json
      172 2858a48f-30a0-421e-8773-3815105fd513/24699402-1bff-40df-b2bd-cdde7eb5e421-master.m3u8
  4304031 2858a48f-30a0-421e-8773-3815105fd513/e7ee1b26-2385-4eb3-b27f-f6e146ab2075-576-fragmented.mp4
      837 2858a48f-30a0-421e-8773-3815105fd513/e7ee1b26-2385-4eb3-b27f-f6e146ab2075-576.m3u8
     3247 3642e3fc-d163-4a87-bfee-1da69d611b17/07d54559-daab-4c98-ac8a-7b74886c3004-segments-sha256.json
  1236373 3642e3fc-d163-4a87-bfee-1da69d611b17/2057868a-a1cf-49b9-bb3d-47d4abb31694-720-fragmented.mp4
     1146 3642e3fc-d163-4a87-bfee-1da69d611b17/2057868a-a1cf-49b9-bb3d-47d4abb31694-720.m3u8
  2233068 3642e3fc-d163-4a87-bfee-1da69d611b17/327bfba4-8196-47f3-ad35-ae831096bdfc-1080-fragmented.mp4
     1161 3642e3fc-d163-4a87-bfee-1da69d611b17/327bfba4-8196-47f3-ad35-ae831096bdfc-1080.m3u8
      611 3642e3fc-d163-4a87-bfee-1da69d611b17/33921bb8-449d-49ef-af08-11bd8bc861c3-master.m3u8
  4008881 3642e3fc-d163-4a87-bfee-1da69d611b17/8f73c305-790b-40e5-b3aa-b72176a783e4-1626-fragmented.mp4
     1164 3642e3fc-d163-4a87-bfee-1da69d611b17/8f73c305-790b-40e5-b3aa-b72176a783e4-1626.m3u8
   767992 3642e3fc-d163-4a87-bfee-1da69d611b17/a31b0eb7-a2bb-4ce8-83ec-669229d567f3-480-fragmented.mp4
     1140 3642e3fc-d163-4a87-bfee-1da69d611b17/a31b0eb7-a2bb-4ce8-83ec-669229d567f3-480.m3u8
      174 59abd6a0-1935-428d-b80a-b04f2431a998/007a556f-b63f-49c3-85ec-cca720494d8c-master.m3u8
     1352 59abd6a0-1935-428d-b80a-b04f2431a998/3d44ac58-37b6-4fbe-85f5-6e4e98d2af99-segments-sha256.json
 18634095 59abd6a0-1935-428d-b80a-b04f2431a998/a1616ae5-a19a-4ed6-8f37-18e177bb6258-1080-fragmented.mp4
     1826 59abd6a0-1935-428d-b80a-b04f2431a998/a1616ae5-a19a-4ed6-8f37-18e177bb6258-1080.m3u8

If needed I can try with some other method. I will upload some new test video for that.

We use ListObjectsV2Command to list files inside a prefix.

Can you dirty console.log(listedObjects) inside this function? I’m surprised there is no other things than $metadata key in the response.

I most likely can. You wanted this after line 317 or 338?

Put it line 321

hmm. Am I looking at the right logs: docker logs -f peertube-peertube-1? After adding that line w/o and /w compose down/up. I don’t see any new logging entries during that deletion attempt.

file /home/docker/.local/share/docker/fuse-overlayfs/<longstring>/diff/app/server/core/lib/object-storage/shared/object-storage-helpers.ts function now looks like:

async function applyOnPrefix (options: {
  prefix: string
  bucketInfo: BucketInfo
  commandBuilder: (obj: _Object) => Parameters<S3Client['send']>[0]

  continuationToken?: string
}) {
  const { prefix, bucketInfo, commandBuilder, continuationToken } = options

  const s3Client = await getClient()

  const { ListObjectsV2Command } = await import('@aws-sdk/client-s3')

  const commandPrefix = buildKey(prefix, bucketInfo)
  const listCommand = new ListObjectsV2Command({
    Bucket: bucketInfo.BUCKET_NAME,
    Prefix: commandPrefix,
    ContinuationToken: continuationToken
  })

  const listedObjects = await s3Client.send(listCommand)
  console.log(listedObjects)

  if (isArray(listedObjects.Contents) !== true) {
    const message = `Cannot apply function on ${commandPrefix} prefix in bucket ${bucketInfo.BUCKET_NAME}: no files listed.`

    logger.error(message, { response: listedObjects, ...lTags() })
    throw new Error(message)
  }

  await Bluebird.map(listedObjects.Contents, object => {
    const command = commandBuilder(object)

    return s3Client.send(command)
  }, { concurrency: 10 })

  // Repeat if not all objects could be listed at once (limit of 1000?)
  if (listedObjects.IsTruncated) {
    await applyOnPrefix({ ...options, continuationToken: listedObjects.ContinuationToken })
  }
}

and logs ar still look like:

[<host>:443] 2023-11-29 15:32:47.573 info: Creating job to broadcast delete of video https://<host>/videos/watch/288401ff-7526-44a3-8056-4f16873648f4.
[<host>:443] 2023-11-29 15:32:47.578 info: Removing files of video https://<host>/videos/watch/288401ff-7526-44a3-8056-4f16873648f4.
[<host>:443] 2023-11-29 15:32:47.587 info: Removing preview file 90de76b7-8b1d-4ff7-bc28-658a71594e5d.jpg.
[<host>:443] 2023-11-29 15:32:47.588 info: Removing miniature file a260f4c2-084e-4dd2-8cb0-814fd1926c8a.jpg.
[<host>:443] 2023-11-29 15:32:47.597 info: Removing storyboard file 8bf97fc4-2a3b-4fb4-8cb6-287ca08beac8.jpg.
[<host>:443] 2023-11-29 15:32:47.613 info: Video with name MSFC_11162023_Resource Reel_4769~large and uuid 288401ff-7526-44a3-8056-4f16873648f4 deleted.
[<host>:443] 2023-11-29 15:32:47.614 info: 10.5.217.29 - - [29/Nov/2023:15:32:47 +0000] "DELETE /api/v1/videos/21 HTTP/1.0" 204 - "https://<host>/w/61bfhxiSp515KhkDMzVhgo" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61"
[<host>:443] 2023-11-29 15:32:47.650 error: Cannot apply function on peertube-streaming/hls/288401ff-7526-44a3-8056-4f16873648f4 prefix in bucket peertube-streaming-cdn: no files listed. {
  "response": {
    "$metadata": {
      "httpStatusCode": 200,
      "requestId": "179C21D73F4E4139",
      "extendedRequestId": "fe074e5c965e5bde521147179318c7e2667684bd44dc8133e770e3bdac500cd4",
      "attempts": 1,
      "totalRetryDelay": 0
    }
  }
}
[<host>:443] 2023-11-29 15:32:47.650 error: Some errors when removing files of video 288401ff-7526-44a3-8056-4f16873648f4 in before destroy hook. {
  "err": {
    "stack": "Error: Cannot apply function on peertube-streaming/hls/288401ff-7526-44a3-8056-4f16873648f4 prefix in bucket peertube-streaming-cdn: no files listed.\n    at applyOnPrefix (file:///app/dist/core/lib/object-storage/shared/object-storage-helpers.js:175:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async VideoModel.removeStreamingPlaylistFiles (file:///app/dist/core/models/video/video.js:940:17)\n    at async Promise.all (index 0)",
    "message": "Cannot apply function on peertube-streaming/hls/288401ff-7526-44a3-8056-4f16873648f4 prefix in bucket peertube-streaming-cdn: no files listed."
  }
}
[<host>:443] 2023-11-29 15:32:47.763 info: 10.5.217.29 - - [29/Nov/2023:15:32:47 +0000] "GET /api/v1/videos/categories HTTP/1.0" 200 274 "https://<host>/videos/trending" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61"
[<host>v:443] 2023-11-29 15:32:47.852 info: 10.5.217.29 - - [29/Nov/2023:15:32:47 +0000] "GET /api/v1/videos?start=0&count=25&sort=-trending&skipCount=true&nsfw=false HTTP/1.0" 200 11 "https://<host>/videos/trending" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61"

You need to edit the built file so dist/core/lib/object-storage/shared/object-storage-helpers.js just after line 171

Ok, got a bit more.

[<host>:443] 2023-11-30 16:17:58.956 info: Creating job to broadcast delete of video https://<host>/videos/watch/974b107d-926a-49d8-9e16-9ac96602fbb6.
[<host>:443] 2023-11-30 16:17:58.961 info: Removing files of video https://<host>/videos/watch/974b107d-926a-49d8-9e16-9ac96602fbb6.
[<host>:443] 2023-11-30 16:17:58.986 info: Removing preview file 62f791a3-cfa6-43b2-b078-789731e6dbbf.jpg.
[<host>:443] 2023-11-30 16:17:58.987 info: Removing miniature file 76a6612a-795f-4a1f-918b-19c3a1f1f276.jpg.
{
  '$metadata': {
    httpStatusCode: 200,
    requestId: '179C72E31B03B230',
    extendedRequestId: '68dd90b70cc795eb93042acaa477221e136a8e9c2d3c25ac8f6475eab01788ce',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}
[<host>:443] 2023-11-30 16:17:59.024 error: Cannot apply function on peertube-streaming/hls/974b107d-926a-49d8-9e16-9ac96602fbb6 prefix in bucket peertube-streaming-cdn: no files listed. {
  "response": {
    "$metadata": {
      "httpStatusCode": 200,
      "requestId": "179C72E31B03B230",
      "extendedRequestId": "68dd90b70cc795eb93042acaa477221e136a8e9c2d3c25ac8f6475eab01788ce",
      "attempts": 1,
      "totalRetryDelay": 0
    }
  }
}
[<host>:443] 2023-11-30 16:17:59.025 error: Some errors when removing files of video 974b107d-926a-49d8-9e16-9ac96602fbb6 in before destroy hook. {
  "err": {
    "stack": "Error: Cannot apply function on peertube-streaming/hls/974b107d-926a-49d8-9e16-9ac96602fbb6 prefix in bucket peertube-streaming-cdn: no files listed.\n    at applyOnPrefix (file:///app/dist/core/lib/object-storage/shared/object-storage-helpers.js:176:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async VideoModel.removeStreamingPlaylistFiles (file:///app/dist/core/models/video/video.js:940:17)\n    at async Promise.all (index 0)",
    "message": "Cannot apply function on peertube-streaming/hls/974b107d-926a-49d8-9e16-9ac96602fbb6 prefix in bucket peertube-streaming-cdn: no files listed."
  }
}
[<host>:443] 2023-11-30 16:17:59.028 info: Removing storyboard file 075b9b83-965b-4a7b-bc5b-c557f6d9ea5c.jpg.
[<host>:443] 2023-11-30 16:17:59.199 info: Video with name MSFC_11162023_Resource Reel_4769~large and uuid 974b107d-926a-49d8-9e16-9ac96602fbb6 deleted.
[<host>:443] 2023-11-30 16:17:59.200 info: <client_ip> - - [30/Nov/2023:16:17:59 +0000] "DELETE /api/v1/videos/23 HTTP/1.0" 204 - "https://<host>/w/jFzn8DtYfJuRHPaiZ6iADU" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61"
[<host>:443] 2023-11-30 16:17:59.365 info: <client_ip> - - [30/Nov/2023:16:17:59 +0000] "GET /api/v1/videos/categories HTTP/1.0" 304 - "https://<host>/videos/trending" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61"
[<host>:443] 2023-11-30 16:17:59.449 info: <client_ip> - - [30/Nov/2023:16:17:59 +0000] "GET /api/v1/videos?start=0&count=25&sort=-trending&skipCount=true&nsfw=false HTTP/1.0" 304 - "https://<host>/videos/trending" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61"

Do you have error or interesting logs on Minio side? Because I don’t understand what happens. You can help yourself using the requestId

Log from minio side looks like this.

Time            Name                          Status                  Location                                                Load Time           Upload          Download
### user - list files from rclone a test
10:04:06:0      s3.ListObjectsV2              200 OK                  peertube-streaming-cdn.<domain> <user_pc_ip>            5.175205ms          120.0 B         721.0 B
10:04:06:0      s3.HeadObject                 404 Not Found           peertube-streaming-cdn.<domain> <user_pc_ip>            4.716712ms          104.0 B         0.0 B
10:04:27:0      s3.HeadObject                 404 Not Found           peertube-streaming-cdn.<domain> <user_pc_ip>            4.686745ms          104.0 B         0.0 B
10:04:27:0      s3.ListObjectsV2              200 OK                  peertube-streaming-cdn.<domain> <user_pc_ip>            4.72441ms           120.0 B         400.0 B
### peertube - upload after transcoding
10:13:58:0      s3.PutObject                  200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       51.306281ms         36.9 KiB        0.0 B
10:13:58:0      s3.NewMultipartUpload         200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       7.155972ms          159.0 B         411.0 B
10:13:58:0      s3.PutObjectPart              200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       7.478823662s        43.1 MiB        0.0 B
10:13:58:0      s3.PutObjectPart              200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       16.818063767s       100.0 MiB       0.0 B
10:14:15:0      s3.PutObject                  200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       98.156665ms         37.7 KiB        0.0 B
10:14:15:0      s3.NewMultipartUpload         200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       7.454945ms          159.0 B         412.0 B
10:14:15:0      s3.CompleteMultipartUpload    200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       51.756393ms         508.0 B         527.0 B
10:14:16:0      s3.PutObjectPart              200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       23.374343768s       100.0 MiB       0.0 B
10:14:17:0      s3.PutObjectPart              200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       22.295448201s       100.0 MiB       0.0 B
10:14:15:0      s3.PutObjectPart              200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       23.712316941s       100.0 MiB       0.0 B
10:14:16:0      s3.PutObjectPart              200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       24.342424123s       100.0 MiB       0.0 B
10:14:39:0      s3.PutObjectPart              200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       8.316574308s        82.4 MiB        0.0 B
10:14:48:0      s3.PutObject                  200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       121.243043ms        60.2 KiB        0.0 B
10:14:48:0      s3.CompleteMultipartUpload    200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       232.042037ms        796.0 B         529.0 B
10:14:48:0      s3.PutObject                  200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       27.70877ms          498.0 B         0.0 B
### user - checking upload from web int
10:15:04:0      s3.GetObject                  200 OK                  peertube-streaming-cdn.<domain> <user_pc_ip>            18.288965ms         203.0 B         319.0 B
10:15:10:0      s3.GetObject                  206 Partial Content     peertube-streaming-cdn.<domain> <user_pc_ip>            86.830008ms         209.0 B         1.3 KiB
10:15:10:0      s3.GetObject                  206 Partial Content     peertube-streaming-cdn.<domain> <user_pc_ip>            19.261358ms         218.0 B         574.8 KiB
10:15:10:0      s3.GetObject                  206 Partial Content     peertube-streaming-cdn.<domain> <user_pc_ip>            18.153588ms         218.0 B         397.7 KiB
10:15:10:0      s3.GetObject                  200 OK                  peertube-streaming-cdn.<domain> <user_pc_ip>            23.102735ms         203.0 B         37.5 KiB
10:15:10:0      s3.GetObject                  200 OK                  peertube-streaming-cdn.<domain> <user_pc_ip>            34.944871ms         203.0 B         60.0 KiB
10:15:10:0      s3.GetObject                  206 Partial Content     peertube-streaming-cdn.<domain> <user_pc_ip>            47.278251ms         218.0 B         301.1 KiB
10:15:10:0      s3.GetObject                  206 Partial Content     peertube-streaming-cdn.<domain> <user_pc_ip>            16.742044ms         218.0 B         309.9 KiB
10:15:11:0      s3.GetObject                  206 Partial Content     peertube-streaming-cdn.<domain> <user_pc_ip>            658.53551ms         218.0 B         2.0 MiB
10:15:12:0      s3.GetObject                  206 Partial Content     peertube-streaming-cdn.<domain> <user_pc_ip>            132.202591ms        218.0 B         1.8 MiB
10:15:13:0      s3.GetObject                  206 Partial Content     peertube-streaming-cdn.<domain> <user_pc_ip>            63.927204ms         218.0 B         1.3 MiB
10:15:13:0      s3.GetObject                  206 Partial Content     peertube-streaming-cdn.<domain> <user_pc_ip>            142.103319ms        218.0 B         1.4 MiB
### peertube - attempt to delete video
10:15:34:0      s3.ListBuckets                200 OK                  peertube-streaming-cdn.<domain> <peertube_srv_ip>       42.70302ms          176.0 B         474.0 B

Do you also have raw HTTP logs in front of Minio so we can check what HTTP request peertube is making?

I’m not sure if it helps but I’m using this bucket policy in Minio: Set object storage bucket acl via environment variable - #10 par MakerTube

And this key policy:

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Action": [
    "admin:*"
   ]
  },
  {
   "Effect": "Allow",
   "Action": [
    "kms:*"
   ]
  },
  {
   "Effect": "Allow",
   "Action": [
    "s3:*"
   ],
   "Resource": [
    "arn:aws:s3:::*"
   ]
  }
 ]
}

Which might be a bit broad while thinking of it.

@Chocobozzz Will try to setup nginx or apache(most likely this as colleague already has example for apache config of such setup) tomorrow or test env of our minio for raw http log capture. Haproxy didn’t want to budge to that config.

@MakerTube In one of test attempts I did try to use key with with same unlimited access policies. It seem’s that it’s not the case in my setup.

1 Like

RAW log from apache in share link. I also added one more file with access log from apache and peertube for timestamp references. I think I filtered out only deletion attempt logs, so there should be no other actions in between in logs.

edit. messed up. deletion attempt starts at 2023-12-04 13:54* overlooked timestamps…
fiuxed, uploaded correct raw log.

Thanks, from what I see the request seems correct: GET /?list-type=2&prefix=peertube-streaming%2Fhls%2F312521cb-567f-49e3-be88-202f218f071a

But Minio returns the list of buckets and not the list of objects:

<?xml version="1.0" encoding="UTF-8"?>\n<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID><DisplayName>minio</DisplayName></Owner><Buckets><Bucket><Name>peertube-streaming</Name><CreationDate>2023-11-10T14:27:10.741Z</CreationDate></Bucket><Bucket><Name>peertube-videos</Name><CreationDate>2023-11-10T14:26:35.353Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

Can you try to compare minio logs with rclone ls... to see how their request differs?

Yes. Rclone does return correct data and when previously looking at logs it also uses correct command ListObjectsV2 ?

rclone ls peertube-streaming-minio0.test.mt.lv:/peertube-streaming
 25627621 hls/312521cb-567f-49e3-be88-202f218f071a/3b5a5a22-7356-472f-aaba-91ea85126884-720-fragmented.mp4
     5956 hls/312521cb-567f-49e3-be88-202f218f071a/3b5a5a22-7356-472f-aaba-91ea85126884-720.m3u8
 14366213 hls/312521cb-567f-49e3-be88-202f218f071a/b3b69a9b-bec5-4465-b2c1-108dcd9afb12-480-fragmented.mp4
     5925 hls/312521cb-567f-49e3-be88-202f218f071a/b3b69a9b-bec5-4465-b2c1-108dcd9afb12-480.m3u8
     9374 hls/312521cb-567f-49e3-be88-202f218f071a/bf5b8e8d-829c-4d16-a239-4054d0a80e77-segments-sha256.json
      316 hls/312521cb-567f-49e3-be88-202f218f071a/e7c5433f-026f-45d1-a06f-4442d17c5afa-master.m3u8

Sorry there was a typo in my message: can you paste the minio logs when you use rclone command?

added file rclone_minio_raw.log

Seems like rclone uses the path style request which is not supported by peertube: Support object storage path style requests · Issue #4455 · Chocobozzz/PeerTube · GitHub

You need to configure minio so it can handle host style requests: aws sdk go - How to setup MinIO Server to use Virtual Hosted-Style - Stack Overflow