I am testing out peertube setup on docker with minio S3 object storage configured for transcoded video storage.
Everything seems to be working except deleting videos from S3 buckets.
I also did test bucket access just in case via rlone and it had no problem with functions: listing, directory-listing, delete.
With the same credentials that peertube uses so I guess permissions should be a problem. Since key has « * » for S3 actions allowed which is given peertube.
Noticed one issue similar to mine in github, but user with that problem stopped replying before any decent info exchange…
Any ideas?
Peertube container log at that time:
[<peertube_host>:443] 2023-11-29 08:54:42.149 error: Cannot apply function on peertube-streaming/hls/730dadfa-d91c-42df-9e02-fb3a6af46af3 prefix in bucket peertube-streaming-cdn: no files listed. {
"response": {
"$metadata": {
"httpStatusCode": 200,
"requestId": "179C0C1DF8B73EBA",
"extendedRequestId": "fe074e5c965e5bde521147179318c7e2667684bd44dc8133e770e3bdac500cd4",
"attempts": 1,
"totalRetryDelay": 0
}
}
}
[<peertube_host>:443] 2023-11-29 08:54:42.149 error: Some errors when removing files of video 730dadfa-d91c-42df-9e02-fb3a6af46af3 in before destroy hook. {
"err": {
"stack": "Error: Cannot apply function on peertube-streaming/hls/730dadfa-d91c-42df-9e02-fb3a6af46af3 prefix in bucket peertube-streaming-cdn: no files listed.\n at applyOnPrefix (file:///app/dist/core/lib/object-storage/shared/object-storage-helpers.js:175:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async VideoModel.removeStreamingPlaylistFiles (file:///app/dist/core/models/video/video.js:940:17)\n at async Promise.all (index 0)",
"message": "Cannot apply function on peertube-streaming/hls/730dadfa-d91c-42df-9e02-fb3a6af46af3 prefix in bucket peertube-streaming-cdn: no files listed."
}
}
hmm. Am I looking at the right logs: docker logs -f peertube-peertube-1? After adding that line w/o and /w compose down/up. I don’t see any new logging entries during that deletion attempt.
file /home/docker/.local/share/docker/fuse-overlayfs/<longstring>/diff/app/server/core/lib/object-storage/shared/object-storage-helpers.ts function now looks like:
async function applyOnPrefix (options: {
prefix: string
bucketInfo: BucketInfo
commandBuilder: (obj: _Object) => Parameters<S3Client['send']>[0]
continuationToken?: string
}) {
const { prefix, bucketInfo, commandBuilder, continuationToken } = options
const s3Client = await getClient()
const { ListObjectsV2Command } = await import('@aws-sdk/client-s3')
const commandPrefix = buildKey(prefix, bucketInfo)
const listCommand = new ListObjectsV2Command({
Bucket: bucketInfo.BUCKET_NAME,
Prefix: commandPrefix,
ContinuationToken: continuationToken
})
const listedObjects = await s3Client.send(listCommand)
console.log(listedObjects)
if (isArray(listedObjects.Contents) !== true) {
const message = `Cannot apply function on ${commandPrefix} prefix in bucket ${bucketInfo.BUCKET_NAME}: no files listed.`
logger.error(message, { response: listedObjects, ...lTags() })
throw new Error(message)
}
await Bluebird.map(listedObjects.Contents, object => {
const command = commandBuilder(object)
return s3Client.send(command)
}, { concurrency: 10 })
// Repeat if not all objects could be listed at once (limit of 1000?)
if (listedObjects.IsTruncated) {
await applyOnPrefix({ ...options, continuationToken: listedObjects.ContinuationToken })
}
}
and logs ar still look like:
[<host>:443] 2023-11-29 15:32:47.573 info: Creating job to broadcast delete of video https://<host>/videos/watch/288401ff-7526-44a3-8056-4f16873648f4.
[<host>:443] 2023-11-29 15:32:47.578 info: Removing files of video https://<host>/videos/watch/288401ff-7526-44a3-8056-4f16873648f4.
[<host>:443] 2023-11-29 15:32:47.587 info: Removing preview file 90de76b7-8b1d-4ff7-bc28-658a71594e5d.jpg.
[<host>:443] 2023-11-29 15:32:47.588 info: Removing miniature file a260f4c2-084e-4dd2-8cb0-814fd1926c8a.jpg.
[<host>:443] 2023-11-29 15:32:47.597 info: Removing storyboard file 8bf97fc4-2a3b-4fb4-8cb6-287ca08beac8.jpg.
[<host>:443] 2023-11-29 15:32:47.613 info: Video with name MSFC_11162023_Resource Reel_4769~large and uuid 288401ff-7526-44a3-8056-4f16873648f4 deleted.
[<host>:443] 2023-11-29 15:32:47.614 info: 10.5.217.29 - - [29/Nov/2023:15:32:47 +0000] "DELETE /api/v1/videos/21 HTTP/1.0" 204 - "https://<host>/w/61bfhxiSp515KhkDMzVhgo" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61"
[<host>:443] 2023-11-29 15:32:47.650 error: Cannot apply function on peertube-streaming/hls/288401ff-7526-44a3-8056-4f16873648f4 prefix in bucket peertube-streaming-cdn: no files listed. {
"response": {
"$metadata": {
"httpStatusCode": 200,
"requestId": "179C21D73F4E4139",
"extendedRequestId": "fe074e5c965e5bde521147179318c7e2667684bd44dc8133e770e3bdac500cd4",
"attempts": 1,
"totalRetryDelay": 0
}
}
}
[<host>:443] 2023-11-29 15:32:47.650 error: Some errors when removing files of video 288401ff-7526-44a3-8056-4f16873648f4 in before destroy hook. {
"err": {
"stack": "Error: Cannot apply function on peertube-streaming/hls/288401ff-7526-44a3-8056-4f16873648f4 prefix in bucket peertube-streaming-cdn: no files listed.\n at applyOnPrefix (file:///app/dist/core/lib/object-storage/shared/object-storage-helpers.js:175:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async VideoModel.removeStreamingPlaylistFiles (file:///app/dist/core/models/video/video.js:940:17)\n at async Promise.all (index 0)",
"message": "Cannot apply function on peertube-streaming/hls/288401ff-7526-44a3-8056-4f16873648f4 prefix in bucket peertube-streaming-cdn: no files listed."
}
}
[<host>:443] 2023-11-29 15:32:47.763 info: 10.5.217.29 - - [29/Nov/2023:15:32:47 +0000] "GET /api/v1/videos/categories HTTP/1.0" 200 274 "https://<host>/videos/trending" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61"
[<host>v:443] 2023-11-29 15:32:47.852 info: 10.5.217.29 - - [29/Nov/2023:15:32:47 +0000] "GET /api/v1/videos?start=0&count=25&sort=-trending&skipCount=true&nsfw=false HTTP/1.0" 200 11 "https://<host>/videos/trending" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61"
Time Name Status Location Load Time Upload Download
### user - list files from rclone a test
10:04:06:0 s3.ListObjectsV2 200 OK peertube-streaming-cdn.<domain> <user_pc_ip> 5.175205ms 120.0 B 721.0 B
10:04:06:0 s3.HeadObject 404 Not Found peertube-streaming-cdn.<domain> <user_pc_ip> 4.716712ms 104.0 B 0.0 B
10:04:27:0 s3.HeadObject 404 Not Found peertube-streaming-cdn.<domain> <user_pc_ip> 4.686745ms 104.0 B 0.0 B
10:04:27:0 s3.ListObjectsV2 200 OK peertube-streaming-cdn.<domain> <user_pc_ip> 4.72441ms 120.0 B 400.0 B
### peertube - upload after transcoding
10:13:58:0 s3.PutObject 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 51.306281ms 36.9 KiB 0.0 B
10:13:58:0 s3.NewMultipartUpload 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 7.155972ms 159.0 B 411.0 B
10:13:58:0 s3.PutObjectPart 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 7.478823662s 43.1 MiB 0.0 B
10:13:58:0 s3.PutObjectPart 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 16.818063767s 100.0 MiB 0.0 B
10:14:15:0 s3.PutObject 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 98.156665ms 37.7 KiB 0.0 B
10:14:15:0 s3.NewMultipartUpload 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 7.454945ms 159.0 B 412.0 B
10:14:15:0 s3.CompleteMultipartUpload 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 51.756393ms 508.0 B 527.0 B
10:14:16:0 s3.PutObjectPart 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 23.374343768s 100.0 MiB 0.0 B
10:14:17:0 s3.PutObjectPart 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 22.295448201s 100.0 MiB 0.0 B
10:14:15:0 s3.PutObjectPart 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 23.712316941s 100.0 MiB 0.0 B
10:14:16:0 s3.PutObjectPart 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 24.342424123s 100.0 MiB 0.0 B
10:14:39:0 s3.PutObjectPart 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 8.316574308s 82.4 MiB 0.0 B
10:14:48:0 s3.PutObject 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 121.243043ms 60.2 KiB 0.0 B
10:14:48:0 s3.CompleteMultipartUpload 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 232.042037ms 796.0 B 529.0 B
10:14:48:0 s3.PutObject 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 27.70877ms 498.0 B 0.0 B
### user - checking upload from web int
10:15:04:0 s3.GetObject 200 OK peertube-streaming-cdn.<domain> <user_pc_ip> 18.288965ms 203.0 B 319.0 B
10:15:10:0 s3.GetObject 206 Partial Content peertube-streaming-cdn.<domain> <user_pc_ip> 86.830008ms 209.0 B 1.3 KiB
10:15:10:0 s3.GetObject 206 Partial Content peertube-streaming-cdn.<domain> <user_pc_ip> 19.261358ms 218.0 B 574.8 KiB
10:15:10:0 s3.GetObject 206 Partial Content peertube-streaming-cdn.<domain> <user_pc_ip> 18.153588ms 218.0 B 397.7 KiB
10:15:10:0 s3.GetObject 200 OK peertube-streaming-cdn.<domain> <user_pc_ip> 23.102735ms 203.0 B 37.5 KiB
10:15:10:0 s3.GetObject 200 OK peertube-streaming-cdn.<domain> <user_pc_ip> 34.944871ms 203.0 B 60.0 KiB
10:15:10:0 s3.GetObject 206 Partial Content peertube-streaming-cdn.<domain> <user_pc_ip> 47.278251ms 218.0 B 301.1 KiB
10:15:10:0 s3.GetObject 206 Partial Content peertube-streaming-cdn.<domain> <user_pc_ip> 16.742044ms 218.0 B 309.9 KiB
10:15:11:0 s3.GetObject 206 Partial Content peertube-streaming-cdn.<domain> <user_pc_ip> 658.53551ms 218.0 B 2.0 MiB
10:15:12:0 s3.GetObject 206 Partial Content peertube-streaming-cdn.<domain> <user_pc_ip> 132.202591ms 218.0 B 1.8 MiB
10:15:13:0 s3.GetObject 206 Partial Content peertube-streaming-cdn.<domain> <user_pc_ip> 63.927204ms 218.0 B 1.3 MiB
10:15:13:0 s3.GetObject 206 Partial Content peertube-streaming-cdn.<domain> <user_pc_ip> 142.103319ms 218.0 B 1.4 MiB
### peertube - attempt to delete video
10:15:34:0 s3.ListBuckets 200 OK peertube-streaming-cdn.<domain> <peertube_srv_ip> 42.70302ms 176.0 B 474.0 B
@Chocobozzz Will try to setup nginx or apache(most likely this as colleague already has example for apache config of such setup) tomorrow or test env of our minio for raw http log capture. Haproxy didn’t want to budge to that config.
@MakerTube In one of test attempts I did try to use key with with same unlimited access policies. It seem’s that it’s not the case in my setup.
RAW log from apache in share link. I also added one more file with access log from apache and peertube for timestamp references. I think I filtered out only deletion attempt logs, so there should be no other actions in between in logs.
edit. messed up. deletion attempt starts at 2023-12-04 13:54* overlooked timestamps…
fiuxed, uploaded correct raw log.