S3 migration: Segment error and videos get hidden from channels

PeerTube
1

Hi there, i have already tried to ask on Matrix, but maybe this would be a better place:

I have initiated a migration from local (network storage) to S3 (Backblaze). Uploads work, playback without proxy works now too (i had to learn how important the custom bucket rules are, oops, some failed jobs there), but with the proxy, i am getting a segment error similar to this already closed issue.

The nginx proxy is adopted from the Mastodon docs with minor adjustments:

  location @s3 {
    limit_except GET OPTIONS HEAD { # GET didnt seem enough
      deny all;
    }

    resolver 8.8.8.8;
#    proxy_set_header Host YOUR_S3_HOSTNAME; # backblaze generally doesnt seem to like this one
    proxy_set_header Connection '';
    proxy_set_header Authorization '';
    proxy_hide_header Set-Cookie;
    proxy_hide_header 'Access-Control-Allow-Origin';
    proxy_hide_header 'Access-Control-Allow-Methods';
#    proxy_hide_header 'Access-Control-Allow-Headers'; # this seemed to get rid of errors like "range"
    proxy_hide_header x-amz-id-2;
    proxy_hide_header x-amz-request-id;
    proxy_hide_header x-amz-meta-server-side-encryption;
    proxy_hide_header x-amz-server-side-encryption;
    proxy_hide_header x-amz-bucket-region;
    proxy_hide_header x-amzn-requestid;
    proxy_ignore_headers Set-Cookie;
    proxy_pass $s3_backend$uri;
    proxy_intercept_errors off;

    proxy_cache tubecache;
    proxy_cache_valid 48h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;

    expires 1y;
    add_header Cache-Control public;
    add_header 'Access-Control-Allow-Origin' '*';
    add_header X-Cache-Status $upstream_cache_status;

Now i wonder if the resulting commit of that bugreport would fix my issue as well, even tho i indeed see 206 statuscodes. And how i could cherry-pick that commit without bigger setup changes (i suppose i can replace that symlink with a git version though, now that i write about it).

Edit: just removing that proxy from the peertube config makes it two errors:

XHROPTIONShttps://media.tube.tchncs.de/videos/...
CORS Missing Allow Origin

XHROPTIONShttps://tube-tchncs-de.s3.eu-central-003.backblazeb2.com/videos/...
CORS Missing Allow Origin

Second issue

A user just reported that almost all of their videos are not visible anymore. If i click to see all videos as moderator, their videos are still there and neither private nor unlisted. The hidden videos appear to be migrated ones, to object storage. This is very confusing and i am not sure how to address this problem at the moment.

2

Can you provide a link of a broken video?

3

Hi, thanks for your reply!
I couldn’t reproduce the exact error this morning and am not sure what’s different… only last thing i’ve tried yesterday was removing and re-adding the base_urls as far as i recall.

  1. it looks like the hidden videos are the ones that are pending for migration
  2. migrated videos currently return Access-Control-Allow-Origin errors. Here is an example as requested.
object_storage:
  enabled: true
  endpoint: 's3.eu-central-003.backblazeb2.com'
  region: 'eu-central-003'
  credentials:
    access_key_id: 'redacted'
    secret_access_key: 'redacted'
  videos:
    bucket_name: 'tube-tchncs-de'
    prefix: 'videos/'
    base_url: 'https://media.tube.tchncs.de'
  streaming_playlists:
    bucket_name: 'tube-tchncs-de'
    prefix: 'streaming-playlists/'
    base_url: 'https://media.tube.tchncs.de'

I have tried to catch the OPTIONS / HEAD requests in similar if statements as in the exampleconfig on the proxy but had no luck.

Here is the current state of the failing cache proxy:

  location @s3 {
#    limit_except GET OPTIONS HEAD {
#      deny all;
#    }

    resolver 8.8.8.8;
#    proxy_set_header Host https://tube.tchncs.de;
#    proxy_set_header Connection '';
#    proxy_set_header Authorization '';
#    proxy_hide_header Set-Cookie;
#    proxy_hide_header 'Access-Control-Allow-Origin';
#    proxy_hide_header 'Access-Control-Allow-Methods';
#    proxy_hide_header 'Access-Control-Allow-Headers';
    proxy_hide_header x-amz-id-2;
    proxy_hide_header x-amz-request-id;
    proxy_hide_header x-amz-meta-server-side-encryption;
    proxy_hide_header x-amz-server-side-encryption;
    proxy_hide_header x-amz-bucket-region;
    proxy_hide_header x-amzn-requestid;
#    proxy_ignore_headers Set-Cookie;
    proxy_pass $s3_backend$uri;
    proxy_intercept_errors off;

    proxy_cache tubecache;
    proxy_cache_valid 48h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;

    expires 1y;
    add_header Cache-Control public;
    add_header 'Access-Control-Allow-Origin' '*'; # switching these to
                                                  # proxy_set_header doesn't help
    add_header Access-Control-Allow-Methods   'GET, OPTIONS, HEAD';
    add_header Access-Control-Allow-Headers   'Range,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
    add_header X-Cache-Status $upstream_cache_status;
  }

I am not sure how to catch this Access-Control-Allow-Origin right now… :frowning:

4

It seems the range header is not sent to object storage because it returns the entire file every time

5

Hmm, in order to find a fix for that, i have found and tried the following:

    proxy_pass_request_headers      on;
    proxy_set_header   Range $slice_range;
    proxy_force_ranges on;
    proxy_http_version 1.1;

for Firefox, this doesn’t do the trick. Curiously in Vivaldi however it looks like it is working. Very strange. For Firefox, it is also not helping to jump into a private tab.

6

There is something wrong with your S3 provider:

curl 'https://media.tube.tchncs.de/videos/eb0a8a13-cc07-43c4-97e9-01aa177790dc-720.mp4' -X OPTIONS -H 'Access-Control-Request-Method: GET' -H 'Access-Control-Request-Headers: range,user-agent' -H 'Origin: https://tube.tchncs.de'

Returns a 403

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>This CORS request is not allowed.  See https://www.backblaze.com/b2/docs/cors_rules.html.</Message>
</Error>
7

Thank you, this hint made me realize that the cors rules from the peertube wiki are possibly outdated and i have updated them accordingly:

        {
            "allowedHeaders": [
                "range",
                "user-agent"
            ],
            ...
        },
        {
            "allowedHeaders": [
                "range",
                "user-agent"
            ],
            ...
        }

Now also Firefox works with Peertube again (still confused why Vivaldi worked).

Now, only thing left is the hidden videos from user channels, but maybe thats worth a bugreport / feature request on Github?
Will mark this as solution anyway, because this possibly was the most disruptive issue. Thanks again!

8

Can you update the documentation with the rules that work?

For your second bug yes please create a dedicated topic it will be simpler to follow

1 « J'aime »
9

Could it be related to a v5.0.0 upgrade?
There is a migration script to move files for hidden/private videos. Have you launch it after upgrade? (see v5.0.0 changelog)

PS: I don’t know if S3 storage is concerned by this upgrade script, I never take a look how S3 integration works in Peertube. I’m just asking to help find the problem.

10

Hi, yes, i have first updated to v5 and also ran dist/scripts/migrations/peertube-5.0.js. I have initiated the move to object storage afterwards. But thanks for the suggestion!
…however this might be a misunderstanding: the videos that are not visible on user channels (at least?) are configured to be publicly visible, however it looks like they are the ones pending for migration to object storage.
Will try to report this to the repo soon.

11

I have combined the settings mentioned above and come up with the follow config to added to peertube ngixn config.

location @s3 {
#    limit_except GET OPTIONS HEAD {
#      deny all;
#    }

    resolver 8.8.8.8;
#    proxy_set_header Host https://qien.tv;
#    proxy_set_header Connection '';
#    proxy_set_header Authorization '';
     proxy_force_ranges on;
     proxy_http_version 1.1;
     proxy_set_header   Range $slice_range;
#    proxy_hide_header Set-Cookie;
#    proxy_hide_header 'Access-Control-Allow-Origin';
#    proxy_hide_header 'Access-Control-Allow-Methods';
#    proxy_hide_header 'Access-Control-Allow-Headers';
    proxy_hide_header x-amz-id-2;
    proxy_hide_header x-amz-request-id;
    proxy_hide_header x-amz-meta-server-side-encryption;
    proxy_hide_header x-amz-server-side-encryption;
    proxy_hide_header x-amz-bucket-region;
    proxy_hide_header x-amzn-requestid;
#    proxy_ignore_headers Set-Cookie;
    proxy_pass_request_headers      on;
    proxy_pass $s3_backend$uri;
    proxy_intercept_errors off;

    proxy_cache tubecache;
    proxy_cache_valid 48h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;

    expires 1y;
    add_header Cache-Control public;
    add_header 'Access-Control-Allow-Origin' '*'; # switching these to
                                                  # proxy_set_header doesn't help
    add_header Access-Control-Allow-Methods   'GET, OPTIONS, HEAD';
    add_header Access-Control-Allow-Headers   'Range,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
    add_header X-Cache-Status $upstream_cache_status;
  }

I tried

this with my url:

curl ‹ https://cdn.qien.tv/qienhls/9237b7be-67cd-406a-9c48-410553a0ff68/738ec598-7baf-426e-b107-258d7a4787d2-576-fragmented.mp4 › -X OPTIONS -H ‹ Access-Control-Request-Method: GET › -H ‹ Access-Control-Request-Headers: range,user-agent › -H ‹ Origin: https://cdn.qien.tv ›

But it returns with nothing.

Would some one be so kind to offer some wisdom please?