Recovering instance from crash due to full filesystem

I made a mistake with server CLI commands and now my instance is down.

Basically, I ordered all the files in object storage to move to the local disk and then return to remote storage. I also issued a couple prune-storage commands because I thought that was helping.

Now I have zero bytes left on /. I go into a little more detail here.

Any advice from a smart sysadmin or developer would be greatly appreciated.


Edit: I stopped peertube and freed 39 MB on the server by deleting the apt cache. My best idea so far is to somehow expand the root filesystem and then revive peertube to let its file transfers run their course. My total archive is less than a TB.

If I were using lvm I could do this (more) easily by adding another physical volume, but alas.


Edit2: I’m copying the server’s storage folder to my personal computer with plenty of free space. I was thinking of then mounting this folder on the server using sshfs to temporarily expand the local server storage, but I can’t reach my PC from the net because it’s behind a CGNAT.

Update

The site is back up. I talked to my hosting provider and arranged additional local storage.

Now my concern is any data loss or corruption that could have occurred while I was issuing several conflicting server commands in quick succession (including prune-storage!). I’m poring over every file.


I’ve encountered one error already, which occurred when trying to issue the originally intended server command.

> peertube@6.1.0 create-move-video-storage-job
> node ./dist/scripts/create-move-video-storage-job.js --to-object-storage -v --all-videos

[tankie.tube:443 uploadx] 2024-06-16 06:09:55.941 debug: DiskStorage config: {
  directory: '/var/www/peertube/storage/tmp/resumable-uploads',
  expiration: { maxAge: undefined, rolling: true },
  maxMetadataSize: '10MB',
  logger: [Function: Object],
  userIdentifier: [Function: userIdentifier],
  filename: [Function: filename]
}
[tankie.tube:443 uploadx] 2024-06-16 06:09:55.944 debug: Registered handlers: DELETE, GET, OPTIONS, PATCH, POST, PUT
[tankie.tube:443] 2024-06-16 06:09:56.250 debug: Executed SQL request - Executing (default): SELECT 1 FROM pg_available_extensions WHERE name = 'pg_trgm' AND installed_version IS NOT NULL;
[tankie.tube:443] 2024-06-16 06:09:56.251 debug: Executed SQL request - Executing (default): SELECT 1 FROM pg_available_extensions WHERE name = 'unaccent' AND installed_version IS NOT NULL;
[tankie.tube:443] 2024-06-16 06:09:56.254 debug: Executed SQL request - Executing (default): CREATE OR REPLACE FUNCTION immutable_unaccent(text)
  RETURNS text AS
$func$
SELECT public.unaccent('public.unaccent', $1::text)
$func$  LANGUAGE sql IMMUTABLE;
[tankie.tube:443] 2024-06-16 06:09:56.267 debug: Executed SQL request - Executing (default): SELECT "video".*, "Thumbnails"."id" AS "Thumbnails.id", "Thumbnails"."type" AS "Thumbnails.type", "Thumbnails"."filename" AS "Thumbnails.filename", "Thumbnails"."height" AS "Thumbnails.height", "Thumbnails"."width" AS "Thumbnails.width", "Thumbnails"."fileUrl" AS "Thumbnails.fileUrl", "Thumbnails"."onDisk" AS "Thumbnails.onDisk", "Thumbnails"."automaticallyGenerated" AS "Thumbnails.automaticallyGenerated", "Thumbnails"."videoId" AS "Thumbnails.videoId", "Thumbnails"."videoPlaylistId" AS "Thumbnails.videoPlaylistId", "Thumbnails"."createdAt" AS "Thumbnails.createdAt", "Thumbnails"."updatedAt" AS "Thumbnails.updatedAt" FROM (SELECT * FROM "video" WHERE uuid = true LIMIT 1) AS "video" LEFT OUTER JOIN "thumbnail" AS "Thumbnails" ON "video"."id" = "Thumbnails"."videoId"
Error
    at Query.run (/var/www/peertube/versions/peertube-v6.1.0/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)
    at /var/www/peertube/versions/peertube-v6.1.0/node_modules/sequelize/lib/sequelize.js:315:28
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 0)
    at async VideoModelGetQueryBuilder.queryVideo (file:///var/www/peertube/versions/peertube-v6.1.0/dist/core/models/video/sql/video/video-model-get-query-builder.js:16:76)
    at async run (file:///var/www/peertube/versions/peertube-v6.1.0/dist/scripts/create-move-video-storage-job.js:40:23) {
  name: 'SequelizeDatabaseError',
  parent: error: operator does not exist: uuid = boolean
      at Parser.parseErrorMessage (/var/www/peertube/versions/peertube-v6.1.0/node_modules/pg-protocol/dist/parser.js:287:98)
      at Parser.handlePacket (/var/www/peertube/versions/peertube-v6.1.0/node_modules/pg-protocol/dist/parser.js:126:29)
      at Parser.parse (/var/www/peertube/versions/peertube-v6.1.0/node_modules/pg-protocol/dist/parser.js:39:38)
      at Socket.<anonymous> (/var/www/peertube/versions/peertube-v6.1.0/node_modules/pg-protocol/dist/index.js:11:42)
      at Socket.emit (node:events:517:28)
      at addChunk (node:internal/streams/readable:368:12)
      at readableAddChunk (node:internal/streams/readable:341:9)
      at Readable.push (node:internal/streams/readable:278:10)
      at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
    length: 200,
    severity: 'ERROR',
    code: '42883',
    detail: undefined,
    hint: 'No operator matches the given name and argument types. You might need to add explicit type casts.',
    position: '665',
    internalPosition: undefined,
    internalQuery: undefined,
    where: undefined,
    schema: undefined,
    table: undefined,
    column: undefined,
    dataType: undefined,
    constraint: undefined,
    file: 'parse_oper.c',
    line: '647',
    routine: 'op_error',
    sql: 'SELECT "video".*, "Thumbnails"."id" AS "Thumbnails.id", "Thumbnails"."type" AS "Thumbnails.type", "Thumbnails"."filename" AS "Thumbnails.filename", "Thumbnails"."height" AS "Thumbnails.height", "Thumbnails"."width" AS "Thumbnails.width", "Thumbnails"."fileUrl" AS "Thumbnails.fileUrl", "Thumbnails"."onDisk" AS "Thumbnails.onDisk", "Thumbnails"."automaticallyGenerated" AS "Thumbnails.automaticallyGenerated", "Thumbnails"."videoId" AS "Thumbnails.videoId", "Thumbnails"."videoPlaylistId" AS "Thumbnails.videoPlaylistId", "Thumbnails"."createdAt" AS "Thumbnails.createdAt", "Thumbnails"."updatedAt" AS "Thumbnails.updatedAt" FROM (SELECT * FROM "video" WHERE uuid = true LIMIT 1) AS "video" LEFT OUTER JOIN "thumbnail" AS "Thumbnails" ON "video"."id" = "Thumbnails"."videoId"',
    parameters: undefined
  },
  original: error: operator does not exist: uuid = boolean
      at Parser.parseErrorMessage (/var/www/peertube/versions/peertube-v6.1.0/node_modules/pg-protocol/dist/parser.js:287:98)
      at Parser.handlePacket (/var/www/peertube/versions/peertube-v6.1.0/node_modules/pg-protocol/dist/parser.js:126:29)
      at Parser.parse (/var/www/peertube/versions/peertube-v6.1.0/node_modules/pg-protocol/dist/parser.js:39:38)
      at Socket.<anonymous> (/var/www/peertube/versions/peertube-v6.1.0/node_modules/pg-protocol/dist/index.js:11:42)
      at Socket.emit (node:events:517:28)
      at addChunk (node:internal/streams/readable:368:12)
      at readableAddChunk (node:internal/streams/readable:341:9)
      at Readable.push (node:internal/streams/readable:278:10)
      at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
    length: 200,
    severity: 'ERROR',
    code: '42883',
    detail: undefined,
    hint: 'No operator matches the given name and argument types. You might need to add explicit type casts.',
    position: '665',
    internalPosition: undefined,
    internalQuery: undefined,
    where: undefined,
    schema: undefined,
    table: undefined,
    column: undefined,
    dataType: undefined,
    constraint: undefined,
    file: 'parse_oper.c',
    line: '647',
    routine: 'op_error',
    sql: 'SELECT "video".*, "Thumbnails"."id" AS "Thumbnails.id", "Thumbnails"."type" AS "Thumbnails.type", "Thumbnails"."filename" AS "Thumbnails.filename", "Thumbnails"."height" AS "Thumbnails.height", "Thumbnails"."width" AS "Thumbnails.width", "Thumbnails"."fileUrl" AS "Thumbnails.fileUrl", "Thumbnails"."onDisk" AS "Thumbnails.onDisk", "Thumbnails"."automaticallyGenerated" AS "Thumbnails.automaticallyGenerated", "Thumbnails"."videoId" AS "Thumbnails.videoId", "Thumbnails"."videoPlaylistId" AS "Thumbnails.videoPlaylistId", "Thumbnails"."createdAt" AS "Thumbnails.createdAt", "Thumbnails"."updatedAt" AS "Thumbnails.updatedAt" FROM (SELECT * FROM "video" WHERE uuid = true LIMIT 1) AS "video" LEFT OUTER JOIN "thumbnail" AS "Thumbnails" ON "video"."id" = "Thumbnails"."videoId"',
    parameters: undefined
  },
  sql: 'SELECT "video".*, "Thumbnails"."id" AS "Thumbnails.id", "Thumbnails"."type" AS "Thumbnails.type", "Thumbnails"."filename" AS "Thumbnails.filename", "Thumbnails"."height" AS "Thumbnails.height", "Thumbnails"."width" AS "Thumbnails.width", "Thumbnails"."fileUrl" AS "Thumbnails.fileUrl", "Thumbnails"."onDisk" AS "Thumbnails.onDisk", "Thumbnails"."automaticallyGenerated" AS "Thumbnails.automaticallyGenerated", "Thumbnails"."videoId" AS "Thumbnails.videoId", "Thumbnails"."videoPlaylistId" AS "Thumbnails.videoPlaylistId", "Thumbnails"."createdAt" AS "Thumbnails.createdAt", "Thumbnails"."updatedAt" AS "Thumbnails.updatedAt" FROM (SELECT * FROM "video" WHERE uuid = true LIMIT 1) AS "video" LEFT OUTER JOIN "thumbnail" AS "Thumbnails" ON "video"."id" = "Thumbnails"."videoId"',
  parameters: {}
}



Move to external storage failed

Edit: Could be related to this feat: config option object_storage.max_request_attempts by kontrollanten · Pull Request #6418 · Chocobozzz/PeerTube · GitHub

This happens to every new video that tries to get published. It’s the biggest problem.

Job: 537
Type: move-to-object-storage
Processed on 6/16/24, 1:42:52.585 PM
Finished on 6/16/24, 1:42:52.630 PM

{
  "videoUUID": "02627d6c-72fd-4893-8fde-86dd7d47b4c9",
  "isNewVideo": false,
  "previousVideoState": 2
}

InvalidRequest: Missing required header for this request: Content-MD5
    at throwDefaultError (/var/www/peertube/versions/peertube-v6.1.0/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:20)
    at /var/www/peertube/versions/peertube-v6.1.0/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:5
    at de_CommandError (/var/www/peertube/versions/peertube-v6.1.0/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4756:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/www/peertube/versions/peertube-v6.1.0/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /var/www/peertube/versions/peertube-v6.1.0/node_modules/@aws-sdk/middleware-signing/dist-cjs/index.js:192:18
    at async /var/www/peertube/versions/peertube-v6.1.0/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /var/www/peertube/versions/peertube-v6.1.0/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:173:18
    at async /var/www/peertube/versions/peertube-v6.1.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:97:20
    at async /var/www/peertube/versions/peertube-v6.1.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:120:14
    at async /var/www/peertube/versions/peertube-v6.1.0/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async Promise.all (index 0)
    at async _Upload.__uploadUsingPut (/var/www/peertube/versions/peertube-v6.1.0/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:217:22)
    at async _Upload.__doConcurrentUpload (/var/www/peertube/versions/peertube-v6.1.0/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:280:18)
    at async Promise.all (index 0)
    at async _Upload.__doMultipartUpload (/var/www/peertube/versions/peertube-v6.1.0/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:366:5)

Failed to play due to technical issues

Edit: I think I have this one solved. It happens because peertube thinks the video is on the filesystem but it actually still resides on storage.

HLS.js error: networkError - fatal: false - levelLoadError

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://medias.tankie.tube/streaming-playlists/hls/dc46b5d1-fa3a-402c-9929-2ff2121d3777/aadfbdc8-4436-45d6-be2d-ca161fb0c7d8-576.m3u8. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 404.

HTTP load failed with status 404. Load of media resource https://tankie.tube/static/streaming-playlists/hls/b8f3972b-e91a-43bb-8c99-e3851fd46e38/5cb3e73b-0d45-4664-bd07-4016c0943617-720-fragmented.mp4 failed.

I think I found the cause of « external storage move failed ». It was because (Edit: The issue remains.) I had enabled object lock on my backblaze bucket.

I was horrified to learn that backblaze’s object lock cannot be disabled once enabled, but setting the lock retention to zero or null seems to work help.

I say « seems to work » because all but one of the move-to-storage commands I’ve issued are still in the « waiting » or « delayed » state, even after running overnight. I’m hoping that the these move jobs are not hung indefinitely.


By the way, I just recently discovered the /admin/system/jobs panel, which is how I was able to check up on the state of those jobs, and I have to say I’m impressed! This is a really cool and useful page. Beautifully organized, color-coded, and the error messages are neatly included. Wow! I’ve never seen JSON objects visible on the frontend like this before. I’d just like to give a round of applause to that adorable yellow genius, @Chocobozzz, who created this incredible software that has enamored me.

1 Like