Suggestion : automated subtitles/ closed captions

Hi everyone ! :love_you_gesture:

I admit i’m kinda sad because most of peertube’s videos aren’t accessible for deaf people. :sob:

I remember the first day of youtube, we created FB group to request subtitles on various videos with a team of interpreters. Today, youtube does it automatically, it’s magical ! It’s not flawless but it allowed me watch lot more video. However there 2 issues :

  • Youtube removed the possibility to edit the subtitle :-1:
  • video creator can remove the CC buttons. :-1:

So, it deactivate the possibility to benefit from automated subtitles.

So i cocked up some ideas, which i hope it will catches the attention of developers. BTW, I read there were only 1 Dev which is crazy and amazing O.O

I want to improves accessibility for everyone, there is deaf people but also old people too, hard hearing… There is also bad mic or bad speakers. I’m not aware of the feasibility but if it is impossible, just tell me :slight_smile:

Here is my dream video player :star_struck:

CC button alway displayed with several option
Always display close captions button, don’t allow the host or uploader to decide the fate of this precious button. Anyway, it won’t activate subtitles by default but people can activate or add subtitles. The close caption button could purpose 4 options :

  • Activate/Deactivate subtitles
  • Activate/Deactivate Sign languages
  • Request subtitles/Sign langages (link to a subtitle company)
  • Add subtitles/Sign language (locked by default so only registered user can add subtitle/ sign video)

Transcript
Use subtitles to create an transcript below the video or an document with the full text to download. (See ted talk for reference). The transcription should be below the video in a collapsible element, so we can see comments. Or alternative design, we display transcription side to side with the video. The text follow the video.

Transcript text boost web search
And transcript is an amazing way the improve video search because of keyword. When we search a video, we are limited by its title, description or keyword. If the whole video’s audio is turned into a transcript, Sepia could use it. Imagines, i don’t remember the title but i know some guy said that and this or that the video what about this topic. Dunno if it would improve search result but why not ?

Editing/adding subtitles
Since subtitle are collaborative, I think it should work as mediawiki. A page with revisions and various element to create a group of trusted users to avoid vandalism issues.(And an IP blacklist across all peertube instance ?)

Adding Sign video
I’m not really fan of this solution, however it could improve deaf accessibility by leap and bound as their comprehension skill are limited due to a lack of good pedagogy and interpreters. Here, the main problem is the server memory. We will have 2 videos : The video and the sign language video so it will increase bandwidth and server’s memory a lot. Anyway, the sign language video would appear in a resizable.

Requesting video/subtitle
It would be a cool feature. I’m watching a video and want to paid subtitle so my deaf/ hard hearing friend can watch it. Peertube/Framasoft can create a page with a list of studios willing to add subtitle/sign langage on peertube per country. So when we request the subtitle, we will be directed to this page, we filter by country, choose a company and then fill a contact form with : Peertube Instance, Video name, subject, video ID, and email and send it to them. Then they will send us the cost. Maybe we can imagine a price based on video duration so we can paid them directly

Automated close captions
Integrate a speech to text based on language (AVA ? Mozilla deepspeech ? ). We can have the option to register the subtitle so we can edit them later and improve the result.

Thank for reading my long text. What do you think ? If this project is impossible, just be blunt so i don’t get my hope up. but i also want to understand why it won’t work :slight_smile:

1 Like

There is a french project, named S2S, that provides a Peertube instance with signed videos (or subtitles for deaf people).

When i heard of it, my first ideas was to create a plugin to match sign video and original video, and playback the two side by side. The basic idea: you upload the signed version, and add a link to the original video in a special field. The original video could be on another instance.
Then, when you play the video, the 2 will be side by side and synchronized.

This plugin does not exist yet, but if people are interested, we could mount a team, and search for some money to pay the team members (i have some leads to find sponsors).

The french education ministry produced a plugin that can make automatic captions (using Vosk):

This is also this fork of the previous project, that i don’t know:

(i don’t know the difference between the two)

Thank for your help !

Yeah, i knew this instance since 2020, but it’s the only one i can follow and it’s only about deaf community. I would like to follow other video to improve my knownledge on libre software.

Wonderful idea ! Sadly, i can’t help i’m just an sysadmin (during my free time) and a farmer. I think we can find some deaf developpers if we make a video in international sign language. I know there is also some foundation related to deaf and education (in Europe), we tried during the Deaf Alternative International project. But we didn’t pursue it as they didn’t want to paid the Dev’s team.

Yeah i go there sometime :slight_smile: I sent an mail so the sysadmin could update it’s instance to the v5 since it was stuck to v2. One of my deaf friend related is to s2s. I’m not sure if he is the admin (or you ?) but we will see ! I will meet him the upcoming days during the hacker space factory. I will talk about it, i hope he can teaches me how to manage a peertube serveur too. :slight_smile:

So that mean i need to be root to activate this plugin…That somewhat limit the accessibility whereas i want this plugin activated by default as part of peertube player. As for the difference, there is only one way : testing ! Right now, I’m trying to install vosk live caption but it’s not easy, my installation failled :

I have on my todo list to write a document to present this project. To see if the deaf community is interested, and if so, to begging searching some funding (I’m sure that NlNet would be interested!).

I would be very happy to work on such project (i’m kind of «professional peertube plugin dev», lol). But of course, if there are deaf devs that want to participate, that would be awesome, and I can let them do it (priority for the concerned people!).

I personnaly can’t start working on this project before end 2023.
If you find people that want to work on it before that, i can help to find funding. As said, I have some leads.

i did not test this plugin for now (that is also on my todo list…). Can’t help you, but the dev is sometime reading this forum. Maybe he will see this post.

1 Like

If you have the document ready, i can help you translating into french sign language. Some of my friend can do that in international sign language, so we may reach deaf dev. The only problem may be the communication.

As for the fund, i think we may be able to ask for European’s fund. They have lot project related to accessibility. However, i don’t think it will be easy : Current Projects - European Union of the Deaf

I’ll let you know when the document is ready. It will be a written one, so no need to sign it.