The development of an app for people with visual impairment in collaboration with Sber

As part of the Golden Mask Festival 2017, Sberbank, together with the Russian Society of the Blind (RSB), decided to present the first Russian application for listening to the commentary on theatrical productions. We were invited to be a technology partner and executor of the project.

To create a unique mobile app with tiflo-commentary for theatre-goers, we had to analyze available technical solutions and offer a suitable option for the theater.

It was necessary to take into account the specific features of users when designing and developing the app, so that a person with visual impairments can use applications without any difficulties.

In addition to iOS and Android applications, we have developed a backend, a database, and an operator panel, which is used by a responsible employee to launch audio descriptions at the right time of the performance. This provides the simplest and most reliable system operation scenario.

To start listening to the tiflo-commentary, a user only needs to come to the theater, run the application, put on the headphones and that’s it! When the performance begins, the user will hear an audio description of what is happening. Connection to Wi-Fi happens automatically on Android. iPhone technically does not have this feature, so the application tells you to which network connect to, and offers to do so.

The format of the application with tiflo-commentary was used only for movies before. But unlike the cinema in the theater, the launch of the tiflo-commentary can not be automated. There can be delays, improvisation, and changes in the scenario.

We analyzed 3 ways of synchronization to link the comments to the pauses in the performances and proposed the best option in terms of implementation time, reliability, and usability. The option of using the local network turned out to be the most reliable and required minimal effort from users.

In theaters, there are often problems with the connection. Sometimes there are special signal jammers, so as not to interfere with the performance. Therefore, the presence of a local network is a necessary condition for the reliable transmission of information. Each theater participating in the project was given a customized Wi-Fi router and a laptop — they just need to be plugged into the outlet and the tiflo-commentary infrastructure is ready.

A tiflo operator is a person who runs audio files and accompanies the performance. Similar technology is used to provide subtitles in operas. Therefore, in every theater, there is a staff member who can cope with this task. We created a simple application for running audio files. It is not difficult to work with the operator panel, and it could be easily introduced into any theater.

What should be the app interface for people who do not see the screen? How do they understand what to tap? The design and font in the application «Iskusstvo.Vsluh» are tailored to meet the needs of the visually impaired and are coordinated with the RSB. Such voice interface technology support as TalkBack for Android and VoiceOver for iOS allows even completely blind people to navigate the application.

The testers specially checked all functions with their eyes closed to make sure that the visually impaired people can use the application and to check the correctness of the voice assistant work. The Russian Society of the Blind members also took part in testing the application.

Tiflo-commentary was voiced by professional actors: Chulpan Khamatova, Yuliya Peresild, and Anatoly Bely. It allows the users to feel the theater atmosphere even more. Now people with visual impairment have an opportunity to participate in theatrical life.

It was a pleasure to take part in such a kind and bright project. In addition to interesting technical problems, we solved the main one — a human one. We helped people feel the theater better.