Vincent Vandeghinste, Mirella De Sisto, Dimitar Shterionov, Aoife Brady, Mathieu De Coster, Lorraine Leeson, Josep Blat, Frankie Picron, Marcello Scipioni, Aditya Parikh, Louis ten Bosch, John O’Flaherty, Joni Dambre and Jorn Rijckaert
The SignON project (https://signon-project.eu/) is a Horizon 2020 project that focuses on the research and development of a Sign Language (SL) translation mobile application and an open communications framework. According to the World Federation of the Deaf (WFD), Sign Languages (SLs) are the primary means of communication for over 70 million deaf and hard of hearing (DHH) individuals. Despite this, these languages are rarely included in the ongoing developments of NLP and other language technology advancements (Yin et al., 2021). Machine translation (MT) research which target SLs is still in its infancy, due mainly to the lack of data and effective representation of signs (including the lack of a standardized written form for Sls).
SignON rectifies the lack of technology and services for the automatic translation between signed and spoken languages, through an inclusive, human-centric solution which facilitates communication between deaf, hard of hearing (DHH) and hearing individuals. The project revolves around 4 spoken languages (English, Spanish, Dutch, Irish) and 5 sign languages (ISL, NGT, VGT, LSE and BSL), aiming to provide translation services for all possible pairs and for all modalities (audio and text for spoken languages and video for SLs).
Developing translation solutions for this many language pairs (72 pairs) is a computationally challenging task. Building models for each of them would require a substantial amount of time and effort, far beyond the time-span of the project. In SignON, we employ a pipeline approach that (i) focuses on processing and understanding individual languages, (ii) employs a common multi-lingual representation (InterL) to facilitate translation and (iii) uses symbolic as well as deep learning (DL) methods for the synthesis of a 3D virtual signer.
Along with the technological and academic innovations that come in terms of new models, methods and methodologies for SLMT, SignON strives towards having a large societal impact. First, that is only possible through the effective delivery of the service to the user groups. To this end, we develop a mobile application and a cloud-based framework. The SignON App connects users to the SignON Framework which handles the internal data flow and processing. The first release of the SignON Mobile Application is due in June 2022, and will then evolve to its final release at the end of the project (December 2023). Second, we need to ensure that the developed solutions are in line with the needs and expectations of the potential users. Currently we face societal challenges such as clashes between the views of deaf, hard of hearing and hearing people, with respect to use-cases, technological importance and communication needs. To address these challenges we organized 2 sets of interviews with deaf participants, an online survey and we have 2 round tables planned. Via workshops we inform both the research and user communities about the progress of SignON and the state-of-the-art in SLMT.
Through the research and development on these languages, SignON aims to become a foundation for the current and future development of automatic translation for SLs.