Elisa Colletti


2025

pdf bib
DEEP: an automatic bidirectional translator leveraging an ASR for translation of Italian sign language
Nicolas Tagliabue | Elisa Colletti | Francesco Roberto Dani | Roberto Tedesco | Sonia Cenceschi | Alessandro Trivilini
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)

DEEP is a bidirectional translation system for the Italian Sign Language, tailored to two specific, common use cases: pharmacies and the registry office of the municipality, for which a custom corpus has been collected. Two independent pipelines permit to create a chatlike interaction style, where the deaf subject just signs in front of a camera, and sees a virtual LIS interpreter, while the hearing subject reads and writes messages into a chat UI. The LIS-to-Italian pipeline leverages, in a novel way, a customized Whisper model (a wellknown speech recognition system), by means of “pseudo-spectrograms”. The Italian-to-LIS pipeline leverages a virtual avatar created with Viggle.ai. DEEP has been evaluated with LIS signers, obtaining very promising results.