Frank Zimmer


2020

pdf
Immersive Language Exploration with Object Recognition and Augmented Reality
Benny Platte | Anett Platte | Christian Roschke | Rico Thomanek | Thony Rolletschke | Frank Zimmer | Marc Ritter
Proceedings of the Twelfth Language Resources and Evaluation Conference

The use of Augmented Reality (AR) in teaching and learning contexts for language is still young. The ideas are endless, the concrete educational offers available emerge only gradually. Educational opportunities that were unthinkable a few years ago are now feasible. We present a concrete realization: an executable application for mobile devices with which users can explore their environment interactively in different languages. The software recognizes up to 1000 objects in the user’s environment using a deep learning method based on Convolutional Neural Networks and names this objects accordingly. Using Augmented Reality the objects are superimposed with 3D information in different languages. By switching the languages, the user is able to interactively discover his surrounding everyday items in all languages. The application is available as Open Source.