Do You Speak Holo?
Service
Mixed Reality (MR)
Industry
Education
Year
2020
Do You Speak Holo?, is an application for Microsoft HoloLens co-designed with linguistics experts that aims to facilitate the learning process of a foreign language for children and adolescents with language disorders (LD).
The application includes several educational activities, aimed at improving both vocabulary and morpho-syntactic aspects of the language through the interaction with virtual content immersed in the real world.
The application includes a virtual assistant, which acts as a virtual teacher and guides the user step by step in the comprehension and execution of the activities.
Project content 🎯
Problems, audience and main challenge
Main Target
People with language disorder
Educational Context
E.g. School, Therapy
Needs
Training in foreign language
Constraints
Difficulty performing testing
Children comprehension ability
Goals
Learning process trough
MR application
Scenarios 🎮
Three playable learning experiences
Each activity seeks to promote the learning of a particular set of words (vocabulary) and morpho-syntactic structures
of the English language such as functional words (possessives, prepositions of place, comparatives, and superlatives)
in a context-oriented activity using MR. At the current stage of development, we designed and developed three scenarios, of which we provide some examples taken from the practice phases:
Possessive adjectives: the virtual teacher asks the user to move the holographic items into baskets belonging to different characters. The digital content is based on the vocabulary of fruits and vegetables, for instance: “This is her apple”
Comparatives and superlatives: in this scenario, we focus on the vocabulary of animals. The virtual teacher asks the user to identify an animal according to a comparative/superlative relationship with the other ones and to move it toward a holographic representation of Noah’s ark. For instance: “Move the animal that is bigger than the rabbit but smaller than the rhino”.
Prepositions of place: the vocabulary involves school/office items. The user is asked Possessive to tidy up a real desk (recognized by the device thanks to spatial mapping capabilities), moving the virtual objects in some specific spots, according to the prepositions of place identified by the virtual teacher. For instance: “Pick the rubber behind the ruler
and put it on the book”.
Possessive adjectives
Comparatives & Superlatives
Prepositions of place
Levels 🏋️♀️
Select your level of the game difficulty
Comparatives & Superlatives
Comparatives & Superlatives
Comparatives & Superlatives
Less is More 👁️
Focus on user attention
In designing the menu for the Mixed Reality app, I focused on the principle of 'less is more' to enhance user experience and maintain immersion. Given the unique affordances of VR/AR/MR environments, where users interact within a 180-to-360-degree space, it was crucial to avoid overwhelming them with excessive options or complex navigation.
By placing the menu within a narrow 90-degree field of view, I reduced the need for excessive neck movement and avoided the overwhelm of 'Terminator'-style interfaces. Studies show that limiting the field of view improves user comprehension.
The menus are purposely designed with large buttons to accommodate the air-tap gesture used in Microsoft HoloLens. The buttons are separated for easier selection and interaction within the MR space, ensuring an intuitive, seamless, and engaging user experience.
3D Sceens 🏡
Environments for effective learning
The first scenario is set on a farm, with a house, a tree, two baskets of fruits, three farmer characters, and our avatar to guide the game.
The second scenario features Noah's Ark along with animals such as a koala, lion, and rabbit. As the game goes, more animals will be included.
The third scenario includes school objects for learning, ideal for playing on a desk surface, featuring items like a lamp, pencil eraser, sharpener, mug, and a book.
For faster rendering and animation, it was crucial to create the characters in 3D with correct topology and a minimal number of polygons to avoid bugs and lagging.
Virtual Assistant 🤖
A friendly character to assist kids in learning!
An animated virtual assistant is integrated into the activities and plays the role of a friendly virtual teacher, illustrating the words/linguistic constructs during the learning phase, explaining to the user what to do in the practice phase, and giving feedback depending on whether the performed action is correct or wrong, and ultimately promoting attention and fun.
Concerning the choice of the avatar, we opted for a character that is easily recognizable and appreciated especially by younger people. After considering more options, our decision was Groot, from the movie Guardians of the Galaxy.
Showreel 🎬
Take a couple of minutes to watch this video. It showcases the final app prototype we designed and developed, demonstrating how children learn languages through an engaging and interactive mixed reality game experience.
Future Works 🔮
What are we going to develop next?
Do You Speak Holo? is an innovative MR application designed to help people with language disorders learn a foreign language. The potential of Do You Speak Holo? is that it has been co-designed with linguistics experts to address the real needs of people with LD during the learning process, and the use of HoloLens, which has been used to stimulate the association between visual elements and their corresponding term in the FL.
Our research agenda foresees many points in which this work can be extended. From a technological perspective, it is fundamental to have designed Do You Speak Holo? in a highly modular and extensible way: new activities, vocabulary, virtual teacher avatars, and languages can be integrated with a little effort.
Another essential extension will be enabling the user to practice language production. To achieve this, we will use speech recognition techniques to simulate a simple spoken interaction between the user and the virtual teacher.
We will perform an experimentative evaluation on Do You Speak Holo? since we believe it will be helpful in real-world use cases. A systematic empirical study is required to assess its effectiveness for people with LD and to gather feedback from users, to understand the strengths and weaknesses of the experience.
More languages
More virtual teacher behaviours
More virtual teachers
Mobile / Tablet application
The team 🫶
Dream combo!
Bogdan Novakovic
Designer
Ibrahim El Shemy
Developer
Luca Teraccianno
Developer
Mattia Righetti
Developer
Franca Garzotto
Mentor