top of page

Do You Speak Holo?

Service

Mixed Reality (MR)

Industry

Education

Year

2020

Do-You-Speak-Holo-Cover.jpg

Do You Speak Holo?, is an application for Microsoft HoloLens co-designed with linguistics experts that aims to facilitate the learning process of a foreign language for children and adolescents with language disorders (LD).

 

The application includes several educational activities, aimed at improving both vocabulary and morpho-syntactic aspects of the language through the interaction with virtual content immersed in the real world.

 

The application includes a virtual assistant, which acts as a virtual teacher and guides the user step by step in the comprehension and execution of the activities.

Project content 🎯

Problems, audience and main challenge

Main Target

People with language disorder 

Educational Context

E.g. School, Therapy

Needs

Training in foreign language

Constraints

Difficulty performing testing

Children comprehension ability

Goals

Learning process trough

MR application

Scenarios 🎮

Three playable learning experiences

Each activity seeks to promote the learning of a particular set of words (vocabulary) and morpho-syntactic structures

of the English language such as functional words (possessives, prepositions of place, comparatives, and superlatives)

in a context-oriented activity using MR. At the current stage of development, we designed and developed three scenarios, of which we provide some examples taken from the practice phases:

Possessive adjectives: the virtual teacher asks the user to move the holographic items into baskets belonging to different characters. The digital content is based on the vocabulary of fruits and vegetables, for instance: “This is her apple” 

Comparatives and superlatives: in this scenario, we focus on the vocabulary of animals. The virtual teacher asks the user to identify an animal according to a comparative/superlative relationship with the other ones and to move it toward a holographic representation of Noah’s ark. For instance: “Move the animal that is bigger than the rabbit but smaller than the rhino”.

 

Prepositions of place: the vocabulary involves school/office items. The user is asked Possessive to tidy up a real desk (recognized by the device thanks to spatial mapping capabilities), moving the virtual objects in some specific spots, according to the prepositions of place identified by the virtual teacher. For instance: “Pick the rubber behind the ruler

and put it on the book”.

Possessive adjectives

Comparatives & Superlatives

Prepositions of place

Levels 🏋️‍♀️

Select your level of the game difficulty

Comparatives & Superlatives

Comparatives & Superlatives

Comparatives & Superlatives

Less is More 👁️

Focus on user attention

In designing the menu for the Mixed Reality app, I focused on the principle of 'less is more' to enhance user experience and maintain immersion. Given the unique affordances of VR/AR/MR environments, where users interact within a 180-to-360-degree space, it was crucial to avoid overwhelming them with excessive options or complex navigation.

By placing the menu within a narrow 90-degree field of view, I reduced the need for excessive neck movement and avoided the overwhelm of 'Terminator'-style interfaces. Studies show that limiting the field of view improves user comprehension.

Menu-1-Wide.png

The menus are purposely designed with large buttons to accommodate the air-tap gesture used in Microsoft HoloLens. The buttons are separated for easier selection and interaction within the MR space, ensuring an intuitive, seamless, and engaging user experience.

3D Sceens 🏡

Environments for effective learning

Scene-1.png

The first scenario is set on a farm, with a house, a tree, two baskets of fruits, three farmer characters, and our avatar to guide the game.

The second scenario features Noah's Ark along with animals such as a koala, lion, and rabbit. As the game goes, more animals will be included.

Scene-2.png

The third scenario includes school objects for learning, ideal for playing on a desk surface, featuring items like a lamp, pencil eraser, sharpener, mug, and a book.

For faster rendering and animation, it was crucial to create the characters in 3D with correct topology and a minimal number of polygons to avoid bugs and lagging.

Virtual Assistant 🤖

A friendly character to assist kids in learning!

An animated virtual assistant is integrated into the activities and plays the role of a friendly virtual teacher, illustrating the words/linguistic constructs during the learning phase, explaining to the user what to do in the practice phase, and giving feedback depending on whether the performed action is correct or wrong, and ultimately promoting attention and fun.

 

Concerning the choice of the avatar, we opted for a character that is easily recognizable and appreciated especially by younger people. After considering more options, our decision was Groot, from the movie Guardians of the Galaxy.

Showreel 🎬

Take a couple of minutes to watch this video. It showcases the final app prototype we designed and developed, demonstrating how children learn languages through an engaging and interactive mixed reality game experience.

Future Works 🔮

What are we going to develop next?

Do You Speak Holo? is an innovative MR application designed to help people with language disorders learn a foreign language. The potential of Do You Speak Holo? is that it has been co-designed with linguistics experts to address the real needs of people with LD during the learning process, and the use of HoloLens, which has been used to stimulate the association between visual elements and their corresponding term in the FL.

 

Our research agenda foresees many points in which this work can be extended. From a technological perspective, it is fundamental to have designed Do You Speak Holo? in a highly modular and extensible way: new activities, vocabulary, virtual teacher avatars, and languages can be integrated with a little effort.

 

Another essential extension will be enabling the user to practice language production. To achieve this, we will use speech recognition techniques to simulate a simple spoken interaction between the user and the virtual teacher.

 

We will perform an experimentative evaluation on Do You Speak Holo? since we believe it will be helpful in real-world use cases. A systematic empirical study is required to assess its effectiveness for people with LD and to gather feedback from users, to understand the strengths and weaknesses of the experience.

More languages

Flags.png

More virtual teacher behaviours

Beaviours.png

More virtual teachers

Virtual teachers.png

Mobile / Tablet application

Mobile _ Tablet.png

Please continue here! ✌️

This project also includes a second part, which involves designing a web platform that can be used on both tablets and desktops by educators, along with a fully immersive VR experience for children.

The team 🫶

Dream combo!

Bogdan Novakovic

Designer

Ibrahim El Shemy

Developer

Luca Teraccianno

Developer

Mattia Righetti

Developer

Franca Garzotto

Mentor

Special Thanks! 🙏

I want to thank my teammates Ibrahim El Shemy, Mattia Righetti, and Luca Terracciano for their fundamental contributions to the development of the application, as well as the research team of the INCLUDI project for their support during the design phase.

bottom of page