Assignment 3 Report

2 years ago by TAmbrose

Share this post:

Video showing application in action, apologies for not cropping.

Introduction

This is an interactive language learning augmented reality application. The goal of this application is to teach basic Indonesian language structure and vocabulary.

Description of application

The application uses a web camera to dynamically track flash cards with Indonesian words on them. The list of image targets includes basic Indonesian vocabulary for animals, colors and size. The application takes the tracked cards and displays the translated sentence in the bottom left of the screen along with 3D models for the animals. The main focus of the application is on Indonesian sentence structure which is similar to English but with a few key differences that can be confusing for beginners. The key difference between English and Indonesian structure being highlighted in this application is that Indonesian sentences usually have the verb following the subject, e.g. "pig green" rather than "green pig" [1].

Language learning can be quite a challenge for many students, to master a new language they must learn both vocabulary and grammar. Indonesian, whilst being quite simple in comparison for English speakers to learn due to its similarities, still proves challenging for many students [2]. The use of augmented reality in this application provides a unique environment where a student can easily check their vocabulary knowledge and play around with sentence structure in a fun and simple way. Whilst more conventional methods of learning sentence structure such as wrote learning can achieve a similar result it is not as engaging and feedback is not received as quickly. Through conventional methods a student might write down their sentence and wait for it to be checked by a teacher or check it against other sentences, using the application a student gets instant feedback in the form of the translated sentence and the 3D model. This instant feedback leads to a more streamlined and efficient learning experience, where students can easily explore sentence structure.

Interaction Design

The interactions included in this application are, adding/removing words in the displayed sentence, changing the color of the 3D animal models, changing the size of the 3D animal, and changing the position of words in the displayed sentence.

Adding/removing words from the displayed sentence is a key interaction, it allows the user to pick a word that they would like to know the translation of and display it by simply showing that card to the camera. If there are no words in the sentence and a user shows one to the camera, that word's translation is added to the sentence in the bottom left of the UI. If there are less than three words in the current sentence then any new word that the user wants to add must be placed next to the current word(s) in the sentence. This is illustrated in the storyboard bellow.

Created using: https://www.storyboardthat.com/storyboard-creator

Changing the color of the 3D animal models gives the user further visual feedback when a color is placed next to an animal, this makes the application more engaging for the user. When a user puts a color card next to a tracked animal card the animal model will change to match the specified color. This is illustrated in the storyboard above.

Changing the position of words in the displayed sentence allows the user to experiment with the order of the sentence. All image targets have left and right colliders so that the translated sentence translates not only the vocabulary but also the structure of the sentence. If the user wants to change the structure of the current sentence they just have to re-arrange the cards in the desired order and the change will be displayed. This is illustrated in the storyboard bellow.

Created using: https://www.storyboardthat.com/storyboard-creator

Changing the size of the 3D animal models provides a similar benefit to that of changing the color, in that it provides more user feedback. This interaction can be seen in the storyboard above.

All user interactions are achieved through the manipulation of the cards, this makes the application uniquely accessible. Compared with interactions involving mouse and keyboard or touch screen this applications interactions are quite simple, the user only needs to arrange the cards in the desired order.

Technical Development

The specific interface technology that this application was developed for is augmented reality through the use of a web camera and the vuforia engine. In order to use this application users must also have all of the image targets printed as all user interactions are achieved through the manipulation of these.

- Animal cards: Domba, Babi, Ayam, Anjing, Burung, Kucing

- Color cards: Merah, Biru, Kuning, Hijau, Jingga, Ungu, Hitam, Putih

- Size cards: Besar, Kecil

Words can be added to a sentence with no words in it by simply showing the camera the desired card.

Words can be added to a sentence with words in it by placing the card next to cards currently in the sentence.

Words can be removed from the sentence by taking cards out of view of the camera.

Color of an animal can be changed by placing any of the color cards next to an animal card.

Size of an animal can be changed by placing either size card next to an animal card.

Descriptions of 3D Models

Pig, Sheep and Chicken model