Videogame to Support the Teaching of Reading to Deaf Children using Gamification

Abstract

In some countries technological alternatives have been developed for the teaching of languages for deaf-mutes, which consider augmented reality, 3d avatars that simulate gestures, gesture recognition gloves, among others; which have shown improvements in the teaching-learning process. [...]the conclusions follow in section 6. [...]the video game captures the image of the gesture by using the webcam and then sends it to the gesture processor. [...]the latter processes the image of the gesture and obtains the spelled letter.

Headnote

Abstract: The present work shows the development of an educational videogame of support for the education of deaf children whose objective is to reinforce their literacy. In this videogame, the children interact using the static gestures of the Peruvian sign language, in this way, they complete the levels previously created by the teacher. In addition, gamification elements that seek to encourage the participation and competitiveness of children are used. The use of the videogame in 9 deaf children shows an increase of 22% in the students' grades, and the SUS test, a high usability of 81/100.

Keywords: gamification; educational videogame; sign language; usability.

1. Introduction

The videogame industry has grown over the years to overcome the film and music industry (Marchand & Henning-Thurau, 2013), thus, it has been incorporated in different sectors such as work operations (Marchand & Henning-Thurau, 2013), marketing (Montola et al., 2009) and in education (Lee, 2012).

Recent research has shown that videogames have a positive impact on children's education and that these improve writing and reading depending on the thematic of the videogame (Cano, Arteaga et al., 2015). For deaf children, sign language is their primary language through which they can learn their mother tongue, which is necessary to not have limitations in their language skills (Cano et al., 2015).

In some countries technological alternatives have been developed for the teaching of languages for deaf-mutes, which consider augmented reality, 3d avatars that simulate gestures, gesture recognition gloves, among others; which have shown improvements in the teaching-learning process. However, they do not consider gamification, a concept that creates motivation and interest in the learning process (Bidarra et al., 2015) and that could improve the teaching-learning process even more. In this work, a low-cost videogame is developed for teaching Spanish to deaf children based on gamification, gesture recognition and rote learning.

The article is organized in 5 sections. In section 2, we make a brief review of the literature on video for deaf children, usability in videogames and gamification. The proposed game model is presented in section 3. In section 4, the implementation of the solution is shown. The validation of the proposed videogame is shown in section 5. Finally, the conclusions follow in section 6.

2.Literature Review

2.1. Videogames for Deaf Children

There are several investigations where videogames that support the education of deaf children have been developed. In (Zikky et al., 2017), a videogame is proposed for the teaching of the Indian sign language, where the movement of the hands is used to play. In (Soares et al., 2015), a videogame is made using Kinect to teach Portuguese sign language, where the goal is to promote the teaching of this language to children from 6 to 10 years old. The game has as mechanics to make the child perform the signs of a certain word that indicated the game while progressing in the story. A videogame for the teaching of the sign language of Macedonia, where there was a 3D avatar that made the gestures that the child had to learn is presented in (Bouzid et al., 2016), where it is shown that the inclusion of a 3D avatar improves the teaching quality. In (Bouzis et al., 2015), a tool was created that allowed the teacher to create his own videogame without the need to know about programming. However, the authors of this research found that this method of teaching lacked certain important elements in videogames, such as the visual part, which was not attractive to the players and there were no rewards or competitiveness between these. A videogame of 2D platforms, which contains 3 game mechanics (3D simulation, memorization, exploration and discovery) for sign teaching in Macedonia, is presented in (Ackovska & Kostoska, 2014), which shows that the most successful mechanics is exploration and discovery, that is, the child was finding packages that contained the signs to learn.

2.2. Usability in Videogames

The usability, according to ISO 9126, is the ability of a software to be understood and liked by the user when it is used under certain specific conditions (Xu et al., 2014).

In (Bronner et al., 2015), it is said that usability can be measured by the documentation, help, tutorials and the user interface provided by the application. A methodology to ensure usability in videogames for deaf children, which considers 5 main factors: efficiency, satisfaction, learning ability, memory facility and low percentage of errors, is given in (Cano et al., 2018). In this research it was found that it is important to take the opinions of children and specialists into account, since children have different difficulties when making gestures or when carrying out certain activities, thereby reducing the cases in which children can not play correctly. In (Frutos-Pascual & Zapirain, 2014), a crossword videogame is developed for children with attention problems, which evaluates its usability using the System Usability Scale (SUS), where they obtained a score of 87.11 out of 100. A videogame was developed for the United States Navy, where the usability of this one was tested using the SUS (Chung & Wu, 2017), obtaining a score of 57.95. In (Gonzalez, 2016), a racing videogame was made that used low-cost technologies for wind simulation. In this research the SUS was also used, where they obtained a score of 85 out of 100. In (Pruna et al., 2017), a videogame was developed for the rehabilitation of people with disabilities in their hands, which validated the usability with the SUS, where a score of 79.5 was obtained.

As evidenced by the literature, the SUS is a mechanism widely used to measure usability, so we will use it to measure the proposed videogame.

2.3.Gamification

The gamification in videogames consists of using game elements, such as points, rewards, points ranking, game levels, among others, in order to increase the players participation and motivate them to use the game (Figueroa, 2015). In this investigation, gamification is used to learn a second language, which describes various elements of gamification that can be used in a video game such as scores obtained while playing, rewards for reaching certain objectives, score comparison tables, avatars that represent the user, system of levels, progress bars, among others. This work recommends the five-step model for the gamification of (Huang & Soman, 2013): understanding the needs of the target audience, defining learning objectives and game requirements, structuring the experience, identifying the resources for gamification and applying the elements of gamification. In (Bidarra et al., 2015), an educational videogame is proposed for the Portuguese sign language where gamification is used, and a system of levels is used through which progress is made based on the objects obtained by the player and a scoring system based on the game time.

3.Videogame Model

It is proposed a game model that can be customized by the class teacher, so that he can freely create levels of game that adjusts to the needs of the moment, that is, he can create games according to the theme that the children require to learn. The model has as components the actors, the basic module, the arcade module, the crossword module, the teacher module and the gestures Peruvian sign language processor. This model can be seen in figure 1.

In order for the model to work correctly, first, the teacher must upload the class subject information corresponding to the basic module through the module of the teacher. The child with hearing disability will learn with the videogame, interacting with the basic module and the game modules. The basic module is intended to reinforce the spelling in the sign language and the reading of words in Spanish, this is done as follows. First, the video game shows the letter or word to spell, and then the child makes the corresponding gesture. Second, the video game captures the image of the gesture by using the webcam and then sends it to the gesture processor. Third, the latter processes the image of the gesture and obtains the spelled letter. Fourth, the spelled letter is shown on the interface, and if it is correct, the child is rewarded. In a similar way, the child interacts with the game modules to reinforce learning, but now in a game context.

3.1. Actors

Deaf children and teachers where the videogame is played are our actors. The teacher is responsible for feeding the game based on the lessons he wants to take in class. The children are the main players, who will have a personal account in which they will keep all their scores and all the medals obtained throughout the use of the videogame.

3.2. Basic Module

This module consists of reinforcing the gestures of sign language and learning, little by little, how words are spelled, thus fulfilling the basic needs of deaf children. The data of this module belongs to the information loaded by the teacher. Using the challenge and sensory immersions, seen in (Kwon & Woo, 2013), the reward-time gamification element was implemented, through which it was sought to slightly pressure the child to fulfill its objective.

3.3. Arcade Module

The arcade module seeks to work the user's memory, giving words to spell until the time runs out. The gamification elements used in this module are score, medals and time limit. With these gamification elements it is sought that children are competitive with each other; thus, they will want to overcome each other while unconsciously working their memory.

3.4. Crossword Module

The crossword module, like the previous module, seeks to work the memory of the child, making him remember the words seen in class sessions. The crosswords of the game are thematic and cover the basic knowledge for children. As elements of gamification, the reward for time was used, where the child starts with 5 medals, which decreases over time.

3.5. Teacher Module

The teacher module is used to upload information, which will allow you to freely design the game levels, both for the learning module and for the arcade module, giving you the facility to create class sessions focused on specific topics.

The teacher is responsible for loading the information of the basic module by using this module. It was taken into account that teachers are allowed to choose the topic to teach and thus create learning phases, which contain letters and words that children will have to spell during the basic game, in such a way that they can cover different topics such as months of the year, the days of the week, among others.

3.6. Gesture Recognizer

This component will be responsible for processing the gesture made by the user, which is received from the videogame. It was taken into account that the teacher, through the use of the gesture dataset, which contains the static gestures of the Peruvian sign language alphabet, obtains the result of the processed gesture, which is returned in response to the videogame. The dataset of gestures, provided by Berrú et al. (2017), contains only 24 letters of the alphabet, where the letters J, Ñ and Z are omitted, since they are represented with dynamic gestures.

4.Videogame: EnSeñas

4.1.Logical Architecture

The logical architecture of the game, showed in figure 2, has two main components: the videogame and the API that contains the gesture processor. Both components work on the Windows 10 operating system. It was thought to work the gesture recognizer as a separate server, in order to have this in the cloud and user it as a service. The advantage of this is that, if the game starts to be used by several institution, all the game instances could be connected to this server, allowing improvements to the dataset or gesture recognizer, making all game instances be benefited.

4.2. Technologies Used

For the development of the game, Unity v 2017.3.0f3 (64-bit) was used as a game engine, together with C # as a programming language. In addition, we used websocket-sharp to create a websocket client in Unity, allowing us to establish a connection with the gesture recognizer. Also, we used Newtonsoft to format the data type to JSON, which is a lightweight format to transport and store data.

For the gesture recognizer, Python v 3.7 was used, together with the Tornado web framework v 5.1.1 in order to create a websocket server, allowing the gesture recognizer to receive all the videogame requests and responding accordingly. OpenCV was used in the gesture recognizer module to process the images received from the websocket server.

4.3. Development Methodology

For the development of the project, the Scrum methodology was used. For this, a product backlog was made with the tasks to be developed, prioritizing them by importance and difficulty. There were 7 sprints, where the artifacts generated in each one were validated with the association of deaf children who helped us during the investigation.

4.4. Game Modules

User Login

When the game was designed, it was taken into account that it was necessary to keep information of each player, their starts, their attempts, etc. and at the same time, it was necessary to give total control to the teacher, who will need to create game phases. Therefore, a user registration and a login module were created.

Teacher Module

In the teacher module, it is allowed to create phases or levels of game, which can be edited and eliminated at any time. The phase creation screen varies depending on which module the teacher wants to configure. If it is the basic module, the teacher will be asked to enter letters and words, the latter formed based on the letters added at the same level and at previously created levels. If it is the arcade module, you will only be asked to enter words without restrictions. This phase creation screen can be seen in figure 3.

Basic Module

The game interface is simple, in order not to distract the child while playing. In the interface is shown the word or letter that the child must spell, the stars that the child has and the time it has before losing of the stars. As time goes by, the green bar will decrease until it is empty, where the child will lose his first star, then the green bar will fill up again and continuously until the child runs out of stars. With this, we want to measure how well and quickly the child was able to finish the phase of the game.

Arcade Module

This game module seeks to reinforce the knowledge of the child by spelling words constantly. The child has a time limit to spell the word that is asked, which decreases every second and increases each time a word is finished. A scoring system is managed, which is awarded based on the number of letters of the word that the child spells. Also, stars will be awarded to the child when it reaches a certain score, up to a maximum of 5 stars (see figure 4). It was sought that the teacher can also customize this mode, in such a way that he can create thematic levels of game or of different difficulties. That is why we also created a teacher module to edit the arcade module.

Crossword Module

In this game module there is a thematic crossword, where the child is presented with several animals, whose images are shown in the initial game interface. These images can be pressed by the child to begin the spelling of the respective animal. To do this, the initial box of the animal inside the crossword puzzle is lit. In addition, a panel is shown where the image captured by the web camera is captured, the figure of the selected animal and the letter that is detected by the gesture recognizer. This panel can be seen in figure 5. Once the child finished spelling the animal, the figure disappears from the list. The player starts with 5 stars, which decrease as time passes, this mechanics being similar to the basic module. With this, we seek to know the time it takes the child to solve the crossword puzzle, measuring the effort it takes to finish it. This module was created in order to see the impact it has on children, so that, if positive, we can create a personalization of crosswords puzzles in the future.

Game Ranking

Each game mode has its own ranking, which is filled from the attempts made by each player. Each attempt is recorded at the end of the game, thus, the game time, the attempt number and the stars obtained are saved. This can be seen in the figure 8. With the implementation of this module, we seek to create competitiveness in children, which is a way to motive them to continue playing, reinforcing their knowledge. It is important to keep the record of each attempt per child, because it is a measure of the improvement that has been throughout the game at that level.

5.Validation

Game sessions were conducted to test how effective the proposed videogame is in the learning process, in addition, System Usability Scale was used to measure its usability.

5.1.Experiment

The validations and tests of the videogame were made in the ENSEÑAS PERÚ association, in a class with 9 children between the ages of 8 to 12 years, as we can see in figure 6. The children in the class to be tested had difficulty learning the names of the months of the year, of which 4 out of 9 children got a passing grade.

5.2. Learning Effectiveness

A normal class session of 2 hours on the writing in Spanish of the months of the year for the 9 deaf children was made. Then the knowledge test was taken to measure what was learned from each student, as seen in table 1, which is calculated from the correct writing of the months. Then there were two sessions of the use of videogames in the classroom, in the first session there was a brief explanation of the videogame and its operation, and each child was allowed to play 8 minutes twice in the arcade mode. In the second session, each child was allowed to play 8 minutes in arcade mode and crossword mode. At the end of the second session of the game, the knowledge test was again taken, where the notes shown in table 1 were obtained.

The use of the videogame helped the children to memorize the writing of some months of the year. As can be seen in Table 1, there was an average increase of 2.37 points in the evaluation note taken after two sessions of videogame use.

5.3. Usability

The SUS is a 10 questions questionnaire where the user's experience of the game is evaluated, and where he was asked to answer on a scale from totally disagree to totally agree (1 - 5) (Lewis & Sauro, 2009).

The teacher was invited to know and use the videogame in the three videogame modules and the teacher's module, in a period of 30 minutes.

At the end of the knowledge test, the evaluation of usability was carried out through the SUS, for this, the teacher asked the questions in both Spanish and sign language to all participating children, who answered the questionnaire for each question. The results of the SUS for the children and the teacher are shown in table 2.

An 81 of 100 points was obtained (M = 81, SD = 3.94), which, compared to past research, it can be said that the game has a good usability.

6.Conclusions

In this work we introduce a videogame model that uses gamification and image processing for the teaching of Spanish to deaf children, as well as its implementation through a videogame that includes a teaching module with three videogames, a teacher module to load teaching topics and a gesture recognizer.

The use of the videogame in 10 children shows an improvement in the learning process, which is reflected in the 22% increase of the average grade. In addition, the SUS usability test shows that the game has a usability of 81/100, above many videogames of the literature.

Acknowledgement

The authors thank the Universidad Peruana de Ciencias Aplicadas (UPC), for the partial funding of the present investigation, the ENSEÑAS PERÚ association, and Bryan Berrú, for providing us their Peruvian sign language gestures dataset.

References

References

Ackovska, N., & Kostoska, M. (2014). Sign language tutor - Rebuilding and optimizing. 37th International Convention on Information and Communication Technology, Electronics and Microelectronics, Opatija. Croacia: IEEE.

Berru-Novoa, B., Gonzalez-Valenzuela, R., & Shiguihara-Juarez, P. (2018). Peruvian sign language recognition using low resolution cameras. 2018 IEEE XXV International Conference on Electronics, Electrical Engineering and Computing (INTERCON). doi:10.1109/intercon.2018.8526408

Bidarra, J., Escudeiro, P., Escudeiro, N., & Barbosa, F. (2015). Game design of content: assessing a project for learning sign language. EDULEARN 2015 - 7th International Conference, Barcelona. España.

Bouzid, Y., Khenissi, M. A., & Jemni, M. (2015). Designing a game generator as an educational technology for the deaf learners. 2015 5th International Conference on Information & Communication Technology and Accessibility (ICTA). doi:10.1109/ icta.2015.7426914

Bouzid, Y., Khenissi, M. A., & Jemni, M. (2016). The effect of avatar technology on sign writing vocabularies acquisition for deaf learners. 2016 IEEE 16th International Conference on Advanced Learning Technologies (ICALT). doi:10.1109/ icalt.2016.127

Bronner, S., Pinsker, R., & Adam Noah, J. (2015). Physiological and psychophysiological responses in experienced players while playing different dance exer-games. Computers in Human Behavior, 51, 34-41. doi:10.1016/j.chb.2015.04.047

Cano, S., Arteaga, J. M., Collazos, C., & Amador, V. B. (2015). Aplicación móvil para el aprendizaje de la lectoescritura con Fitzgerald para niños con discapacidad auditiva. Anais Dos Workshops Do IV Congresso Brasileiro De Informática Na Educaçâo (CBIE 2015). doi:10.5753/cbie.wcbie.2015.240

Cano, S., Arteaga, J. M., Collazos, C. A., & Amador, V. B. (2015). Model for analysis ofserious games for literacy in deaf children from a user experience approach. Proceedings of the XVI International Conference on Human Computer Interaction - Interacción 15, Vilanova i la Geltru. España: ACM.

Cano, S., Collazos, C. A., Aristizábal, L. F., Gonzalez, C. S., & Moreira, F. (2018). Towards a methodology for user experience assessment of serious games with children with cochlear implants. Telematics and Informatics, 35(4), 993-1004. doi:io.ioi6/j. tele.2017.09.011

Chung, S.-M., & Wu, C.-T. (2017). Designing music games and mobile apps for early music learning. In Serious Games and Edutainment Applications (pp. 57-75). doi:10.1007/978-3-319-51645-5_3

Figueroa, F. (2015). Using gamification to enhance second language learning. Revistes Científiques de la Universität de Barcelona. Retrieved from: http://revistes. ub.edu/index.php/der/article/view/11912

Frutos-Pascual, M., & Zapirain, B. G. (2014). Guided crossword-puzzle games aimed at children with Attentional Deficit: Preliminary results. 2014 Computer Games: AI, Animation, Mobile, Multimedia, Educational and Serious Games (CGAMES). doi:10.1109/cgames.2014.6934137

Gonzalez, J. (2016). Una propuesta metodológica para la gamificación de pruebas psicométricas. Un caso práctico. Congreso Ambiental Sobre Soluciones en Ingeniería Ambiental, Ingeniería de Software y Salud Electrónica y Móvil, 1(1), 71-78. Retrieved from: https://revistas.utp.ac.pa/index.php/memoutp/article/ view/1300

Huang Hsin Yuan, W., & Soman, D. (2013). A practitioner's guide to gamification of education. Research Report Series: Behavioral Economics in Action. University of Toronto -Rotman. School of Management.

Kwon, C., & Woo, T. (2013). A research on gamification methodology for korean language education. Journal of Korea Game Society, 13(1), 61-74. doi:10.7583/ jkgs.2013.13.1.61

Lee, J. (2012). Study on the successful case of gamification and its application to e-Book, Policy Industrial Graduate School, Sookmyung Women's University, 2012.

Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. Human Centered Design Lecture Notes in Computer Science, 94-103. doi:10.1007/978-3642-02806-9_12

Marchand, A., & Hennig-Thurau, T. (2013). Value creation in the video game industry: industry economics, consumer benefits, and research opportunities. Journal of Interactive Marketing, 27(3), 141-157. doi:10.1016/j.intmar.2013.05.001

Montola, M., Stenros, J., Waern, A., Pervasive games: theory and design. Morgan Kaufmann Publishers, 2009. doi:10.1201/9780080889795

Pruna, E., Acurio, A., Escobar, I., Albiol, S., Zumbana, P., Meythaler, A., & Álvarez, F. (2017). 3D Virtual System Using a Haptic Device for Fine Motor Rehabilitation. Advances in Intelligent Systems and Computing Recent Advances in Information Systems and Technologies, 648-656. doi:10.1007/978-3-319-56538-5_66

Soares, F., Esteves, J. S., Carvalho, V., Lopes, G., Barbosa, F., & Ribeiro, P. (2015). Development of a serious game for Portuguese Sign Language. 2015 7th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT). doi:10.1109/icumt.2015.7382432

Xu, J., Ding, X., Huang, K., & Chen, G. (2014). A pilot study of an inspection framework for automated usability guideline reviews of mobile health applications. In Proceedings of the Wireless Health 2014 on National Institutes of Health - WH '14. doi:10.1145/2668883.2669585

Zikky, M., Hakkun, R. Y., Rizqi, A. F., Hamid, A., & Basuki, A. (2017). Development of educational game for recognizing indonesian sign language (SIBI) and breaking down the communication barrier with deaf people. 2017 21st International Computer Science and Engineering Conference (ICSEC). doi:10.1109/icsec.2017.8443936