Sign Language Communication through Augmented Reality and Speech Recognition (LEARNSIGN)

Sign Language Communication through Augmented Reality and Speech Recognition (LEARNSIGN)

  IJETT-book-cover  International Journal of Engineering Trends and Technology (IJETT)          
  
© 2021 by IJETT Journal
Volume-69 Issue-4
Year of Publication : 2021
Authors : Siti Nurulain Mohd Rum, Braxton Isaiah Boilis
DOI :  10.14445/22315381/IJETT-V69I4P218

Citation 

MLA Style: Siti Nurulain Mohd Rum, Braxton Isaiah Boilis  "Sign Language Communication through Augmented Reality and Speech Recognition (LEARNSIGN)" International Journal of Engineering Trends and Technology 69.4(2021):125-130. 

APA Style:Siti Nurulain Mohd Rum, Braxton Isaiah Boilis. Sign Language Communication through Augmented Reality and Speech Recognition (LEARNSIGN)  International Journal of Engineering Trends and Technology, 69(4),125-130.

Abstract
For the last few decades, Augmented Reality (AR) has become a popular technology for real-time interaction in virtual worlds. The advancement of AR is the result of an increase in the number of devices that able to unleash the potential of this technology. The primary value of AR is the way it combines the components of the artificial world with the experience of one`s physical world. Studies show that this powerful feature can increase the effectiveness of the teaching and learning process in many fields. In this paper, we present the development of LEARN SIGN, an android-based mobile application for learning sign language. With the incorporation of AR and speech detection technologies, this system can also help people with disabilities, especially the deaf and mute community, to communicate with non-disabled people and vice-versa. To realize this research work, several technologies have been used, such as Unity Software and C# programming language, to develop the proposed system.

Reference
[1] R. Aggarwal and A. Singhal, Augmented Reality and its effect on our life., in 2019 9th International Conference on Cloud Computing, Data Science & Engineering (Confluence), (2019) 510-515.
[2] P. Schueffel, The concise FINTECH compendium, Fribourg, Switzerland, (2017).
[3] P. Thomas and W. David, Augmented reality: An application of heads-up display technology to manual manufacturing processes, in Hawaii international conference on system sciences, (1992) 659-669.
[4] M. Bajura, H. Fuchs, and R. Ohbuchi, Merging virtual objects with the real world: Seeing ultrasound imagery within the patient, ACM SIGGRAPH Computer Graphics,26(203-210)(1992).
[5] M. Z. Masmuzidin and N. A. A. Aziz, The current trends of augmented reality in early childhood education, The International Journal of Multimedia & Its Applications (IJMA), 10(47)(2018).
[6] A. Nesterov, I. Kholodilin, A. Shishkov, and P. Vanin, "Augmented reality in engineering education: Opportunities and advantages, Communications-Scientific letters of the University of Zilina, 19(2017) 117-120.
[7] D. Sahin and R. M. Yilmaz, The effect of Augmented Reality Technology on middle school students` achievements and attitudes towards science education, Computers & Education, 144(2020) 103710.
[8] S. Cai, E. Liu, Y. Shen, C. Liu, S. Li, and Y. Shen, Probability learning in mathematics using augmented reality: impact on student’s learning gains and attitudes, Interactive Learning Environments, 28(2020) 560-573.
[9] X. Ran, C. Slocum, M. Gorlatova, and J. Chen, ShareAR: Communication-efficient multi-user mobile augmented reality, in Proceedings of the 18th ACM Workshop on Hot Topics in Networks, (2019) 109-116.
[10] T. Khan, K. Johnston, and J. Ophoff, The impact of an augmented reality application on learning motivation of students, Advances in Human-Computer Interaction, (2019).
[11] H. F. Hanafi, C. S. Said, M. H. Wahab, and K. Samsuddin, Improving students’ motivation in learning ict course with the use of a mobile augmented reality learning environment, in IOP Conference Series: Materials Science and Engineering, (2017)(012114).
[12] G. Kiryakova, N. Angelova, and L. Yordanova, The potential of augmented reality to transform education into smart education, TEM Journal,7(556)(2018).
[13] H. Ahmad, N. M. M. Zainuddin, and R. C. M. Yusoff Augmented Operational Reality Framework to Aid Al-Quran Memorization for Hearing Impaired Students, Open International Journal of Informatics (OIJI), 6(2018) 22-32.
[14] P. Vijayalakshmi and M. Aarthi, Sign language to speech conversion, in 2016 International Conference on Recent Trends in Information Technology (ICRTIT), (2016) 1-6.
[15] A. R. Hasdak, I. Al Nur, A. Al Neon, and H. U. Zaman, Deaf-Vibe: A Vibrotactile Communication Device Based on Morse Code for Deaf-Mute Individuals, in 2018 9th IEEE Control and System Graduate Research Colloquium (ICSGRC), (2018) 39-44.
[16] C. Baker and D. Cokely, American sign language," Silver Spring, MD: TJ Publishers, (1980).
[17] R. Saha, A. Sharma, and M. Srivastava, Psychiatric assessment of deaf and mute patients–A case series, Asian journal of psychiatry, 25(2017) 31-35.
[18] M. J. Cheok, Z. Omar, and M. H. Jaward, A review of hand gesture and sign language recognition techniques, International Journal of Machine Learning and Cybernetics, 10(2019) 131-153.
[19] P. Milgram and F. Kishino, A taxonomy of mixed reality visual displays, IEICE TRANSACTIONS on Information and Systems, 77(1994) 1321-1329.
[20] A. Van Dam, A. S. Forsberg, D. H. Laidlaw, J. J. LaViola, and R. M. Simpson, Immersive VR for scientific visualization: A progress report, IEEE Computer Graphics, and Applications, 20(2000) 26-52.
[21] W. R. Sherman and A. B. Craig, Understanding virtual reality: Interface, application, and design: Morgan Kaufmann, (2018).
[22] I. Radu and B. Schneider, What Can We Learn from Augmented Reality (AR)? Benefits and Drawbacks of AR for Inquiry-based Learning of Physics, in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, (2019)(1-12).
[23] A. Petrovskaya and P. Varvak, Optimizing head-mounted displays for augmented reality, ed: Google Patents, (2018).
[24] H. M. Nguyen and E. L. Coons, Augmented reality glasses, ed: Google Patents, (2018).
[25] S. Serubugo, D. Škantárová, L. K. Nielsen, and M. Kraus, Comparison of wearable optical see-through and handheld devices as a platform for an augmented reality museum guide, in International Conference on Computer Graphics Theory and Applications, (2017) 179-186.
[26] R. Silva, J. C. Oliveira, and G. A. Giraldi, Introduction to augmented reality, National laboratory for scientific computation, 11(2003).
[27] T. Plewan, B. Mättig, V. Kretschmer, and G. Rinkenauer, Exploring the benefits and limitations of augmented reality for palletization, Applied Ergonomics, 90(2021) 103250.
[28] B. Forutanpour, S. Balasubramanyam, and L. E. Mann IV, Input method designed for augmented reality goggles, ed: Google Patents, (2015).
[29] N. Shimoda, Y. Fujita, T. Arakawa, and A. Misawa, "Head-up display, ed: Google Patents, (2020).
[30] T. A. Furness III and J. S. Kollin, Virtual retinal display and method for tracking eye position, ed: Google Patents, (2001).
[31] A. K. O. Sudana, I. G. A. A. M. Aristamy, and N. K. A. Wirdiani Augmented reality application of sign language for deaf people in Android based on the smartphone, International Journal of Software Engineering and Its Applications, 10(2016) 139-150.
[32] J. J. C. Garnica and M. A. G. Arrieta, Augmented reality sign language teaching model for deaf children, in Distributed Computing and Artificial Intelligence, 11th International Conference, (2014) 351-358.
[33] A. Vaitkevi?ius, M. Taroza, T. Blažauskas, R. Damaševi?ius, R. Maskeli?nas, and M. Wo?niak, Recognition of American sign language gestures in virtual reality using leap motion, Applied Sciences, 445(9)(2019).
[34] N. M. M. Zainuddin, H. B. Zaman, and A. Ahmad, "Developing augmented reality book for the deaf in science: the determining factors, in 2010 International Symposium on Information Technology, (2010) 1-4.
[35] M. Billinghurst, H. Kato, and I. Poupyrev, The MagicBook: a transitional AR interface, Computers & Graphics, 25(2001) 745-753.
[36] B. Noor-Un-Nissah and M. H. Joseph, SignAR: A Sign Language Translator Application With Augmented Reality, Journal of Applied Technology and Innovation (e-ISSN: 2600-7304), 3(2019).

Keywords
Augmented Reality (AR), Sign Language, Disabled Community