Self Organizing Markov Map for Speech and Gesture Recognition

Ms. Nutan D. Sonwane, Prof. S.A. Chhabria, Dr. R.V. Dharaskar


Gesture and Speech based human Computer interaction is attractive attention across various areas such as pattern recognition, computer vision. Thus kind of research areas find many kind of application in Multimodal HCI, Robotics control, Sign language recognition. This paper presents head and hand Gesture as well as Speech recognition system for human computer interaction (HCI).This kind of vision based system can show the capability of computer, which understand and responding to the hand and head gesture also for Speech in form of sentence. This recognition system consists of two main modules namely 1.Gesture recognition 2.Speech recognition, Gesture recognition consists of various phases.i. image capturing, ii. Feature extraction of gesture iii.Gesture modeling (Direction, Position, generalized), 2.Speech recognition consists of various phases i. taking voice signals ii. Spectral coding iii. Unit matching (BMU) iv. Lexical decoding v.syntactic, semantic analysis.  Compared with many existing algorithms for gesture and speech recognition, SOM provides flexibility, robustness against noisy environment. The detection of gestures is based on discrete predestinated symbol sets, which are manually labeled during the training phase. The gesture-speech correlation is modelled by examining the co-occurring speech and gesture patterns. This correlation can be used to fuse gesture and speech modalities for edutainment applications (i.e. video games, 3-D animations) where natural gestures of talking avatars are animated from speech. A speech driven gesture animation example has been implemented for demonstration.


Soloman Raju Kota,J.L Reheja,Ashutosh Gupta,Archna rathi , Shashikant Sharma”Principal component analysis for Gesture recognition using systemC”2009 international conferences in advance technology in communication and computing 2009IEEE

Yean Choon Ham, Yu Shi “Developing a Smart Camera for Gesture Recognition in HCI Applications” The 13th IEEE International Symposium on Consumer Electronics (ISCE2009) 978-1-4244-2976-9/09/$25.00 ©2009 IEEE

E. Stergiopoulou and N. Papamarkos “A New Technique For Hand Gesture Recognition” 1-4244-0481-9/06/ © 2006 IEEE

Anjali Kalra, Sarbjeet Singh, Sukhvinder Singh”SpeechRecognition” International Journal of Computer Science and Network Security, VOL.10,2010.

George Caridakis , Kostas Karpouzis, Athanasios Drosopoulos, Stefanos Kollias” SOMM: Self organizing Markov map for gesture recognition” Pattern Recognition Letters 31, 2010

WU Song-Lin, CUI Rong-Yi “Human Behavior Recognition Based on Sitting Postures” 2010 International Symposium on Computer, Communication, Control and Automation. 978-1-4244-5567-6/10/ © 2010 IEEE

Jagdish Lal Raheja, Radhey shyam “Real Time Robotic Hand Control Using Hand Gesture” 978-0-7695-3977-5/10 © 2010 IEEE.

Mr. Chetan A. Burande, Prof. Raju M. Tugnayat, Prof.Dr. Nitin K. Choudhary “Advanced Recognition Techniques for Human Computer Interaction.” 978-1-4244-5586-7/10. 2010 IEEE

Shuai Jin, Guang-ming Lu, Jian-xun Luo, Wei-dong Chen Xiao-xiang Zheng ”SOM-based Hand Gesture Recognition for Virtual Interactions” in IEEE International Symposium on Virtual Reality Innovation 2011.

G.R.S Murthy, R.S Jadon “Hand gesture recognition using neural network” in 2nd International Advance Computing Conference 2010 Mr. Chetan A. Burande, Prof. Raju M. Tugnayat, Prof.Dr. Nitin K. Choudhary “Advanced Recognition Techniques for Human Computer Interaction.” 978-1-4244-5586-7/10. 2010 IEEE

M. Ajallooeian, A. Borji, B. N. Araabi , M. Nili Ahmadabadi, H. Moradi “Fast Hand Gesture Recognition based on Saliency Maps: An Application to Interactive Robotic Marionette Playing” The 18th IEEE International Symposium on Robot and Human Interactive Communication Toyama, Japan, Sept. 27-Oct. 2, 2009. 978-1-4244-5081-7 /09/ ©2009 IEEE

wei-hua andrew wang, chun-liang tung Proceedings of the Seventh International Conference on Machine Learning and Cybernetics, Kunming, 12-15 July 2008 “Dynamic Hand Gesture Recognition Using Hierarchical Dynamic Bayesian Networks Through Low-Level Image Processing.” 978-1-4244-2096-4/08 ©2008 IEEE

Sridhar P. Arjunan, Dinesh K. Kumar School of Electrical and Computer Engineering “Recognition of facial movements and hand gestures using surface Electromyogram (sEMG) for HCI based applications”. 0-7695-3067-2/07 © 2007 IEEE

T Nakanot , T Mori&, M. Nagata , and A. Iwatat “A Cellular-Automaton-Type Image Extraction Algorithm and Its Implementation Using An Fpga” 0-7803-7690-0/02/$17.00 @2002 IEEE

Full Text: PDF


  • There are currently no refbacks.


All Rights Reserved © 2012 IJARCSEE

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.