By Ingrid Fadelli
Researchers at NIT Silchar, India, have recently developed a new dynamic hand-gesture-based keyboard character recognition system. This virtual keyboard system, presented in a paper published in Springer’s Journal of Ambient Intelligence and Humanized Computing, uses an image-based approach for gesture recognition that is pattern, speed and scale invariant in nature.
“Gesture recognition is a promising field of study due to its huge range of applications,” Songhita Misra, one of the researchers who carried out the study, told TechXplore. “A gesture recognition system can be applied in virtual reality systems, augmented reality, health care, vehicles, to aid patients with vision or motion impairments, for home appliances, robots, mining and several other applications, which are increasing with every passing day.”
Gesture recognition could enhance human-computer interactions in a variety of areas. Although gesture recognition tools could have a vast range of applications, so far, very few organizations and institutions have tried to introduce these systems into society.
“While doing the literature survey, I observed that most existing studies are either limited to static gestures or small duration dynamic gestures such as ‘move left,’ ‘move right,’ ‘click,’ ‘stop,’ etc., which can basically replace the traditional mouse and T.V. remotes at most,” Songhita said. “With the increase in the demand of applications, the complexity of the system from the designer end is bound to increase. Therefore, a thorough study and analysis in the field of long dynamic systems is required.”
Flow chart of the proposed hierarchical classification model. Credit: S. Misra & R. H. Laskar.
Traditional keyboards support a broad range of characters, including capital and small English letters, arithmetic operators, Arabic numbers, and other printable ASCII characters. A gesture recognition system that covers all these characters is very challenging to develop due to its significant database requirements, as well as possible complications associated with hand detection, tracking, extraction of features and the use of classifiers.
In their recent study, Songhita and her colleagues set out to develop a virtual keyboard system with approximately 95 characters. Yet, due to the difficulties associated with this task, their system currently supports 58.
“Our team, which includes my guide Dr. Rabul Hussain Laskar, Dr. Joyeeta singha and me, managed to develop a 58 printable keyboard character system using both color markers and bare hand,” Songhita explained. “Our research in this field started back in 2013 at our Speech and Image lab at NIT Silchar.”
The researchers developed a hierarchical approach to gesture recognition that is based on self co-articulation, position and trajectory features. Existing state-of-the-art models for gesture recognition are based on temporal trajectory features, which are dependent on the frame-wise 2-D sequential path followed by particular gestures.
The 58 keyboard characters classified in the study. Credit: S. Misra & R. H. Laskar.
Due to this dependency, the features analyzed by these approaches can be affected by trajectory noise or other variations in pattern, speed or scale. The approach devised by Songhita and her colleagues, on the other hand, uses image models that are not obtained frame-wise, and are hence unaffected by pattern, speed, scale or trajectory variations.
The researchers fused these image-based and trajectory features in a hybrid hierarchical classification model. Their model achieved 3.9 percent greater accuracy than a baseline non-hierarchical trajectory classification model, with lower misclassification rates for characters such as ‘0’ and ‘O’ or ‘Z’ and ‘2’.
“The extended version of our work has been approved by IMPRINT-II for sponsorship under SERB, DST, India, for a duration of three years,” Songhita said. “Our project, which is in collaboration with IIT Guwahati, was among the 121 projects selected from 2000+ proposals. This is a big achievement for us, as well as for the institute. Ours will definitely be one of the first projects in India to solely focus on the development of a virtual text-entry interface system.”
The recent study carried out by Songhita and her colleagues focused on developing a hierarchical classification model that can tackle large databases without reducing the system’s accuracy. The aim of the broader project approved by IMPRINT-II, however, will be to develop a disabled-friendly gesture recognition system for 95 printable keyboard characters using both color markers and bare-hand detection. Once this system is complete, it will be deployed for use by elderly and vision-impaired users, as well as others who might benefit from it.
“Development of such a large vocabulary system will be a challenging task,” Songhita said. “Up until now, we have developed a 58-character recognition system using vision based techniques.”
