Vision based real-time continuous 3D hand gesture recognition interface for generic applications based on input-output hidden Markov models

Abstract

This study focuses on the application of InputOutput Hidden Markov Models (IOHMM) to the recognition of 3D hand gestures that involve both hand motion and hand posture in a stereo visionbased approach. The proposed system is designed as a real time gestural interface that allows both communicative and manipulative gestures to control target PC applications. The system allows training of new communicative gestures and automatically distinguishes these from manipulative gestures in continuous streams. Uniquely colored gloves with no preset colors are used as markers, which increases the recognition rate and simplifies the hand localization problem. Camera calibration and gesture training tools are provided with the system. The hand shape is modeled with Hu moments and the angle of the hand. As a novel approach the inclusion of normalized time information as an input to the IOHMMs is proposed, which has the effect of a continuous state variable and is shown to handle temporal information better. Another novel method is proposed for gesture spotting in IOHMMbased frameworks, which distinguishes the meaningful gestures in a continuous input stream in real time using a threshold model with a single hidden state. The system is tested on two datasets that have 10 gestures with distinct trajectories, and 20 gestures in 10 pairs that share the same trajectories. The proposed system is able to attain a recognition rate of 97.6% on the former dataset in 2885 trials, and a recognition rate of 94.1% on the latter dataset in 8675 trials. The gesture segmentation test results for the same datasets with the inclusion of the proposed threshold model are 97.1% for the former and 93.8% for the latter dataset.