Hand Gesture Recognition using Multi-Scale Colour Features, Hierarchical Models and Particle Filtering

Lars Bretzner, Ivan Laptev and Tony Lindeberg

Proc. Face and Gesture 2002, Washington DC, 423--428.

Abstract

This paper presents algorithms and a prototype system for hand tracking and hand posture recognition. Hand postures are represented in terms of hierarchies of multi-scale colour image features at different scales, with qualitative inter-relations in terms of scale, position and orientation. In each image, detection of multi-scale colour features is performed. Hand states are then simultaneously detected and tracked using particle filtering, with an extension of layered sampling referred to as hierarchical layered sampling. Experiments are presented showing that the performance of the system is substantially improved by performing feature detection in colour space and including a prior with respect to skin colour. These components have been integrated into a real-time prototype system, applied to a test problem of controlling consumer electronics using hand gestures. In a simplified demo scenario, this system has been successfully tested by participants at two fairs during 2001.

PDF: (159 kb)

Video clips illustrating the performance of the method for real-time hand tracking and gesture recognition:

Example of internal states of the multi-state particle filtering algorithm (click for larger image):

Background and related material: (Tracking of multi-state hand models using particle filtering and a hierarchy of multi-scale image features) (Qualitative multi-scale feature hierarchies for object tracking) (Feature tracking with automatic selection of spatial and temporal scales) (Example of demo scenario for gesture control)

Related project: Computer vision based human-computer interaction