Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition

Wu, Di, Pigou, Lionel, Kindermans, Pieter-Jan, Le, Nam, Shao, Ling, Dambre, Joni and Odobez, Jean-Marc (2016) Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38 (8). pp. 1583-1597. ISSN 0162-8828

Full text not available from this repository. (Request a copy)

Abstract

This paper describes a novel method called Deep Dynamic Neural Networks (DDNN) for multimodal gesture recognition. A semi-supervised hierarchical dynamic framework based on a Hidden Markov Model (HMM) is proposed for simultaneous gesture segmentation and recognition where skeleton joint information, depth and RGB images, are the multimodal input observations. Unlike most traditional approaches that rely on the construction of complex handcrafted features, our approach learns high-level spatio-temporal representations using deep neural networks suited to the input modality: a Gaussian-Bernouilli Deep Belief Network (DBN) to handle skeletal dynamics, and a 3D Convolutional Neural Network (3DCNN) to manage and fuse batches of depth and RGB images. This is achieved through the modeling and learning of the emission probabilities of the HMM required to infer the gesture sequence. This purely data driven approach achieves a Jaccard index score of 0.81 in the ChaLearn LAP gesture spotting challenge. The performance is on par with a variety of state-of-the-art hand-tuned feature-based approaches and other learning-based methods, therefore opening the door to the use of deep learning techniques in order to further explore multimodal time series data.

Item Type:	Article
Uncontrolled Keywords:	Deep learning, convolutional neural networks, deep belief networks, gesture recognition, hidden Markov models
Subjects:	G400 Computer Science
Department:	Faculties > Engineering and Environment > Computer and Information Sciences
Depositing User:	Becky Skoyles
Date Deposited:	06 Apr 2016 15:43
Last Modified:	12 Oct 2019 22:52
URI:	http://nrl.northumbria.ac.uk/id/eprint/26502

Downloads