3DPalsyNet: A Facial Palsy Grading and Motion Recognition Framework using Fully 3D Convolutional Neural Networks

Storey, Gary, Jiang, Richard, Keogh, Shelagh, Bouridane, Ahmed and Li, Chang-Tsun (2019) 3DPalsyNet: A Facial Palsy Grading and Motion Recognition Framework using Fully 3D Convolutional Neural Networks. IEEE Access, 7. pp. 121655-121664. ISSN 2169-3536

[img]
Preview
Text
08811497.pdf - Published Version
Available under License Creative Commons Attribution 4.0.

Download (1MB) | Preview
[img]
Preview
Text
AAM_IEEEAccess-2019-26263.pdf - Accepted Version

Download (10MB) | Preview
Official URL: https://doi.org/10.1109/ACCESS.2019.2937285

Abstract

The capability to perform facial analysis from video sequences has significant potential to positively impact in many areas of life. One such area relates to the medical domain to specifically aid in the diagnosis and rehabilitation of patients with facial palsy. With this application in mind, this paper presents an end-to-end framework, named 3DPalsyNet, for the tasks of mouth motion recognition and facial palsy grading. 3DPalsyNet utilizes a 3D CNN architecture with a ResNet backbone for the prediction of these dynamic tasks. Leveraging transfer learning from a 3D CNNs pre-trained on the Kinetics data set for general action recognition, the model is modified to apply joint supervised learning using center and softmax loss concepts. 3DPalsyNet is evaluated on a test set consisting of individuals with varying ranges of facial palsy and mouth motions and the results have shown an attractive level of classification accuracy in thesetasks of 82% and 86% respectively. The frame duration and the loss function affect was studied in terms of the predictive qualities of the proposed 3DPalsyNet, where it was found shorter frame duration’s of 8 performed best for this specific task. Centre loss and softmax have shown improvements in spatio-temporal feature learning than softmax loss alone, this is in agreement with earlier work involving the spatial domain.

Item Type: Article
Uncontrolled Keywords: Computer vision, face detection, facial action recognition, machine learning.
Subjects: G400 Computer Science
G500 Information Systems
G600 Software Engineering
Department: Faculties > Engineering and Environment > Computer and Information Sciences
Depositing User: Elena Carlaw
Date Deposited: 22 Aug 2019 12:42
Last Modified: 11 Oct 2019 13:02
URI: http://nrl.northumbria.ac.uk/id/eprint/40440

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics


Policies: NRL Policies | NRL University Deposit Policy | NRL Deposit Licence