A Review of Emotion Recognition Methods from Keystroke, Mouse, and Touchscreen Dynamics

Yang, Liying and Qin, Sheng-feng (2021) A Review of Emotion Recognition Methods from Keystroke, Mouse, and Touchscreen Dynamics. IEEE Access, 9. pp. 162197-162213. ISSN 2169-3536

FINAL Article.pdf - Accepted Version
Available under License Creative Commons Attribution 4.0.

Download (1MB) | Preview
Final published version.pdf - Published Version
Available under License Creative Commons Attribution 4.0.

Download (5MB) | Preview
Official URL: https://doi.org/10.1109/ACCESS.2021.3132233


Emotion can be defined as a subject’s organismic response to an external or internal stimulus event. The responses could be reflected in pattern changes of the subject’s facial expression, gesture, gait, eye-movement, physiological signals, speech and voice, keystroke, and mouse dynamics, etc. This suggests that on the one hand emotions can be measured/recognized from the responses, and on the other hand they can be facilitated/regulated by external stimulus events, situation changes or internal motivation changes. It is well-known that emotion has a close relationship with both physical and mental health, usually affecting an individual’s and a team’s work performance, thus emotion recognition is an important prerequisite for emotion regulation towards better emotional states and work performance. The primary problem in emotion recognition is how to recognize a subject’s emotional states easily and accurately. Currently, there are a body of good research on emotion recognition from facial expression, gesture, gait, eye-tracking, and other physiological signals such as speech and voice, but they are all intrusive and obtrusive to some extent. In contrast, keystroke, mouse and touchscreen (KMT) dynamics data can be collected non-intrusively and unobtrusively as secondary data responding to primary physical actions, thus, this paper aims to review the state-of-the-art research on emotion recognition from KMT dynamics and to identify key research challenges, opportunities and a future research roadmap for referencing. In addition, this paper answers the following six research questions (RQs): (1) what are the commonly used emotion elicitation methods and databases for emotion recognition? (2) which emotions could be recognized from KMT dynamics? (3) what key features are most appropriate for recognizing different specific emotions? (4) which classification methods are most effective for specific emotions? (5) what are the application trends of emotion recognition from KMT dynamics? (6) which application contexts are of greatest concern?

Item Type: Article
Additional Information: Funding information: The authors would like to thank the financial support from China CSC Studentship (No. 201906300082) and Newton Prize 2019 Award: NP2PB100047.
Uncontrolled Keywords: Affective computing, keystroke dynamics, mouse dynamics, touchscreen dynamics, emotional signal features, emotion elicitation, emotion recognition, machine learning, applications of emotion recognition
Subjects: W200 Design studies
Department: Faculties > Arts, Design and Social Sciences > Design
Depositing User: John Coen
Date Deposited: 01 Dec 2021 11:16
Last Modified: 21 Jan 2022 15:15
URI: http://nrl.northumbria.ac.uk/id/eprint/47869

Actions (login required)

View Item View Item


Downloads per month over past year

View more statistics