Movement sonification expectancy model: leveraging musical expectancy theory to create movement-altering sonifications

Newbold, Joseph, Gold, Nicolas E. and Bianchi-Berthouze, Nadia (2020) Movement sonification expectancy model: leveraging musical expectancy theory to create movement-altering sonifications. Journal on Multimodal User Interfaces, 14 (2). pp. 153-166. ISSN 1783-7677

Preview

Text Newbold2020_Article_MovementSonificationExpectancy.pdf - Published Version Available under License Creative Commons Attribution 4.0. Download (1MB) | Preview

Abstract

When designing movement sonifications, their effect on people’s movement must be considered. Recent work has shown how real-time sonification can be designed to alter the way people move. However, the mechanisms through which these sonifications alter people’s expectations of their movement is not well explained. This is especially important when considering musical sonifications, to which people bring their own associations and musical expectation, and which can, in turn, alter their perception of the sonification. This paper presents a Movement Expectation Sonification Model, based on theories of motor-feedback and expectation, to explore how musical sonification can impact the way people perceive their movement. Secondly, we present a study that validates the predictions of this model by exploring how harmonic stability within sonification interacts with contextual cues in the environment to impact movement behaviour and perceptions. We show how musical expectancy can be built to either reward or encourage movement, and how such an effect is mediated through the presence of additional cues. This model offers a way for sonification designers to create movement sonifications that not only inform movement but can be used to encourage progress and reward successes.

Actions (login required)

View Item

Downloads