Analysis of music/speech via integration of audio content and functional brain response

Ji, Xiang, Han, Junwei, Jiang, Xi, Hu, Xintao, Guo, Lei, Han, Jungong, Shao, Ling and Liu, Tianming (2015) Analysis of music/speech via integration of audio content and functional brain response. Information Sciences, 297. pp. 271-282. ISSN 0020-0255

Full text not available from this repository. (Request a copy)
Official URL: http://dx.doi.org/10.1016/j.ins.2014.11.020

Abstract

Effective analysis of music/speech data such as clustering, retrieval, and classification has received significant attention in recent years. Traditional methods mainly rely on the low-level acoustic features derived from digital audio stream, and the accuracy of these methods is limited by the well-known semantic gap. To alleviate this problem, we propose a novel framework for music/speech clustering, retrieval, and classification by integrating the low-level acoustic features derived from audio content with the functional magnetic resonance imaging (fMRI) measured features that represent the brain’s functional response when subjects are listening to the music/speech excerpts. First, the brain networks and regions of interest (ROIs) involved in the comprehension of audio stimuli, such as the auditory, emotion, attention, and working memory systems, are located by a new approach named dense individualized and common connectivity-based cortical landmarks (DICCCOLs). Then the functional connectivity matrix measuring the similarity between the fMRI signals of different ROIs is adopted to represent the brain’s comprehension of audio semantics. Afterwards, we propose an improved twin Gaussian process (ITGP) model based on self-training to predict the fMRI-measured features of testing data without fMRI scanning. Finally, multi-view learning algorithms are proposed to integrate acoustic features with fMRI-measured features for music/speech clustering, retrieval, and classification, respectively. The experimental results demonstrate the superiority of our proposed work in comparison with existing methods and suggest the advantage of integrating functional brain responses via fMRI data for music/speech analysis.

Item Type: Article
Uncontrolled Keywords: Audio analysis; fMRI; fMRI-measured feature; Multi-view learning
Subjects: C800 Psychology
G400 Computer Science
Department: Faculties > Engineering and Environment > Computer Science and Digital Technologies
Related URLs:
Depositing User: Paul Burns
Date Deposited: 10 Jun 2015 09:46
Last Modified: 07 Oct 2015 13:44
URI: http://nrl.northumbria.ac.uk/id/eprint/22804

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics


Policies: NRL Policies | NRL University Deposit Policy | NRL Deposit Licence