Analysis of music/speech via integration of audio content and functional brain response

Ji, Xiang, Han, Junwei, Jiang, Xi, Hu, Xintao, Guo, Lei, Han, Jungong, Shao, Ling and Liu, Tianming (2015) Analysis of music/speech via integration of audio content and functional brain response. Information Sciences, 297. pp. 271-282. ISSN 0020-0255

Full text not available from this repository. (Request a copy)

Abstract

Effective analysis of music/speech data such as clustering, retrieval, and classification has received significant attention in recent years. Traditional methods mainly rely on the low-level acoustic features derived from digital audio stream, and the accuracy of these methods is limited by the well-known semantic gap. To alleviate this problem, we propose a novel framework for music/speech clustering, retrieval, and classification by integrating the low-level acoustic features derived from audio content with the functional magnetic resonance imaging (fMRI) measured features that represent the brain’s functional response when subjects are listening to the music/speech excerpts. First, the brain networks and regions of interest (ROIs) involved in the comprehension of audio stimuli, such as the auditory, emotion, attention, and working memory systems, are located by a new approach named dense individualized and common connectivity-based cortical landmarks (DICCCOLs). Then the functional connectivity matrix measuring the similarity between the fMRI signals of different ROIs is adopted to represent the brain’s comprehension of audio semantics. Afterwards, we propose an improved twin Gaussian process (ITGP) model based on self-training to predict the fMRI-measured features of testing data without fMRI scanning. Finally, multi-view learning algorithms are proposed to integrate acoustic features with fMRI-measured features for music/speech clustering, retrieval, and classification, respectively. The experimental results demonstrate the superiority of our proposed work in comparison with existing methods and suggest the advantage of integrating functional brain responses via fMRI data for music/speech analysis.

Item Type:	Article
Uncontrolled Keywords:	Audio analysis; fMRI; fMRI-measured feature; Multi-view learning
Subjects:	C800 Psychology G400 Computer Science
Department:	Faculties > Engineering and Environment > Computer and Information Sciences
Depositing User:	Paul Burns
Date Deposited:	10 Jun 2015 09:46
Last Modified:	10 Oct 2019 18:00
URI:	http://nrl.northumbria.ac.uk/id/eprint/22804

Downloads