A region-based image caption generator with refined descriptions

Kinghorn, Philip, Zhang, Li and Shao, Ling (2017) A region-based image caption generator with refined descriptions. Neurocomputing. ISSN 0925-2312 (In Press)

[img] Text
journal_optimized_revised (003).pdf - Accepted Version
Restricted to Repository staff only until 12 July 2018.

Download (1MB) | Request a copy
Official URL: https://doi.org/10.1016/j.neucom.2017.07.014

Abstract

Describing the content of an image is a challenging task. To enable detailed description, it requires the detection and recognition of objects, people, relationships and associated attributes. Currently, the majority of the existing research relies on holistic techniques, which may lose details relating to important aspects in a scene. In order to deal with such a challenge, we propose a novel region-based deep learning architecture for image description generation. It employs a regional object detector, recurrent neural network (RNN)-based attribute prediction, and an encoder–decoder language generator embedded with two RNNs to produce refined and detailed descriptions of a given image. Most importantly, the proposed system focuses on a local based approach to further improve upon existing holistic methods, which relates specifically to image regions of people and objects in an image. Evaluated with the IAPR TC-12 dataset, the proposed system shows impressive performance and outperforms state-of-the-art methods using various evaluation metrics. In particular, the proposed system shows superiority over existing methods when dealing with cross-domain indoor scene images.

Item Type: Article
Uncontrolled Keywords: Image description generation, Convolutional and recurrent neural networks, Description generation
Subjects: G400 Computer Science
Department: Faculties > Engineering and Environment > Computer Science and Digital Technologies
Depositing User: Becky Skoyles
Date Deposited: 31 Jul 2017 09:17
Last Modified: 06 Aug 2017 02:49
URI: http://nrl.northumbria.ac.uk/id/eprint/31442

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics


Policies: NRL Policies | NRL University Deposit Policy | NRL Deposit Licence