A region-based image caption generator with refined descriptions

Kinghorn, Philip, Zhang, Li and Shao, Ling (2018) A region-based image caption generator with refined descriptions. Neurocomputing, 272. pp. 416-424. ISSN 0925-2312

journal_optimized_revised (003).pdf - Accepted Version

Download (1MB) | Preview
Official URL: https://doi.org/10.1016/j.neucom.2017.07.014


Describing the content of an image is a challenging task. To enable detailed description, it requires the detection and recognition of objects, people, relationships and associated attributes. Currently, the majority of the existing research relies on holistic techniques, which may lose details relating to important aspects in a scene. In order to deal with such a challenge, we propose a novel region-based deep learning architecture for image description generation. It employs a regional object detector, recurrent neural network (RNN)-based attribute prediction, and an encoder–decoder language generator embedded with two RNNs to produce refined and detailed descriptions of a given image. Most importantly, the proposed system focuses on a local based approach to further improve upon existing holistic methods, which relates specifically to image regions of people and objects in an image. Evaluated with the IAPR TC-12 dataset, the proposed system shows impressive performance and outperforms state-of-the-art methods using various evaluation metrics. In particular, the proposed system shows superiority over existing methods when dealing with cross-domain indoor scene images.

Item Type: Article
Uncontrolled Keywords: Image description generation, Convolutional and recurrent neural networks, Description generation
Subjects: G400 Computer Science
Department: Faculties > Engineering and Environment > Computer and Information Sciences
Depositing User: Becky Skoyles
Date Deposited: 31 Jul 2017 09:17
Last Modified: 11 Dec 2018 13:46
URI: http://nrl.northumbria.ac.uk/id/eprint/31442

Actions (login required)

View Item View Item


Downloads per month over past year

View more statistics

Policies: NRL Policies | NRL University Deposit Policy | NRL Deposit Licence