Modality independent adversarial network for generalized zero shot image classification

Zhang, Haofeng, Wang, Yinduo, Long, Yang, Yang, Longzhi and Shao, Ling (2021) Modality independent adversarial network for generalized zero shot image classification. Neural Networks, 134. pp. 11-22. ISSN 0893-6080

NRL_47921.pdf - Accepted Version
Available under License Creative Commons Attribution Non-commercial No Derivatives 4.0.

Download (3MB) | Preview
Official URL:


Zero Shot Learning (ZSL) aims to classify images of unseen target classes by transferring knowledge from source classes through semantic embeddings. The core of ZSL research is to embed both visual representation of object instance and semantic description of object class into a joint latent space and learn cross-modal (visual and semantic) latent representations. However, the learned representations by existing efforts often fail to fully capture the underlying cross-modal semantic consistency, and some of the representations are very similar and less discriminative. To circumvent these issues, in this paper, we propose a novel deep framework, called Modality Independent Adversarial Network (MIANet) for Generalized Zero Shot Learning (GZSL), which is an end-to-end deep architecture with three submodules. First, both visual feature and semantic description are embedded into a latent hyper-spherical space, where two orthogonal constraints are employed to ensure the learned latent representations discriminative. Second, a modality adversarial submodule is employed to make the latent representations independent of modalities to make the shared representations grab more cross-modal high-level semantic information during training. Third, a cross reconstruction submodule is proposed to reconstruct latent representations into the counterparts instead of themselves to make them capture more modality irrelevant information. Comprehensive experiments on five widely used benchmark datasets are conducted on both GZSL and standard ZSL settings, and the results show the effectiveness of our proposed method.

Item Type: Article
Additional Information: Funding information: This work was supported in part by National Natural Science Foundation of China (NSFC) under Grants No. 61872187, No. 62072246 and No. 61929104, in part by the Natural Science Foundation of Jiangsu Province under Grant No. BK20201306, in part by the Medical Research Council (MRC) Innovation Fellowship (UK) under Grant No. MR/S003916/1, and in part by the “111” Program under Grant No. B13022.
Uncontrolled Keywords: Generalized Zero Shot Learning (GZSL), Orthogonal constraint, Cross reconstruction, Adversarial network, Modality independent learning
Subjects: G400 Computer Science
G500 Information Systems
Department: Faculties > Engineering and Environment > Computer and Information Sciences
Depositing User: Elena Carlaw
Date Deposited: 07 Dec 2021 13:07
Last Modified: 10 Jan 2022 15:55

Actions (login required)

View Item View Item


Downloads per month over past year

View more statistics