Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval

Abstract

Cross-modal retrieval aims to retrieve data in one modality by a query in another modality, which has been avery interesting research issue in the filed of multimedia, information retrieval, and computer vision, anddatabase. Most existing works focus on cross-modal retrieval between text-image, text-video, and lyrics-audio.little research addresses cross-modal retrieval between audio and video due to limited audio-video paireddataset and semantic information. The main challenge of audio-visual cross-modal retrieval task focuses on learning joint embeddings from a shared subspace for computing the similarity across different modalities, were generating new representations is to maximize the correlation between audio and visual modalities space. In this work, we propose a novel deep triplet neural network with cluster-based canonical correlationanalysis (TNN-C-CCA), which is an end-to-end supervised learning architecture with audio branch and videobranch. we not only consider the matching pairs in the common space but also compute the mismatching pairs when maximizing the correlation. In particular, two significant contributions are made in this work: i) abetter representation by constructing deep triplet neural network with triplet loss for optimal projections canbe generated to maximize correlation in the shared subspace. ii) positive examples and negative examplesare used in the learning stage to improve the capability of embedding learning between audio and video. Our experiment is run over 5-fold cross-validation, where average performance is applied to demonstratethe performance of audio-video cross-modal retrieval. The experimental results achieved on two different audio-visual datasets show the proposed learning architecture with two branches outperforms the state-of-art cross-modal retrieval methods.