7
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: not found
      • Article: not found

      Deep Visual-Semantic Alignments for Generating Image Descriptions

      Read this article at

      ScienceOpenPublisherPubMed
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks (RNN) over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions outperform retrieval baselines on both full images and on a new dataset of region-level annotations. Finally, we conduct large-scale analysis of our RNN language model on the Visual Genome dataset of 4.1 million captions and highlight the differences between image and region-level caption statistics.

          Related collections

          Most cited references21

          • Record: found
          • Abstract: not found
          • Conference Proceedings: not found

          Show and tell: A neural image caption generator

            Bookmark
            • Record: found
            • Abstract: not found
            • Conference Proceedings: not found

            BLEU

              Bookmark
              • Record: found
              • Abstract: not found
              • Conference Proceedings: not found

              CIDEr: Consensus-based image description evaluation

                Bookmark

                Author and article information

                Journal
                IEEE Transactions on Pattern Analysis and Machine Intelligence
                IEEE Trans. Pattern Anal. Mach. Intell.
                Institute of Electrical and Electronics Engineers (IEEE)
                0162-8828
                2160-9292
                April 2017
                April 2017
                : 39
                : 4
                : 664-676
                Article
                10.1109/TPAMI.2016.2598339
                27514036
                d5922672-f269-4cd3-a0a0-85e88b673b41
                © 2017
                History

                Comments

                Comment on this article