33
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Good News, Everyone! Context driven entity-aware captioning for news images

      Preprint
      , , ,

      Read this article at

          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Current image captioning systems perform at a merely descriptive level, essentially enumerating the objects in the scene and their relations. Humans, on the contrary, interpret images by integrating several sources of prior knowledge of the world. In this work, we aim to take a step closer to producing captions that offer a plausible interpretation of the scene, by integrating such contextual information into the captioning pipeline. For this we focus on the captioning of images used to illustrate news articles. We propose a novel captioning method that is able to leverage contextual information provided by the text of news articles associated with an image. Our model is able to selectively draw information from the article guided by visual cues, and to dynamically extend the output dictionary to out-of-vocabulary named entities that appear in the context source. Furthermore we introduce `GoodNews', the largest news image captioning dataset in the literature and demonstrate state-of-the-art results.

          Related collections

          Most cited references20

          • Record: found
          • Abstract: not found
          • Book Chapter: not found

          Microsoft COCO: Common Objects in Context

            • Record: found
            • Abstract: not found
            • Conference Proceedings: not found

            Show and tell: A neural image caption generator

              • Record: found
              • Abstract: not found
              • Conference Proceedings: not found

              CIDEr: Consensus-based image description evaluation

                Author and article information

                Journal
                02 April 2019
                Article
                1904.01475
                2d6360d8-002c-43dd-9c21-7f1a01ab4ad8

                http://arxiv.org/licenses/nonexclusive-distrib/1.0/

                History
                Custom metadata
                IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019)
                cs.CV

                Computer vision & Pattern recognition
                Computer vision & Pattern recognition

                Comments

                Comment on this article

                Related Documents Log