76
views
0
recommends
+1 Recommend
1 collections
    1
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Brain-inspired multimodal learning based on neural networks

      Read this article at

      ScienceOpenPublisher
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Modern computational models have leveraged biological advances in human brain research. This study addresses the problem of multimodal learning with the help of brain-inspired models. Specifically, a unified multimodal learning architecture is proposed based on deep neural networks, which are inspired by the biology of the visual cortex of the human brain. This unified framework is validated by two practical multimodal learning tasks: image captioning, involving visual and natural language signals, and visual-haptic fusion, involving haptic and visual signals. Extensive experiments are conducted under the framework, and competitive results are achieved.

          Related collections

          Author and article information

          Contributors
          Journal
          Brain Sci. Adv.
          Brain Science Advances
          Tsinghua University Press and SAGE Publishing
          2096-5958
          25 November 2018
          : 4
          : 1
          : 61-72
          Affiliations
          1Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
          Author notes
          Address correspondence to Fuchun Sun, E-mail: fcsun@ 123456mail.tsinghua.edu.cn
          Article
          BSA.2018.9050004
          10.26599/BSA.2018.9050004
          © The authors 2018. This article is published with open access at journals.sagepub.com/home/BSA

          This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License ( http://www.creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page ( https://us.sagepub.com/en-us/nam/open-access-at-sage).

          Product
          Self URI (journal page): https://journals.sagepub.com/home/bsa
          Categories
          Research Article

          Comments

          Comment on this article