Brain-inspired multimodal learning based on neural networks

Abstract

Modern computational models have leveraged biological advances in human brain research. This study addresses the problem of multimodal learning with the help of brain-inspired models. Specifically, a unified multimodal learning architecture is proposed based on deep neural networks, which are inspired by the biology of the visual cortex of the human brain. This unified framework is validated by two practical multimodal learning tasks: image captioning, involving visual and natural language signals, and visual-haptic fusion, involving haptic and visual signals. Extensive experiments are conducted under the framework, and competitive results are achieved.

Contributors

Chang Liu:

Bio :

Chang Liu received the B.S. degree from the Department of Computer Science and Technology at Harbin Institute of Technology in July 2013. He received his Doctor of Philosophy in Computer Science and Technology at Tsinghua University, Beijing, July 2018. His research focuses on understanding the contents and meanings of multimodal media inputs, like computer vision inputs in image or video form. E-mail: liuchang8am@ 123456gmail.com

Fuchun Sun:

Bio :

Fuchun Sun received the B.S. and M.S. degrees from Naval Aeronautical Engineering Academy, Yantai, China, and the Ph.D. degree from the Department of Computer Science and Technology, Tsinghua University, where he is currently a full professor. His current research interests include intelligent control, networked control system and management, neural networks, fuzzy systems, nonlinear systems, and robotics. He has published two books and over 100 papers which have appeared in various journals and conference proceedings. E-mail: fcsun@ 123456mail.tsinghua.edu.cn

Bo Zhang:

Bio :

Bo Zhang received the degree from the Department of Automatic Control, Tsinghua University, in 1958. He is currently a professor with the Department of Computer Science and Technology, Tsinghua University. He has published over 150 papers and four monographs. His current research interests include artificial intelligence, pattern recognition, neural networks, and intelligent control. He is the fellow of the Chinese Academy of Sciences.

Journal

Journal ID (publisher-id): Brain Sci. Adv.

Title: Brain Science Advances

Publisher: Tsinghua University Press and SAGE Publishing

ISSN (Electronic): 2096-5958

Publication date (Electronic): 25 November 2018

Volume: 4

Issue: 1

Pages: 61-72

Affiliations

¹Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China

Author notes

Address correspondence to Fuchun Sun, E-mail: fcsun@ 123456mail.tsinghua.edu.cn

Article

Publisher ID: BSA.2018.9050004

DOI: 10.26599/BSA.2018.9050004

Copyright statement: © The authors 2018. This article is published with open access at journals.sagepub.com/home/BSA

License:

This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License ( http://www.creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page ( https://us.sagepub.com/en-us/nam/open-access-at-sage).

Product

Self URI (journal page): https://journals.sagepub.com/home/bsa

Categories

Subject: Research Article