- Record: found
- Abstract: found
- Article: found
Brain-inspired multimodal learning based on neural networks
Read this article at
Abstract
Modern computational models have leveraged biological advances in human brain research. This study addresses the problem of multimodal learning with the help of brain-inspired models. Specifically, a unified multimodal learning architecture is proposed based on deep neural networks, which are inspired by the biology of the visual cortex of the human brain. This unified framework is validated by two practical multimodal learning tasks: image captioning, involving visual and natural language signals, and visual-haptic fusion, involving haptic and visual signals. Extensive experiments are conducted under the framework, and competitive results are achieved.
Author and article information
Contributors
Chang Liu received the B.S. degree from the Department of Computer Science and Technology at Harbin Institute of Technology in July 2013. He received his Doctor of Philosophy in Computer Science and Technology at Tsinghua University, Beijing, July 2018. His research focuses on understanding the contents and meanings of multimodal media inputs, like computer vision inputs in image or video form. E-mail: liuchang8am@ 123456gmail.com
Fuchun Sun received the B.S. and M.S. degrees from Naval Aeronautical Engineering Academy, Yantai, China, and the Ph.D. degree from the Department of Computer Science and Technology, Tsinghua University, where he is currently a full professor. His current research interests include intelligent control, networked control system and management, neural networks, fuzzy systems, nonlinear systems, and robotics. He has published two books and over 100 papers which have appeared in various journals and conference proceedings. E-mail: fcsun@ 123456mail.tsinghua.edu.cn
Bo Zhang received the degree from the Department of Automatic Control, Tsinghua University, in 1958. He is currently a professor with the Department of Computer Science and Technology, Tsinghua University. He has published over 150 papers and four monographs. His current research interests include artificial intelligence, pattern recognition, neural networks, and intelligent control. He is the fellow of the Chinese Academy of Sciences.
Journal
Affiliations
Author notes
Article
BSA.2018.9050004This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License ( http://www.creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page ( https://us.sagepub.com/en-us/nam/open-access-at-sage).