From Seeing to Feeling Sound: A Multimodal Interface for Trans-Domain Mapping of Sound

Despite its aural focus, sound can encompass both visual and somatic stimulation. This project aims to develop a system to enhance the multimodality of sound using additional visual and haptic feedback techniques. This paper presents a prototype to demonstrate the concept, in the context of technology-enhanced learning for music. With a brief summary of related background literature survey, the paper describes design and development aspects of the system. It discusses software visual interface designs as well as hardware interfaces for vibrotactile feedback to augment an interactive system. The paper concludes with validations and discusses several plausible future directions.