The supramodal brain: implications for auditory perception

Synchronous presentation of stimuli to the auditory and visual systems can modify the formation of a percept in either modality. For example, perception of auditory speech is improved when the speaker's facial articulatory movements are visible. Neural convergence onto multisensory sites exhibiting supra-additivity has been proposed as the principal mechanism for integration. Recent findings, however, have suggested that putative sensory-specific cortices are responsive to inputs presented through a different modality. Consequently, when and where audiovisual representations emerge remain unsettled. In combined psychophysical and electroencephalography experiments we show that visual speech speeds up the cortical processing of auditory signals early (within 100 ms of signal onset). The auditory-visual interaction is reflected as an articulator-specific temporal facilitation (as well as a nonspecific amplitude reduction). The latency facilitation systematically depends on the degree to which the visual signal predicts possible auditory targets. The observed auditory-visual data support the view that there exist abstract internal representations that constrain the analysis of subsequent speech inputs. This is evidence for the existence of an "analysis-by-synthesis" mechanism in auditory-visual speech perception.

When the brain is deprived of input from one sensory modality, it often compensates with supranormal performance in one or more of the intact sensory systems. In the absence of acoustic input, it has been proposed that cross-modal reorganization of deaf auditory cortex may provide the neural substrate mediating compensatory visual function. We tested this hypothesis using a battery of visual psychophysical tasks and found that congenitally deaf cats, compared with hearing cats, have superior localization in the peripheral field and lower visual movement detection thresholds. In the deaf cats, reversible deactivation of posterior auditory cortex selectively eliminated superior visual localization abilities, whereas deactivation of the dorsal auditory cortex eliminated superior visual motion detection. Our results indicate that enhanced visual performance in the deaf is caused by cross-modal reorganization of deaf auditory cortex and it is possible to localize individual visual functions in discrete portions of reorganized auditory cortex.