Auditory Icons: Using Sound in Computer Interfaces

The mechanical events of bouncing and breaking are acoustically specified by single versus multiple damped quasi-periodic pulse patterns, with an initial noise burst in the case of breaking. Subjects show high accuracy in categorizing natural tokens of bouncing and breaking glass as well as tokens constructed by adjusting only the temporal patterns of components, leaving their spectral properties constant. Differences in average spectral frequency are, therefore, not necessary for perceiving this contrast, though differences in spectral consistency over successive pulses may be important. Initial noise corresponding to glass rupture appears unnecessary to categorize breaking and bouncing. The data indicate that higher order temporal properties of the acoustic signal provide information for the auditory perception of these events.

Three experiments were carried out to investigate the evaluation and integration of visual and auditory information in speech perception. In the first two experiments, subjects identified /ba/ or /da/ speech events consisting of high-quality synthetic syllables ranging from /ba/ to /da/ combined with a videotaped /ba/ or /da/ or neutral articulation. Although subjects were specifically instructed to report what they heard, visual articulation made a large contribution to identification. The tests of quantitative models provide evidence for the integration of continuous and independent, as opposed to discrete or nonindependent, sources of information. The reaction times for identification were primarily correlated with the perceived ambiguity of the speech event. In a third experiment, the speech events were identified with an unconstrained set of response alternatives. In addition to /ba/ and /da/ responses, the /bda/ and /tha/ responses were well described by a combination of continuous and independent features. This body of results provides strong evidence for a fuzzy logical model of perceptual recognition.

A system for the creation of computer-generated sound patterns of two-dimensional line graphs is described. The objectives of the system are to provide the blind with a means of understanding line graphs in the holistic manner used by those with sight. A continuously varying pitch is used to represent motion in the x direction. To test the feasibility of using sound to represent graphs, a prototype system was developed and human factors experimenters were performed. Fourteen subjects were used to compare the tactile-graph methods normally used by the blind to these new sound graphs. It was discovered that mathematical concepts such as symmetry, monotonicity, and the slopes of lines could be determined quickly using sound. Even better performance may be expected with additional training. The flexibility, speed, cost-effectiveness, and greater measure of independence provided the blind or sight-impaired using these methods was demonstrated.