A Paradigm Shift in Interactive Computing: Deriving Multimodal Design Principles from Behavioral and Neurological Foundations

When a person looks at an object while exploring it with their hand, vision and touch both provide information for estimating the properties of the object. Vision frequently dominates the integrated visual-haptic percept, for example when judging size, shape or position, but in some circumstances the percept is clearly affected by haptics. Here we propose that a general principle, which minimizes variance in the final estimate, determines the degree to which vision or haptics dominates. This principle is realized by using maximum-likelihood estimation to combine the inputs. To investigate cue combination quantitatively, we first measured the variances associated with visual and haptic estimation of height. We then used these measurements to construct a maximum-likelihood integrator. This model behaved very similarly to humans in a visual-haptic task. Thus, the nervous system seems to combine visual and haptic information in a fashion that is similar to a maximum-likelihood integrator. Visual dominance occurs when the variance associated with visual estimation is lower than that associated with haptic estimation.

In 1974, Baddeley and Hitch proposed a three-component model of working memory. Over the years, this has been successful in giving an integrated account not only of data from normal adults, but also neuropsychological, developmental and neuroimaging data. There are, however, a number of phenomena that are not readily captured by the original model. These are outlined here and a fourth component to the model, the episodic buffer, is proposed. It comprises a limited capacity system that provides temporary storage of information held in a multimodal code, which is capable of binding information from the subsidiary systems, and from long-term memory, into a unitary episodic representation. Conscious awareness is assumed to be the principal mode of retrieval from the buffer. The revised model differs from the old principally in focussing attention on the processes of integrating information, rather than on the isolation of the subsystems. In doing so, it provides a better basis for tackling the more complex aspects of executive control in working memory.