Redundant representations are required to disambiguate simultaneously presented complex stimuli

Abstract

A pedestrian crossing a street during rush hour often looks and listens for potential danger. When they hear several different horns, they localize the cars that are honking and decide whether or not they need to modify their motor plan. How does the pedestrian use this auditory information to pick out the corresponding cars in visual space? The integration of distributed representations like these is called the assignment problem, and it must be solved to integrate distinct representations across but also within sensory modalities. Here, we identify and analyze a solution to the assignment problem: the representation of one or more common stimulus features in pairs of relevant brain regions—for example, estimates of the spatial position of cars are represented in both the visual and auditory systems. We characterize how the reliability of this solution depends on different features of the stimulus set (e.g., the size of the set and the complexity of the stimuli) and the details of the split representations (e.g., the precision of each stimulus representation and the amount of overlapping information). Next, we implement this solution in a biologically plausible receptive field code and show how constraints on the number of neurons and spikes used by the code force the brain to navigate a tradeoff between local and catastrophic errors. We show that, when many spikes and neurons are available, representing stimuli from a single sensory modality can be done more reliably across multiple brain regions, despite the risk of assignment errors. Finally, we show that a feedforward neural network can learn the optimal solution to the assignment problem, even when it receives inputs in two distinct representational formats. We also discuss relevant results on assignment errors from the human working memory literature and show that several key predictions of our theory already have support.

Author summary

Human and animal behavior relies on the integration of distinct sources of information about the same objects in the world—for instance, social behavior requires the correct integration of people with their words, even when multiple people are talking over each other. We formalize this integration process and show that it relies on at least partial redundancy between these different sources of information. In the case of integrating vocalizations with their source, this redundancy could be provided by the distinct representations of spatial position in the visual and auditory systems. Then, we show that the necessity of this integration process produces a trade-off between the representation of redundant information (for reliable integration) and the representation of non-redundant information (which is to be integrated), with implications for modular organization in the brain. Finally, we show that a simple feedforward neural network can integrate as reliably as predicted by our theory—as well as make predictions from our theory that can be tested in neural data. Overall, this work provides insight into how the brain makes sense of its distributed and sometimes distinct representations of the world.