Predicting eye movement patterns from fMRI responses to natural scenes

Abstract

Eye tracking has long been used to measure overt spatial attention, and computational models of spatial attention reliably predict eye movements to natural images. However, researchers lack techniques to noninvasively access spatial representations in the human brain that guide eye movements. Here, we use functional magnetic resonance imaging (fMRI) to predict eye movement patterns from reconstructed spatial representations evoked by natural scenes. First, we reconstruct fixation maps to directly predict eye movement patterns from fMRI activity. Next, we use a model-based decoding pipeline that aligns fMRI activity to deep convolutional neural network activity to reconstruct spatial priority maps and predict eye movements in a zero-shot fashion. We predict human eye movement patterns from fMRI responses to natural scenes, provide evidence that visual representations of scenes and objects map onto neural representations that predict eye movements, and find a novel three-way link between brain activity, deep neural network models, and behavior.

Abstract

Human eye movements when viewing scenes can reflect overt spatial attention. Here, O’Connell and Chun predict human eye movement patterns from BOLD responses to natural scenes. Linking brain activity, convolutional neural network (CNN) models, and eye movement behavior, they show that brain activity patterns and CNN models share representations that guide eye movements to scenes.