20
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: not found
      • Article: not found

      The successor representation in human reinforcement learning

      Read this article at

      ScienceOpenPublisherPMC
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Theories of reward learning in neuroscience have focused on two families of algorithms, thought to capture deliberative vs. habitual choice. “Model-based” algorithms compute the value of candidate actions from scratch, whereas “model-free” algorithms make choice more efficient but less flexible by storing pre-computed action values. We examine an intermediate algorithmic family, the successor representation (SR), which balances flexibility and efficiency by storing partially computed action values: predictions about future events. These pre-computation strategies differ in how they update their choices following changes in a task. SR’s reliance on stored predictions about future states predicts a unique signature of insensitivity to changes in the task’s sequence of events, but flexible adjustment following changes to rewards. We provide evidence for such differential sensitivity in two behavioral studies with humans. These results suggest that the SR is a computational substrate for semi-flexible choice in humans, introducing a subtler, more cognitive notion of habit.

          Related collections

          Most cited references41

          • Record: found
          • Abstract: found
          • Article: not found

          The role of medial prefrontal cortex in memory and decision making.

          Some have claimed that the medial prefrontal cortex (mPFC) mediates decision making. Others suggest mPFC is selectively involved in the retrieval of remote long-term memory. Yet others suggests mPFC supports memory and consolidation on time scales ranging from seconds to days. How can all these roles be reconciled? We propose that the function of the mPFC is to learn associations between context, locations, events, and corresponding adaptive responses, particularly emotional responses. Thus, the ubiquitous involvement of mPFC in both memory and decision making may be due to the fact that almost all such tasks entail the ability to recall the best action or emotional response to specific events in a particular place and time. An interaction between multiple memory systems may explain the changing importance of mPFC to different types of memories over time. In particular, mPFC likely relies on the hippocampus to support rapid learning and memory consolidation. Copyright © 2012 Elsevier Inc. All rights reserved.
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            Interplay of hippocampus and prefrontal cortex in memory.

            Recent studies on the hippocampus and the prefrontal cortex have considerably advanced our understanding of the distinct roles of these brain areas in the encoding and retrieval of memories, and of how they interact in the prolonged process by which new memories are consolidated into our permanent storehouse of knowledge. These studies have led to a new model of how the hippocampus forms and replays memories and how the prefrontal cortex engages representations of the meaningful contexts in which related memories occur, as well as how these areas interact during memory retrieval. Furthermore, they have provided new insights into how interactions between the hippocampus and prefrontal cortex support the assimilation of new memories into pre-existing networks of knowledge, called schemas, and how schemas are modified in this process as the foundation of memory consolidation. Copyright © 2013 Elsevier Ltd. All rights reserved.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Orbitofrontal cortex as a cognitive map of task space.

              Orbitofrontal cortex (OFC) has long been known to play an important role in decision making. However, the exact nature of that role has remained elusive. Here, we propose a unifying theory of OFC function. We hypothesize that OFC provides an abstraction of currently available information in the form of a labeling of the current task state, which is used for reinforcement learning (RL) elsewhere in the brain. This function is especially critical when task states include unobservable information, for instance, from working memory. We use this framework to explain classic findings in reversal learning, delayed alternation, extinction, and devaluation as well as more recent findings showing the effect of OFC lesions on the firing of dopaminergic neurons in ventral tegmental area (VTA) in rodents performing an RL task. In addition, we generate a number of testable experimental predictions that can distinguish our theory from other accounts of OFC function. Copyright © 2014 Elsevier Inc. All rights reserved.
                Bookmark

                Author and article information

                Journal
                Nature Human Behaviour
                Nat Hum Behav
                Springer Nature
                2397-3374
                September 2017
                August 28 2017
                : 1
                : 9
                : 680-692
                Article
                10.1038/s41562-017-0180-8
                6941356
                31024137
                a6d80445-930c-4aad-92c8-fc48034be2d4
                © 2017

                http://www.springer.com/tdm

                History

                Comments

                Comment on this article