2
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Offline Reinforcement Learning at Multiple Frequencies

      Preprint
      , , ,

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Leveraging many sources of offline robot data requires grappling with the heterogeneity of such data. In this paper, we focus on one particular aspect of heterogeneity: learning from offline data collected at different control frequencies. Across labs, the discretization of controllers, sampling rates of sensors, and demands of a task of interest may differ, giving rise to a mixture of frequencies in an aggregated dataset. We study how well offline reinforcement learning (RL) algorithms can accommodate data with a mixture of frequencies during training. We observe that the \(Q\)-value propagates at different rates for different discretizations, leading to a number of learning challenges for off-the-shelf offline RL. We present a simple yet effective solution that enforces consistency in the rate of \(Q\)-value updates to stabilize learning. By scaling the value of \(N\) in \(N\)-step returns with the discretization size, we effectively balance \(Q\)-value propagation, leading to more stable convergence. On three simulated robotic control problems, we empirically find that this simple approach outperforms na\"ive mixing by 50% on average.

          Related collections

          Author and article information

          Journal
          26 July 2022
          Article
          2207.13082
          31326797-75cb-465a-9113-53c56f7eaa96

          http://creativecommons.org/licenses/by/4.0/

          History
          Custom metadata
          Project website: https://sites.google.com/stanford.edu/adaptive-nstep-returns/
          cs.LG cs.AI cs.RO

          Robotics,Artificial intelligence
          Robotics, Artificial intelligence

          Comments

          Comment on this article