18
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Unifying the Stochastic Spectral Descent for Restricted Boltzmann Machines with Bernoulli or Gaussian Inputs

      Preprint

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Stochastic gradient descent based algorithms are typically used as the general optimization tools for most deep learning models. A Restricted Boltzmann Machine (RBM) is a probabilistic generative model that can be stacked to construct deep architectures. For RBM with Bernoulli inputs, non-Euclidean algorithm such as stochastic spectral descent (SSD) has been specifically designed to speed up the convergence with improved use of the gradient estimation by sampling methods. However, the existing algorithm and corresponding theoretical justification depend on the assumption that the possible configurations of inputs are finite, like binary variables. The purpose of this paper is to generalize SSD for Gaussian RBM being capable of mod- eling continuous data, regardless of the previous assumption. We propose the gradient descent methods in non-Euclidean space of parameters, via de- riving the upper bounds of logarithmic partition function for RBMs based on Schatten-infinity norm. We empirically show that the advantage and improvement of SSD over stochastic gradient descent (SGD).

          Related collections

          Most cited references5

          • Record: found
          • Abstract: not found
          • Article: not found

          Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions

            Bookmark
            • Record: found
            • Abstract: not found
            • Conference Proceedings: not found

            Training restricted Boltzmann machines using approximations to the likelihood gradient

              Bookmark
              • Record: found
              • Abstract: not found
              • Conference Proceedings: not found

              On the quantitative analysis of deep belief networks

                Bookmark

                Author and article information

                Journal
                2017-03-28
                Article
                1703.09766
                a648c506-9f80-499b-8253-eff4605ec239

                http://arxiv.org/licenses/nonexclusive-distrib/1.0/

                History
                Custom metadata
                stat.ML

                Machine learning
                Machine learning

                Comments

                Comment on this article