9
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Asynchronous Distributed Learning with Sparse Communications and Identification

      Preprint
      , , ,

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          In this paper, we present an asynchronous optimization algorithm for distributed learning, that efficiently reduces the communications between a master and working machines by randomly sparsifying the local updates. This sparsification allows to lift the communication bottleneck often present in distributed learning setups where computations are performed by workers on local data while a master machine coordinates their updates to optimize a global loss. We prove that despite its sparse asynchronous communications, our algorithm allows for a fixed stepsize and benefits from a linear convergence rate in the strongly convex case. Moreover, for \(\ell_1\)-regularized problems, this algorithm identifies near-optimal sparsity patterns, so that all communications eventually become sparse. We furthermore leverage on this identification to improve our sparsification technique. We illustrate on real and synthetic data that this algorithm converges faster in terms of data exchanges.

          Related collections

          Most cited references5

          • Record: found
          • Abstract: not found
          • Article: not found

          Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems

            Bookmark
            • Record: found
            • Abstract: not found
            • Article: not found

            Convex Optimization: Algorithms and Complexity

              Bookmark
              • Record: found
              • Abstract: not found
              • Article: not found

              Identifiable Surfaces in Constrained Optimization

                Bookmark

                Author and article information

                Journal
                10 December 2018
                Article
                1812.03871
                22ef6465-1216-4477-875c-020e77e592ab

                http://arxiv.org/licenses/nonexclusive-distrib/1.0/

                History
                Custom metadata
                math.OC cs.DC

                Numerical methods,Networking & Internet architecture
                Numerical methods, Networking & Internet architecture

                Comments

                Comment on this article