16
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization

      Preprint
      , , ,

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          In this paper, we consider the minimization of a nonsmooth nonconvex objective function \(f(x)\) over a closed convex subset \(\mathcal{X}\) of \(\mathbb{R}^n\), with additional nonsmooth nonconvex constraints \(c(x) = 0\). We develop a unified framework for developing Lagrangian-based methods, which takes a single-step update to the primal variables by some subgradient methods in each iteration. These subgradient methods are ``embedded'' into our framework, in the sense that they are incorporated as black-box updates to the primal variables. We prove that our proposed framework inherits the global convergence guarantees from these embedded subgradient methods under mild conditions. In addition, we show that our framework can be extended to solve constrained optimization problems with expectation constraints. Based on the proposed framework, we show that a wide range of existing stochastic subgradient methods, including the proximal SGD, proximal momentum SGD, and proximal ADAM, can be embedded into Lagrangian-based methods. Preliminary numerical experiments on deep learning tasks illustrate that our proposed framework yields efficient variants of Lagrangian-based methods with convergence guarantees for nonconvex nonsmooth constrained optimization problems.

          Related collections

          Author and article information

          Journal
          14 April 2024
          Article
          2404.09438
          b6acec91-e175-40ef-9f87-db487dac6231

          http://arxiv.org/licenses/nonexclusive-distrib/1.0/

          History
          Custom metadata
          30 pages, 4 figures
          math.OC cs.LG stat.ML

          Numerical methods,Machine learning,Artificial intelligence
          Numerical methods, Machine learning, Artificial intelligence

          Comments

          Comment on this article