论文标题
非凸组成优化的随机递归动量法
Stochastic Recursive Momentum Method for Non-Convex Compositional Optimization
论文作者
论文摘要
我们提出了一种新型的随机优化算法，称为随机递归动量（风暴组成）优化，以最大程度地减少两个随机函数的期望的组成，后者是在各种重要的机器学习应用中产生的优化问题。通过在组成梯度更新中引入动量项，Storm-Compositional以指数移动的平均方式运行随机递归方差减少的组成梯度。这导致$ O（\ varepsilon^{ - 3}）$复杂性上限用于暴风雨组合，它与先前宣布的构图优化算法中最著名的复杂性界限相匹配。同时，暴风雨组合是一种单个循环算法，它避免了大型和小批量之间的典型替代调整，以及记录检查点梯度的记录，它持续存在方差减少的随机梯度方法。这允许在数值实验中更简单的参数调整，这证明了暴风雨组合比其他随机组成优化算法的优越性。
We propose a novel stochastic optimization algorithm called STOchastic Recursive Momentum for Compositional (STORM-Compositional) optimization that minimizes the composition of expectations of two stochastic functions, the latter being an optimization problem arising in various important machine learning applications. By introducing the momentum term in the compositional gradient updates, STORM-Compositional operates the stochastic recursive variance-reduced compositional gradients in an exponential-moving average way. This leads to an $O(\varepsilon^{-3})$ complexity upper bound for STORM-Compositional, that matches the best known complexity bounds in previously announced compositional optimization algorithms. At the same time, STORM-Compositional is a single loop algorithm that avoids the typical alternative tuning between large and small batch sizes, as well as recording of checkpoint gradients, that persist in variance-reduced stochastic gradient methods. This allows considerably simpler parameter tuning in numerical experiments, which demonstrates the superiority of STORM-Compositional over other stochastic compositional optimization algorithms.
