论文标题
深网优化中的自适应正则化的随机批次大小
Stochastic batch size for adaptive regularization in deep network optimization
论文作者
论文摘要
我们提出了一种一阶随机优化算法，该算法结合了适用于深度学习框架机器学习问题的自适应正则化。自适应正则化是通过随机过程在每个优化迭代时确定每个模型参数的批处理大小的。随机批量大小由在神经网络体系结构中的局部和全局属性的分布之后通过每个参数的更新概率确定，在梯度构建中，梯度规范的范围可能会在层次内和跨层中变化。我们在经验上使用基于应用于常用基准数据集的常规网络模型的图像分类任务来证明算法的有效性。定量评估表明，我们的算法在概括中优于最新的优化算法，同时对选择批次大小的选择较少敏感，这通常在优化中起着至关重要的作用，从而实现了对定期性选择的更强性。
We propose a first-order stochastic optimization algorithm incorporating adaptive regularization applicable to machine learning problems in deep learning framework. The adaptive regularization is imposed by stochastic process in determining batch size for each model parameter at each optimization iteration. The stochastic batch size is determined by the update probability of each parameter following a distribution of gradient norms in consideration of their local and global properties in the neural network architecture where the range of gradient norms may vary within and across layers. We empirically demonstrate the effectiveness of our algorithm using an image classification task based on conventional network models applied to commonly used benchmark datasets. The quantitative evaluation indicates that our algorithm outperforms the state-of-the-art optimization algorithms in generalization while providing less sensitivity to the selection of batch size which often plays a critical role in optimization, thus achieving more robustness to the selection of regularity.
