论文标题
关于迭代性神经网络修剪，重新定性和面具的相似性
On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
论文作者
论文摘要
我们研究了在修剪过程中的深度学习模型中最近记录的基本现象受到修剪程序的变化的影响。具体而言，我们分析了通过一系列通用的迭代修剪技术发现的修剪模型的连通性结构和学习动力学的差异，以解决可训练的，高的，高比较子网络的唯一性问题及其对所选修剪方法的依赖。在卷积层中，我们记录了由基于大小的非结构化修剪引起的结构的出现，并与重量恢复相似，类似于结构化修剪的效果。我们还显示了经验证据，表明可以通过合适的修剪技术自动实现体重稳定性。
We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to address questions of uniqueness of trainable, high-sparsity sub-networks, and their dependence on the chosen pruning method. In convolutional layers, we document the emergence of structure induced by magnitude-based unstructured pruning in conjunction with weight rewinding that resembles the effects of structured pruning. We also show empirical evidence that weight stability can be automatically achieved through apposite pruning techniques.
