论文标题
更加关注迭代修剪的快照：通过合奏蒸馏改善模型压缩
Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
论文作者
论文摘要
网络修剪是降低深神经网络的沉重推理成本的最主要方法之一。现有的方法通常在迭代修剪网络上达到高压比，而不会产生绩效的重大损失。但是，我们认为，用于重新修剪网络的传统方法（即使用小的，固定的学习率）是不足的，因为它们完全忽略了迭代修剪的快照中的好处。在这项工作中，我们表明，可以从迭代修剪的快照中构建强大的合奏，从而实现竞争性能并在网络结构上变化。此外，我们提出了简单，一般和有效的管道，该管道在修剪过程中会产生强大的网络集合，并重新启动大量学习率，并使用这些团体利用知识蒸馏来提高紧凑型模型的预测能力。在标准图像分类基准（例如CIFAR和TININ-IMAGENET）中，我们通过将简单的L1-Norm过滤器整合到我们的管道中来提高结构化修剪的最新修剪比。具体而言，我们减少了总参数的75-80％和65-70％的MAC，其中许多重置体系结构的变体具有比原始网络相当或更好的性能。与本文的代码副本可在https://github.com/lehduong/kesi上公开获得。
Network pruning is one of the most dominant methods for reducing the heavy inference cost of deep neural networks. Existing methods often iteratively prune networks to attain high compression ratio without incurring significant loss in performance. However, we argue that conventional methods for retraining pruned networks (i.e., using small, fixed learning rate) are inadequate as they completely ignore the benefits from snapshots of iterative pruning. In this work, we show that strong ensembles can be constructed from snapshots of iterative pruning, which achieve competitive performance and vary in network structure. Furthermore, we present simple, general and effective pipeline that generates strong ensembles of networks during pruning with large learning rate restarting, and utilizes knowledge distillation with those ensembles to improve the predictive power of compact models. In standard image classification benchmarks such as CIFAR and Tiny-Imagenet, we advance state-of-the-art pruning ratio of structured pruning by integrating simple l1-norm filters pruning into our pipeline. Specifically, we reduce 75-80% of total parameters and 65-70% MACs of numerous variants of ResNet architectures while having comparable or better performance than that of original networks. Code associate with this paper is made publicly available at https://github.com/lehduong/kesi.
