论文标题
线性卷积层的渐近单数值分布
Asymptotic Singular Value Distribution of Linear Convolutional Layers
论文作者
论文摘要
在卷积神经网络中，具有线性卷积的多通道二维卷积层的线性变换是具有双重toeplitz块的块矩阵。尽管“围绕”操作可以将线性卷积转换为一个圆形的卷积，通过该卷积，可以通过具有双重循环块的块矩阵的计算复杂性降低计算复杂性来近似近似值，但不能保证这种近似值的准确性。在本文中，我们建议通过其渐近光谱表示（光谱密度矩阵）来检查这种线性转化矩阵，通过该矩阵 - 我们开发了一种简单的奇异值近似方法，在圆形近似方面具有提高精度，以及用于光谱规范的上限，并具有降低的计算复杂性。与圆形近似相比，我们通过微妙的奇异值分布进行了适度的改进。我们还证明，光谱标准上限是有效的光谱正规化器，可改善重置中的泛化性能。
In convolutional neural networks, the linear transformation of multi-channel two-dimensional convolutional layers with linear convolution is a block matrix with doubly Toeplitz blocks. Although a "wrapping around" operation can transform linear convolution to a circular one, by which the singular values can be approximated with reduced computational complexity by those of a block matrix with doubly circulant blocks, the accuracy of such an approximation is not guaranteed. In this paper, we propose to inspect such a linear transformation matrix through its asymptotic spectral representation - the spectral density matrix - by which we develop a simple singular value approximation method with improved accuracy over the circular approximation, as well as upper bounds for spectral norm with reduced computational complexity. Compared with the circular approximation, we obtain moderate improvement with a subtle adjustment of the singular value distribution. We also demonstrate that the spectral norm upper bounds are effective spectral regularizers for improving generalization performance in ResNets.
