论文标题
线性宽度神经网络的共轭核和神经切线核的光谱
Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
论文作者
论文摘要
我们研究了与多层前馈神经网络相关的共轭核和神经切线核的特征值分布。在渐近方向上，在重量的随机初始化下，网络宽度在样本大小上线性增加，对于满足近似成对正交性概念的输入样品，我们表明，CK和NTK的特征值分布融合了确定性限制。 CK的限制是通过在隐藏层上迭代Marcenko-Pastur图来描述的。 NTK的极限等效于跨层的CK矩阵的线性组合，并且可以通过扩展此Marcenko-Pastur映射的递归固定点方程来描述。我们证明了这些渐近预测与观察到的合成和CIFAR-10训练数据的光谱的一致性，我们进行了一个小型模拟，以研究这些光谱在训练上的发展。
We study the eigenvalue distributions of the Conjugate Kernel and Neural Tangent Kernel associated to multi-layer feedforward neural networks. In an asymptotic regime where network width is increasing linearly in sample size, under random initialization of the weights, and for input samples satisfying a notion of approximate pairwise orthogonality, we show that the eigenvalue distributions of the CK and NTK converge to deterministic limits. The limit for the CK is described by iterating the Marcenko-Pastur map across the hidden layers. The limit for the NTK is equivalent to that of a linear combination of the CK matrices across layers, and may be described by recursive fixed-point equations that extend this Marcenko-Pastur map. We demonstrate the agreement of these asymptotic predictions with the observed spectra for both synthetic and CIFAR-10 training data, and we perform a small simulation to investigate the evolutions of these spectra over training.
