论文标题
自我任务模块：用于学习正交过滤器的网络体系结构插件
Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
论文作者
论文摘要
在本文中，我们研究了独奏或协作中正交正规化（OR）的经验影响。最近的工作或在准确性方面显示了一些有希望的结果。然而，在我们的消融研究中，与基于重量衰减，辍学和批准化的常规培训相比，我们没有观察到现有或技术的显着改善。为了确定来自或从角度估计的局部敏感哈希（LSH）启发的实际增益，我们建议将网络中滤波角的隐式自我调节引入或以在滤波器之间同时实现（接近）正交性，而无需使用任何其他明确的正规化。我们的正则化可以作为建筑插件实现，并与任意网络集成。我们揭示或有助于稳定训练过程，并导致更快的收敛和更好的概括。
In this paper, we investigate the empirical impact of orthogonality regularization (OR) in deep learning, either solo or collaboratively. Recent works on OR showed some promising results on the accuracy. In our ablation study, however, we do not observe such significant improvement from existing OR techniques compared with the conventional training based on weight decay, dropout, and batch normalization. To identify the real gain from OR, inspired by the locality sensitive hashing (LSH) in angle estimation, we propose to introduce an implicit self-regularization into OR to push the mean and variance of filter angles in a network towards 90 and 0 simultaneously to achieve (near) orthogonality among the filters, without using any other explicit regularization. Our regularization can be implemented as an architectural plug-in and integrated with an arbitrary network. We reveal that OR helps stabilize the training process and leads to faster convergence and better generalization.
