论文标题
转移学习中有什么转移？
What is being transferred in transfer learning?
论文作者
论文摘要
机器的一种理想能力是能够将其对一个域的知识转移到另一个域（通常）稀缺的地方。尽管在各种深度学习应用中充分适应了转移学习，但我们仍然不理解什么是成功的转移以及网络的哪一部分负责。在本文中，我们提供了新的工具和分析来解决这些基本问题。通过一系列有关转移到块状图像的分析，我们将功能重用的效果与学习低级数据统计数据区分开，并表明转移学习的某些好处来自后者。我们表明，当从预训练的权重进行训练时，该模型在损失景观中停留在相同的盆地，并且这种模型的不同实例在特征空间中相似，并且在参数空间中关闭。
One desired capability for machines is the ability to transfer their knowledge of one domain to another where data is (usually) scarce. Despite ample adaptation of transfer learning in various deep learning applications, we yet do not understand what enables a successful transfer and which part of the network is responsible for that. In this paper, we provide new tools and analyses to address these fundamental questions. Through a series of analyses on transferring to block-shuffled images, we separate the effect of feature reuse from learning low-level statistics of data and show that some benefit of transfer learning comes from the latter. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.
