论文标题
测量神经网络的算法效率
Measuring the Algorithmic Efficiency of Neural Networks
论文作者
论文摘要
三个因素推动了AI的进步：算法创新，数据和可用于培训的计算量。传统上，算法进度比计算和数据更难量化。在这项工作中，我们认为算法的进步具有一个直接衡量和有趣的方面：随着时间的推移，在达到过去功能所需的计算中减少。我们表明，在2012年至2019年之间，训练分类器为Alexnet级的性能训练分类器所需的浮点操作数量减少了44倍。这对应于7年中每16个月加倍的算法效率。相比之下，摩尔定律只会提高11倍的成本。我们观察到，硬件和算法效率会增长，并且可以在有意义的视野上达到类似的规模，这表明AI进度的良好模型应整合两者的措施。
Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to quantify than compute and data. In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic efficiency doubling every 16 months over a period of 7 years. By contrast, Moore's Law would only have yielded an 11x cost improvement. We observe that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should integrate measures from both.
