论文标题
MLMODELSCOPE：用于模型评估和基准测试的分布式平台
MLModelScope: A Distributed Platform for Model Evaluation and Benchmarking at Scale
论文作者
论文摘要
机器学习（ML）和深度学习（DL）创新正在以如此快速的速度引入，以至于研究人员很难分析和研究它们。评估创新的复杂程序，以及缺乏指定和配置ML/DL评估的标准和有效方法，是社区的主要“痛苦点”。本文提出了MLModelScope，一种开源，框架/硬件不可知论，可扩展和可自定义的设计，可实现可重复，公平且可扩展的模型评估和基准测试。我们实现了分布式设计，并支持所有主要框架和硬件，并配备Web，命令行和库接口。为了证明MLModelScope的功能，我们执行并行评估，并显示模型评估管道的微妙变化如何影响准确性和HW/SW堆栈选择会影响性能。
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major "pain point" for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope's capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.
