论文标题
GPU加速详尽搜索黑盒优化算法的最佳合奏
GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box Optimization Algorithms
论文作者
论文摘要
黑盒优化对于调整复杂的机器学习算法至关重要，该算法比理解更容易实验。在本文中，我们表明，简单的黑盒优化算法的合奏可以胜过其中任何一个。但是，搜索这种最佳合奏需要大量实验。我们提出了一个多GPU优化的框架，以通过并行运行许多实验来加速蛮力搜索黑盒优化算法的最佳合奏。轻巧的优化是由CPU执行的，而昂贵的模型培训和评估被分配给GPU。我们通过培训270万款型号并运行541,440个优化来评估15个优化器。在DGX-1上，搜索时间从两个20核CPU的10天减少到8-gpus的24小时。随着GPU加速详尽搜索发现的最佳合奏，我们赢得了2020年黑框优化挑战的第二名。
Black-box optimization is essential for tuning complex machine learning algorithms which are easier to experiment with than to understand. In this paper, we show that a simple ensemble of black-box optimization algorithms can outperform any single one of them. However, searching for such an optimal ensemble requires a large number of experiments. We propose a Multi-GPU-optimized framework to accelerate a brute force search for the optimal ensemble of black-box optimization algorithms by running many experiments in parallel. The lightweight optimizations are performed by CPU while expensive model training and evaluations are assigned to GPUs. We evaluate 15 optimizers by training 2.7 million models and running 541,440 optimizations. On a DGX-1, the search time is reduced from more than 10 days on two 20-core CPUs to less than 24 hours on 8-GPUs. With the optimal ensemble found by GPU-accelerated exhaustive search, we won the 2nd place of NeurIPS 2020 black-box optimization challenge.
