Auto-PyTorch

Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

authored by

Lucas Zimmer, Marius Lindauer, Frank Hutter

Abstract

While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data. To thoroughly study our assumptions on how to design such an AutoDL system, we additionally introduce a new benchmark on learning curves for DNNs, dubbed LCBench, and run extensive ablation studies of the full Auto-PyTorch on typical AutoML benchmarks, eventually showing that Auto-PyTorch performs better than several state-of-the-art competitors on average.

Organisation(s)

Machine Learning Section
Institute of Information Processing

External Organisation(s)

University of Freiburg
Bosch Center for Artificial Intelligence (BCAI)

Type

Article

Journal

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volume

43

Pages

3079-3090

No. of pages

12

ISSN

0162-8828

Publication date

01.09.2021

Publication status

Published

Peer reviewed

Yes

ASJC Scopus subject areas

Software, Artificial Intelligence, Applied Mathematics, Computer Vision and Pattern Recognition, Computational Theory and Mathematics

Electronic version(s)

https://arxiv.org/pdf/2006.13799 (Access: Open)
https://doi.org/10.1109/TPAMI.2021.3067763 (Access: Closed)