Publications Details

Publication Details

Winning Solutions and Post-Challenge Analyses of the ChaLearn AutoDL Challenge 2019

authored by
Zhengying Liu, Adrien Pavao, Zhen Xu, Sergio Escalera, Fabio Ferreira, Isabelle Guyon, Sirui Hong, Frank Hutter, Rongrong Ji, Julio C. S. Jacques Junior, Ge Li, Marius Lindauer, Zhipeng Luo, Meysam Madadi, Thomas Nierhoff, Kangning Niu, Chunguang Pan, Danny Stoll, Sebastien Treguer, Jin Wang, Peng Wang, Chenglin Wu, Youcheng Xiong, Arber Zela, Yang Zhang

This paper reports the results and post-challenge analyses of ChaLearn's AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a 'meta-learner', 'data ingestor', 'model selector', 'model/learner', and 'evaluator'. This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark (, the open-sourced code of the winners, and a free 'AutoDL self-service'.

Machine Learning Section
Faculty of Electrical Engineering and Computer Science
External Organisation(s)
Université Paris-Saclay
Universitat de Barcelona
University of Freiburg
Xiamen University
Universitat Oberta de Catalunya
Autonomous University of Barcelona (UAB)
Deep Wisdom Inc.
DeepBlue Technology
La Paillasse
Lenovo Research
IEEE Transactions on Pattern Analysis and Machine Intelligence
No. of pages
Publication date
Publication status
Peer reviewed
ASJC Scopus subject areas
Software, Computer Vision and Pattern Recognition, Computational Theory and Mathematics, Artificial Intelligence, Applied Mathematics
Electronic version(s) (Access: Open) (Access: Open) (Access: Closed)