Winning Solutions and Post-Challenge Analyses of the ChaLearn AutoDL Challenge 2019

authored by: Zhengying Liu, Adrien Pavao, Zhen Xu, Sergio Escalera, Fabio Ferreira, Isabelle Guyon, Sirui Hong, Frank Hutter, Rongrong Ji, Julio C. S. Jacques Junior, Ge Li, Marius Lindauer, Zhipeng Luo, Meysam Madadi, Thomas Nierhoff, Kangning Niu, Chunguang Pan, Danny Stoll, Sebastien Treguer, Jin Wang, Peng Wang, Chenglin Wu, Youcheng Xiong, Arber Zela, Yang Zhang
Abstract: This paper reports the results and post-challenge analyses of ChaLearn's AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a 'meta-learner', 'data ingestor', 'model selector', 'model/learner', and 'evaluator'. This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark (http://autodl.chalearn.org), the open-sourced code of the winners, and a free 'AutoDL self-service'.
Organisation(s): Machine Learning Section
Faculty of Electrical Engineering and Computer Science
External Organisation(s): Université Paris-Saclay
Universitat de Barcelona
University of Freiburg
Xiamen University
Universitat Oberta de Catalunya
Autonomous University of Barcelona (UAB)
4Paradigm
Deep Wisdom Inc.
DeepBlue Technology
La Paillasse
Lenovo Research
Type: Article
Journal: IEEE Transactions on Pattern Analysis and Machine Intelligence
Volume: 43
Pages: 3108-3125
No. of pages: 18
ISSN: 0162-8828
Publication date: 01.09.2021
Publication status: Published
Peer reviewed: Yes
ASJC Scopus subject areas: Software, Computer Vision and Pattern Recognition, Computational Theory and Mathematics, Artificial Intelligence, Applied Mathematics
Electronic version(s): https://hal.archives-ouvertes.fr/hal-02957135v1/file/Post_challenge_analysis_of_AutoDL_challenges_2019%20%281%29.pdf (Access: Open)
https://doi.org/10.48550/arXiv.2201.03801 (Access: Open)
https://doi.org/10.1109/TPAMI.2021.3075372 (Access: Closed)

BibTeX

@article{82377f5e282a4fba9ce4119f66404a35,
title = "Winning Solutions and Post-Challenge Analyses of the ChaLearn AutoDL Challenge 2019",
abstract = "This paper reports the results and post-challenge analyses of ChaLearn's AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a 'meta-learner', 'data ingestor', 'model selector', 'model/learner', and 'evaluator'. This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark (http://autodl.chalearn.org), the open-sourced code of the winners, and a free 'AutoDL self-service'.",
keywords = "AutoML, deep learning, hyperparameter optimization, meta-learning, model selection, neural architecture search",
author = "Zhengying Liu and Adrien Pavao and Zhen Xu and Sergio Escalera and Fabio Ferreira and Isabelle Guyon and Sirui Hong and Frank Hutter and Rongrong Ji and Junior, {Julio C. S. Jacques} and Ge Li and Marius Lindauer and Zhipeng Luo and Meysam Madadi and Thomas Nierhoff and Kangning Niu and Chunguang Pan and Danny Stoll and Sebastien Treguer and Jin Wang and Peng Wang and Chenglin Wu and Youcheng Xiong and Arber Zela and Yang Zhang",
note = "Funding Information: The authors would like to thank NVIDIA Corporation for the donation of the GPU used for this research. This work was supported by in part by the Google Research (Zu€rich), in part by the 4Paradigm, in part by the Amazon, in part by the Microsoft, in part by the ICREA through the ICREA Academia Programme. The work of team automl_freiburg was supported in part by the European Research Council (ERC) through the European Union{\textquoteright}s Horizon 2020 Research and Innovation Programme under Grant 716721, in part by the Robert Bosch GmbH, in part by the institutions of the co-authors, and in part by the Spanish project PID2019-105093GB-I00. The authors would also like to thank Olivier Bousquet and Andr{\'e} Elisseeff at Google for their help with the design of the challenge and the countless hours that Andr{\'e} spent engineering the data format. The special version of the CodaLab platform we used was implemented by Tyler Thomas, with the help of Eric Carmichael, CK Collab, LLC, USA. Many people contributed time to help formatting datasets, prepare baseline results, and facilitate the logistics. The authors would also like to thank Stephane Ayache (AMU, France), Hubert Jacob Banville (INRIA, France), Mahsa Behzadi (Google, Switzerland), Kristin Bennett (RPI, New York, USA), Hugo Jair Escalante (IANOE, Mexico and ChaLearn, USA), Gavin Cawley (U. East Anglia, UK), Baiyu Chen (UC Berkeley, USA), Albert Clapes i Sintes (U. Barcelona, Spain), Bram van Ginneken (Radboud U. Nijmegen, The Netherlands), Alexandre Gramfort (U. Paris-Saclay; INRIA, France), Yi-Qi Hu (4paradigm, China), Tatiana Merku-lova (Google, Switzerland), Shangeth Rajaa (BITS Pilani, India), Herilalaina Rakotoarison (U. Paris-Saclay, INRIA, France), Lukasz Romaszko (The University of Edinburgh, UK), Mehreen Saeed (FAST Nat. U. Lahore, Pakistan), Marc Schoenauer (U. Paris-Saclay, INRIA, France), Michele Sebag (U. Paris-Saclay; CNRS, France), Danny Silver (Acadia University, Canada), Lisheng Sun (U. Paris-Saclay; UPSud, France), Wei-Wei Tu (4paradigm, China), Fengfu Li (4paradigm, China), Lichuan Xiang (4paradigm, China), Jun Wan (Chinese ",
year = "2021",
month = sep,
day = "1",
doi = "10.48550/arXiv.2201.03801",
language = "English",
volume = "43",
pages = "3108--3125",
journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence",
issn = "0162-8828",
publisher = "IEEE Computer Society",
number = "9",
}

Publication Details

Winning Solutions and Post-Challenge Analyses of the ChaLearn AutoDL Challenge 2019

Funded by