Well-tuned Simple Nets Excel on Tabular Datasets

verfasst von

Arlind Kadra, Marius Lindauer, Frank Hutter, Josif Grabocka

Abstract

Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using a joint optimization over the decision on which regularizers to apply and their subsidiary hyperparameters. We empirically assess the impact of these regularization cocktails for MLPs in a large-scale empirical study comprising 40 tabular datasets and demonstrate that (i) well-regularized plain MLPs significantly outperform recent state-of-the-art specialized neural network architectures, and (ii) they even outperform strong traditional ML methods, such as XGBoost.

Organisationseinheit(en)

Fachgebiet Maschinelles Lernen
Institut für Informationsverarbeitung

Externe Organisation(en)

Albert-Ludwigs-Universität Freiburg
Bosch Center for Artificial Intelligence (BCAI)

Typ

Aufsatz in Konferenzband

Anzahl der Seiten

23

Publikationsdatum

2021

Publikationsstatus

Elektronisch veröffentlicht (E-Pub)

Peer-reviewed

Ja

Elektronische Version(en)

https://arxiv.org/abs/2106.11189 (Zugang: Offen)