Publication Details

PriorBand

HyperBand + Human Expert Knowledge

authored by
Neeratyoy Mallik, Carl Hvarfner, Danny Stoll, Maciej Janowski, Eddie Bergman, Marius Thomas Lindauer, Luigi Nardi, Frank Hutter
Abstract

Hyperparameters of Deep Learning (DL) pipelines are crucial for their performance. While a large number of methods for hyperparameter optimization (HPO) have been developed, they are misaligned with the desiderata of a modern DL researcher. Since often only a few trials are possible in the development of new DL methods, manual experimentation is still the most prevalent approach to set hyperparameters, relying on the researcher’s intuition and cheap preliminary explorations. To resolve this shortcoming of HPO for DL, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate the efficiency of PriorBand across a range of DL models and tasks using as little as the cost of 10 training runs and show its robustness against poor expert beliefs and misleading proxy tasks

Organisation(s)
Institute of Artificial Intelligence
External Organisation(s)
University of Freiburg
Lund University
Stanford University
Type
Conference contribution
Publication date
2022
Publication status
Published
Peer reviewed
Yes
Electronic version(s)
https://openreview.net/forum?id=ds21dwfBBH (Access: Open)