PriorBand

HyperBand + Human Expert Knowledge

verfasst von: Neeratyoy Mallik, Carl Hvarfner, Danny Stoll, Maciej Janowski, Eddie Bergman, Marius Thomas Lindauer, Luigi Nardi, Frank Hutter
Abstract: Hyperparameters of Deep Learning (DL) pipelines are crucial for their performance. While a large number of methods for hyperparameter optimization (HPO) have been developed, they are misaligned with the desiderata of a modern DL researcher. Since often only a few trials are possible in the development of new DL methods, manual experimentation is still the most prevalent approach to set hyperparameters, relying on the researcher’s intuition and cheap preliminary explorations. To resolve this shortcoming of HPO for DL, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate the efficiency of PriorBand across a range of DL models and tasks using as little as the cost of 10 training runs and show its robustness against poor expert beliefs and misleading proxy tasks
Organisationseinheit(en): Institut für Künstliche Intelligenz
Externe Organisation(en): Albert-Ludwigs-Universität Freiburg
Lund University
Stanford University
Typ: Aufsatz in Konferenzband
Publikationsdatum: 2022
Publikationsstatus: Veröffentlicht
Peer-reviewed: Ja
Elektronische Version(en): https://openreview.net/forum?id=ds21dwfBBH (Zugang: Offen)

BibTeX

@inproceedings{52c8b0af84db4d958c1f7d323e5a46b2,
title = "PriorBand: HyperBand + Human Expert Knowledge",
abstract = "Hyperparameters of Deep Learning (DL) pipelines are crucial for their performance. While a large number of methods for hyperparameter optimization (HPO) have been developed, they are misaligned with the desiderata of a modern DL researcher. Since often only a few trials are possible in the development of new DL methods, manual experimentation is still the most prevalent approach to set hyperparameters, relying on the researcher{\textquoteright}s intuition and cheap preliminary explorations. To resolve this shortcoming of HPO for DL, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate the efficiency of PriorBand across a range of DL models and tasks using as little as the cost of 10 training runs and show its robustness against poor expert beliefs and misleading proxy tasks",
author = "Neeratyoy Mallik and Carl Hvarfner and Danny Stoll and Maciej Janowski and Eddie Bergman and Lindauer, {Marius Thomas} and Luigi Nardi and Frank Hutter",
note = "Funding Information: Frank Hutter, Neeratyoy Mallik, Danny Stoll, Maciej Janowski and Eddie Bergman acknowledge funding by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215; the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828; the European Research Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765), funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them. Marius Lindauer acknowledges funding by the Europen Union under the ERC Starting Grant ixAu- toML (grant no. 101041029). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them. Luigi Nardi and Carl Hvarfner were supported in part by affiliate members and other supporters of the Stanford DAWN project — Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. Luigi Nardi was also supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348. The computations were also enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at LUNARC, partially funded by the Swedish Research Council through grant agreement no. 2018-05973",
year = "2022",
language = "English",
booktitle = "2022 NeurIPS Workshop on Meta Learning (MetaLearn)",
}

Details zu Publikationen

PriorBand

HyperBand + Human Expert Knowledge

Gefördert vom