Publication Details

Best Arm Identification with Retroactively Increased Sampling Budget for More Resource-Efficient HPO

authored by
Jasmin Brandt, Marcel Wever, Viktor Bengs, Eyke Hüllermeier
Abstract

Hyperparameter optimization (HPO) is indispensable for achieving optimal performance in machine learning tasks. A popular class of methods in this regard is based on Successive Halving (SHA), which casts HPO into a pure-exploration multi-armed bandit problem under finite sampling budget constraints. This is accomplished by considering hyperparameter configurations as arms and rewards as the negative validation losses. While enjoying theoretical guarantees as well as working well in practice, SHA comes, however, with several hyperparameters itself, one of which is the maximum budget that can be allocated to evaluate a single arm (hyperparameter configuration). Although there are already solutions to this meta hyperparameter optimization problem, such as the doubling trick or asynchronous extensions of SHA, these are either practically inefficient or lack theoretical guarantees. In this paper, we propose incremental SHA (iSHA), a synchronous extension of SHA, allowing to increase the maximum budget a posteriori while still enjoying theoretical guarantees. Our empirical analysis of HPO problems corroborates our theoretical findings and shows that iSHA is more resource-efficient than existing SHA-based approaches.

External Organisation(s)
Paderborn University
Ludwig-Maximilians-Universität München (LMU)
Munich Center for Machine Learning (MCML)
Type
Paper
Pages
3742-3750
No. of pages
9
Publication date
08.2024
Publication status
Published
Peer reviewed
Yes
Electronic version(s)
https://doi.org/10.24963/ijcai.2024/414 (Access: Closed)