Symbolic Explanations for Hyperparameter Optimization

authored by

Sarah Segel, Helena Graf, Alexander Tornede, Bernd Bischl, Marius Lindauer

Abstract

Hyperparameter optimization (HPO) methods can determine well-performing hyperparameter configurations efficiently but often lack insights and transparency. We propose to apply symbolic regression to meta-data collected with Bayesian optimization (BO) during HPO. In contrast to prior approaches explaining the effects of hyperparameters on model performance, symbolic regression allows for obtaining explicit formulas quantifying the relation between hyperparameter values and model performance. Overall, our approach aims to make the HPO process more explainable and human-centered, addressing the needs of multiple user groups: First, providing insights into the HPO process can support data scientists and machine learning practitioners in their decisions when using and interacting with HPO tools. Second, obtaining explicit formulas and inspecting their properties could help researchers better understand the HPO loss landscape. In an experimental evaluation, we find that naively applying symbolic regression directly to meta-data collected during HPO is affected by the sampling bias introduced by BO. However, the true underlying loss landscape can be approximated by fitting the symbolic regression on the surrogate model trained during BO. By penalizing longer formulas, symbolic regression furthermore allows the user to decide how to balance the accuracy and explainability of the resulting formulas.

Organisation(s)

Machine Learning Section

External Organisation(s)

Munich Center for Machine Learning (MCML)
Ludwig-Maximilians-Universität München (LMU)

Type

Conference contribution

Publication date

16.05.2023

Publication status

E-pub ahead of print

Peer reviewed

Yes

Electronic version(s)

https://openreview.net/forum?id=JQwAc91sg_x (Access: Open)