Publication Details

CAVE: Configuration Assessment, Visualization and Evaluation

authored by
André Biedenkapp, Joshua Marben, Marius Lindauer, Frank Hutter
Abstract

To achieve peak performance of an algorithm (in particular for problems in AI), algorithm configuration is often necessary to determine a well-performing parameter configuration. So far, most studies in algorithm configuration focused on proposing better algorithm configuration procedures or on improving a particular algorithm’s performance. In contrast, we use all the collected empirical performance data gathered during algorithm configuration runs to generate extensive insights into an algorithm, given problem instances and the used configurator. To this end, we provide a tool, called CAVE, that automatically generates comprehensive reports and insightful figures from all available empirical data. CAVE aims to help algorithm and configurator developers to better understand their experimental setup in an automated fashion. We showcase its use by thoroughly analyzing the well studied SAT solver spear on a benchmark of software verification instances and by empirically verifying two long-standing assumptions in algorithm configuration and parameter importance: (i) Parameter importance changes depending on the instance set at hand and (ii) Local and global parameter importance analysis do not necessarily agree with each other.

External Organisation(s)
University of Freiburg
Type
Conference contribution
Pages
115-130
No. of pages
16
Publication date
31.12.2018
Publication status
Published
Peer reviewed
Yes
ASJC Scopus subject areas
Theoretical Computer Science, Computer Science(all)
Electronic version(s)
https://doi.org/10.1007/978-3-030-05348-2_10 (Access: Closed)