Publication Details

Automatic Risk Adaptation in Distributional Reinforcement Learning

authored by
Frederik Schubert, Theresa Eimer, Bodo Rosenhahn, Marius Lindauer
Abstract

The use of Reinforcement Learning (RL) agents in practical applications requires the consideration of suboptimal outcomes, depending on the familiarity of the agent with its environment. This is especially important in safety-critical environments, where errors can lead to high costs or damage. In distributional RL, the risk-sensitivity can be controlled via different distortion measures of the estimated return distribution. However, these distortion functions require an estimate of the risk level, which is difficult to obtain and depends on the current state. In this work, we demonstrate the suboptimality of a static risk level estimation and propose a method to dynamically select risk levels at each environment step. Our method ARA (Automatic Risk Adaptation) estimates the appropriate risk level in both known and unknown environments using a Random Network Distillation error. We show reduced failure rates by up to a factor of 7 and improved generalization performance by up to 14% compared to both risk-aware and risk-agnostic agents in several locomotion environments.

Organisation(s)
Institute of Information Processing
Machine Learning Section
Type
Preprint
No. of pages
14
Publication date
11.06.2021
Publication status
E-pub ahead of print
Electronic version(s)
https://arxiv.org/abs/2106.06317 (Access: Open)