Episodic Curiosity through Reachability
Abstract
Rewards are sparse in the real world and most today’s reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the
agent to create rewards for itself — thus making rewards dense and more suitable
for learning. In particular, inspired by curious behaviour in animals, observing
something novel could be rewarded with a bonus. Such bonus is summed up with
the real task reward — making it possible for RL algorithms to learn from the
combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation
is compared with the observations in memory. Crucially, the comparison is done
based on how many environment steps it takes to reach the current observation
from those in memory — which incorporates rich information about environment
dynamics. This allows us to overcome the known “couch-potato” issues of prior
work — when the agent finds a way to instantly gratify itself by exploiting actions
which lead to hardly predictable consequences. We test our approach in visually
rich 3D environments in VizDoom, DMLab and MuJoCo. In navigational tasks
from VizDoom and DMLab, our agent outperforms the state-of-the-art curiosity
method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only. The code is available at
https://github.com/google-research/episodic-curiosity.
agent to create rewards for itself — thus making rewards dense and more suitable
for learning. In particular, inspired by curious behaviour in animals, observing
something novel could be rewarded with a bonus. Such bonus is summed up with
the real task reward — making it possible for RL algorithms to learn from the
combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation
is compared with the observations in memory. Crucially, the comparison is done
based on how many environment steps it takes to reach the current observation
from those in memory — which incorporates rich information about environment
dynamics. This allows us to overcome the known “couch-potato” issues of prior
work — when the agent finds a way to instantly gratify itself by exploiting actions
which lead to hardly predictable consequences. We test our approach in visually
rich 3D environments in VizDoom, DMLab and MuJoCo. In navigational tasks
from VizDoom and DMLab, our agent outperforms the state-of-the-art curiosity
method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only. The code is available at
https://github.com/google-research/episodic-curiosity.
Research Areas
