Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Abstract
Meta-reinforcement learning algorithms can enable robots to acquire
new skills much more quickly, by leveraging prior experience to learn how to
learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used
meta-reinforcement learning benchmark uses different running velocities for a
simulated robot as different tasks. When policies are meta-trained on such narrow
task distributions, they cannot possibly generalize to more quickly acquire entirely
new tasks. Therefore, if the aim of these methods is enable faster acquisition of
entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose
an open-source simulated benchmark for meta-reinforcement learning and multitask learning consisting of 50 distinct robotic manipulation tasks. Our aim is to
make it possible to develop algorithms that generalize to accelerate the acquisition
of entirely new, held-out tasks. We evaluate 6 state-of-the-art meta-reinforcement
learning and multi-task learning algorithms on these tasks. Surprisingly, while
each task and its variations (e.g., with different object positions) can be learned
with reasonable success, these algorithms struggle to learn with multiple tasks at
the same time, even with as few as ten distinct training tasks. Our analysis and
open-source environments pave the way for future research in multi-task learning
and meta-learning that can enable meaningful generalization, thereby unlocking
the full potential of these methods
new skills much more quickly, by leveraging prior experience to learn how to
learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used
meta-reinforcement learning benchmark uses different running velocities for a
simulated robot as different tasks. When policies are meta-trained on such narrow
task distributions, they cannot possibly generalize to more quickly acquire entirely
new tasks. Therefore, if the aim of these methods is enable faster acquisition of
entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose
an open-source simulated benchmark for meta-reinforcement learning and multitask learning consisting of 50 distinct robotic manipulation tasks. Our aim is to
make it possible to develop algorithms that generalize to accelerate the acquisition
of entirely new, held-out tasks. We evaluate 6 state-of-the-art meta-reinforcement
learning and multi-task learning algorithms on these tasks. Surprisingly, while
each task and its variations (e.g., with different object positions) can be learned
with reasonable success, these algorithms struggle to learn with multiple tasks at
the same time, even with as few as ten distinct training tasks. Our analysis and
open-source environments pave the way for future research in multi-task learning
and meta-learning that can enable meaningful generalization, thereby unlocking
the full potential of these methods
Research Areas
