Model-Based Reinforcement Learning and Search with Discrete World Models
- 9 views
Abstract
World models capture the dynamics of an environment and can be used to produce new, “imagined,” experiences. This can significantly reduce the number of real-world experiences required for an agent learn how to make decisions and can be combined with search for to allow an agent to plan before acting. However, in environments with a sub-symbolic representation, generating new experiences with a learned model over multiple timesteps can be difficult as small errors can accumulate over time. Furthermore, identifying previously encountered states during search can be difficult as the same state obtained by traversing different paths can result in slightly different representations. In this talk, I will discuss preliminary research on using discrete world models to address both issues. Using discrete world models, small errors can be corrected by simply rounding and identifying previously seen states is as simple as checking for equality between two arrays. Preliminary experiments with raw pixel representations of the Rubik’s cube and Sokoban show that a discrete world model can be learned using an offline dataset and can be unrolled over multiple timesteps without accumulating errors. Furthermore, after using the world model to learn a value function, combining the world model and value function with A* search solves 100% of test cases.
Bio
Forest Agostinelli is an assistant professor at the University of South Carolina. He received his B.S. from the Ohio State University, his M.S. from the University of Michigan, and his Ph.D. from the University of California, Irvine. His research group investigates how deep learning and reinforcement learning can be used to create agents that can solve complex problems and explain their solutions in a manner that humans can understand. His homepage is located at https://cse.sc.edu/~foresta/.
Location:
In person
Swearingen Engineering Center in Room 2A31
