“Learning Robust Policies with Expert Guidance”

Speaker: Yang Cai
Assistant Professor of Computer Science at Yale University

Abstract: In Reinforcement Learning (RL), agent behavior is driven by a reward function. Misspecified rewards may lead to negative side effects when the agent acts unpredictably responding to the aspects of the environment that the designer overlooked, and potentially causes harms to the environment or itself. We propose a framework for ensuring robust behavior of an RL agent when the reward function may be difficult to specify. Assuming the existence of demonstrations from expert policies, we provide a theoretical framework for the agent to optimize in the space of rewards consistent with the expert behavior.  We propose two methods to solve the resulting optimization: an exact ellipsoid-based method and a method in the spirit of the “follow-the-perturbed-leader” algorithm. Our algorithms enable the trained agent to safely avoid states with potential negative effects while imitating the behavior of the expert in the other states.  

Speaker Bio: Yang Cai is an Assistant Professor of Computer Science at Yale University. Prior to joining Yale, he was an Assistant Professor in the Schools of Computer Science at McGill University. He finished his Ph.D. at MIT in Computer Science and received his B.Sc. in EECS at Peking University. His research interests lie in theoretical computer science and its interface with economics, probability, learning, and statistics. He has been honored with the 2019 Sloan Research Fellowship in Computer Science, the William Dawson Scholarship, and the Simons-Berkeley Research Fellowship. His dissertation has been recognized by the George M. Sprowls Award (for best MIT doctoral theses in CS) and the SIGecom Doctoral Dissertation Award.

.