On January 19, Andy Younes, a Research Assistant at the Center for Scientific Innovation and Education (CSIE), led an insightful seminar on the advancements in reinforcement learning (RL), focusing on model-free deep RL algorithms and their real-world applications. She addressed key challenges in deep RL, particularly the high sample complexity and sensitivity to hyperparameters, which can hinder the effectiveness of these methods in practical domains.

Abstract:

Model-free deep reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision-making and control tasks. However, these methods typically suffer from two major challenges: high sample complexity and brittleness to hyperparameters. Both of these challenges limit the applicability of such methods to real-world domains. In this paper, we describe Soft Actor-Critic (SAC), our recently introduced off-policy actor-critic algorithm based on the maximum entropy RL framework. In this framework, the actor aims to simultaneously maximize expected return and entropy; that is, to succeed at the task while acting as randomly as possible. We extend SAC to incorporate a number of modifications that accelerate training and improve stability with respect to the hyperparameters, including a constrained formulation that automatically tunes the temperature hyperparameter. We systematically evaluate SAC on a range of benchmark tasks, as well as challenging real-world tasks such as locomotion for a quadrupedal robot and robotic manipulation with a dexterous hand. With these improvements, SAC achieves state-of-the-art performance, outperforming prior on-policy and off-policy methods in sample-efficiency and asymptotic performance. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving similar performance across different random seeds. These results suggest that SAC is a promising candidate for learning in real-world robotics tasks.

About the Speaker:

Andy Younes is a Research Assistant at CSIE specializing in machine learning and control systems. She completed her Bachelor’s degree in Software Engineering from NPUA in 2024. Andy has contributed to several research projects, including the development of a machine learning algorithm for deblurring, denoising, and colorizing old images, which she explored in her Bachelor's dissertation. Recently, she also led a team to a third-place finish in the 2024 American Control Conference Self-Driving Car Student Competition, where they presented a state-of-the-art control algorithm combined with advanced object detection technology.