Abstract
The “Curse of Dimensionality” issue of dynamic programming-based control approaches for large-scale state and action space of dynamic systems or agents led to the development of approximate dynamic programming (ADP). The approximate dynamic programming unifies the theory of optimal control, adaptive control, and reinforcement learning (RL) to obtain an approximate solution to the Bellman equation online and forward-in-time. In general, the value function, which is the solution to the Bellman equation in a discrete-time framework or Hamilton-Jacobi-Equation (HJB) in a continuous-time framework, is approximated using a neural network-based approximator.  The learning/adaptive nature of the solution often partially or fully relaxes the assumption of complete system information, which leads to optimal decision-making in uncertain/unknown environments. This presentation will traverse the evolution of the ADP/RL-based optimal control designs for dynamic cyber-physical systems, moving from traditional iterative solutions to those that emphasize time-based solutions. Specifically, there will be a focus on computation and communication-saving aspects of the ADP/RL-based designs.  The resource-aware ADP scheme, referred to as event-driven ADP, using Q-learning and Temporal Difference learning approaches will be discussed in detail. The event-driven approaches train the neural network approximators and update the actions at certain events only, thereby considerably minimizing the computational and communication requirements for the implementation of the learning-based control schemes over the communication network. Concluding this presentation, we will probe into some of the unresolved challenges of ADP/RL schemes, emphasizing their potential vulnerabilities in a cyber-physical framework.

Bio
Avimanyu Sahoo received his Ph.D. in Electrical Engineering from  Missouri University of Science and Technology, Rolla, MO, USA, in 2015 and a Masters of Technology (MTech) and the Indian Institute of Technology (BHU), Varanasi, India, in 2011. He is currently an Assistant Professor in the Electrical and Computer Engineering Department at the University of Alabama in Huntsville (UAH), AL. Prior to joining UAH, Dr. Sahoo was an Associate Professor in the Division of Engineering Technology at Oklahoma State University, Stillwater, OK.

Dr. Sahoo’s research interest includes learning-based control and its applications in lithium-ion battery pack modeling, diagnostics, prognostics, cyber-physical systems, and electric machinery health monitoring. Currently, his research focuses on developing intelligent battery management systems (BMS) for lithium-ion battery packs used onboard electric vehicles, computation, and communication-efficient distributed intelligent control schemes for cyber-physical systems using approximate dynamic programming, reinforcement learning, and distributed adaptive state estimation.

Link