A paper by Sili Huang, a PhD student of the School of Artificial Intelligence, titled "Generalizable Causal Reinforcement Learning for Out-of-distribution Environments," has been accepted byIEEE Transactions on Industrial Informatics, a premier academic journal under IEEE that focuses on the field of industrial informatics, covering the application of technologies such as industrial automation, the Internet of Things, big data analytics and artificial intelligence in industry.
Huang is a member of the reinforcement learning project team and has long been engaged in research related to deep reinforcement learning and transfer learning. This work was jointly supervised by Prof. Bo Yang and Prof. Hechang Chen, and was completed in collaboration with Prof. Peng Cui of Tsinghua University and Prof. Lichao Sun of Lehigh University, USA.
Generalizable Causal Reinforcement Learning for Out-of-distribution Environments
First author: Sili Huang
Journal:IEEE Transactions on Industrial Informatics(TII)
Overview
Out-of-distribution (OOD) generalization is key to applying reinforcement learning algorithms in real-world applications. To address the OOD problem, recent work has focused on learning OOD-adaptive policies by capturing the causal factors that influence environment dynamics. These works, however, recover causal factors only in entangled or binary form, and the generalization they achieve requires additional data from the test environment. To break this limitation, this paper proposes Generalizable Causal Reinforcement Learning (GCRL) for learning disentangled representations of causal factors. On this basis, the paper trains policies that achieve OOD generalization without additional training. To capture causal factors, GCRL deploys a weakly supervised signal with two-stage constraints to ensure that all factors are disentangled. Then, to achieve OOD generalization through causal factors, GCRL establishes dependencies between actions and representations, and optimizes the policy model across multiple environments. Experimental results show that, when the learned policy is able to solve the target task in the training environment, the established dependencies recover the correct association between causal factors and behaviors, and achieve OOD generalization on eight benchmarks in CausalWorld and MuJoCo.
