Error controlled actor-critic

Gao, Xingen, Chao, Fei, Zhou, Changle, Ge, Zhen, Yang, Longzhi, Chang, Xiang, Shang, Changjing and Shen, Qiang (2022) Error controlled actor-critic. Information Sciences, 612. pp. 62-74. ISSN 0020-0255

AAM.pdf - Accepted Version
Available under License Creative Commons Attribution Non-commercial No Derivatives 4.0.

Download (4MB) | Preview
Official URL:


The approximation inaccuracy of the value function in reinforcement learning (RL) algorithms unavoidably leads to an overestimation phenomenon, which has negative effects on the convergence of the algorithms. To limit the negative effects of the approximation error, we propose error controlled actor-critic (ECAC) which ensures the approximation error is limited within the value function. We present an investigation of how approximation inaccuracy can impair the optimization process of actor-critic approaches. In addition, we derive an upper bound for the approximation error of the Q function approximator and discover that the error can be reduced by limiting the KL- divergence between every two consecutive policies during policy training. Experiments on a variety of continuous control tasks demonstrate that the proposed actor-critic approach decreases approximation error and outperforms previous model-free RL algorithms by a significant margin.

Item Type: Article
Additional Information: Funding Information: This work was supported by the Natural Science Foundation of Fujian Province of China (No. 2021J01002) and the High-level Talent Project of Xiamen University of Technology (No. YKJ22028R).
Uncontrolled Keywords: Actor-critic, Approximation error, KL-divergence, Overestimation, Reinforcement learning
Subjects: G400 Computer Science
G500 Information Systems
Department: Faculties > Engineering and Environment > Computer and Information Sciences
Depositing User: Rachel Branson
Date Deposited: 16 Sep 2022 08:19
Last Modified: 27 Aug 2023 03:30

Actions (login required)

View Item View Item


Downloads per month over past year

View more statistics